Persistent Memory Programming Blog
Here’s a list of blog posts, shown most recent first:
July 26, 2019 – C++ standard limitations and Persistent Memory
Introduction C++ language restrictions and the persistent memory programming paradigm imply serious restrictions on objects which may be stored on persistent medium. A user can access persistent memory with memory mapped files to take advantage of its byte addressability thanks to libpmemobj and Storage Networking Industry Association non-volatile memory programming…
June 12, 2019 – Multi-level vmemcache
Introduction vmemcache which we have recently described performs close to optimum when either all keys are approximately equally likely to be queried, or when all key:value pairs fit completely into the fastest form of memory you are willing to use. But, in many workloads, some keys are “hot” and queried…
May 07, 2019 – libvmemcache - buffer-based LRU cache
Introduction libvmemcache is a volatile key-value store optimized for operating on NVDIMM based space. However, it can work with any filesystem whether it is stored in memory (tmpfs) or on any storage device. Consequently, libvmemcache will be significantly less performant if it is stored on the storage device other than…
February 20, 2019 – C++ persistent containers - vector
Introduction The main idea behind pmem containers is to fully exploit persistent memory potential by designing optimized on-media layouts and algorithms for persistent memory programming. On November, we published a blog post about pmem containers. If you haven’t read it yet, I encourage you to do that now. We have…
February 04, 2019 – Pmreorder basics
Introduction It’s good practice to run persistent memory application under pmemcheck - a tool which is described here and here. In this post, we are going to learn about another tool for persistence correctness checking. As you might already know if you’ve read posts linked above, pmemcheck verifies if all…
February 01, 2019 – Pool conversion tool
Introduction When we published the first PMDK stable release, we committed to maintaining stable on-media layout. This means that all future PMDK changes have to be backward compatible. Unfortunately, we weren’t successful in adhering to the strict requirements which would be needed to maintain compatibility, mostly because we made changes…
January 22, 2019 – Extended memcpy in PMDK 1.5
In PMDK 1.5 we added new APIs for bulk persistent memory modifications. In short, we did this to: give applications the ability to perform low-level performance optimizations clean up the naming scheme In order to understand what exactly and why we did that, let’s review the old API. In PMDK…
December 05, 2018 – Pool features
Introduction Since the very first release, PMDK pools had internal feature flags. They were mostly a hidden implementation detail. The more observant users might have noticed pool features listing in pmempool info output, but that’s about it. Release 1.5 introduced a set of new feature flags. And since it’s imperative…
November 26, 2018 – Bad blocks
Introduction Over time, storage devices can acquire uncorrectable media errors often called “bad blocks”. A bad block is a part of a storage media that is either inaccessible or unwritable due to a permanent physical damage. In case of memory mapped I/O, if a process tries to access (read or…
November 20, 2018 – C++ persistent containers
PMEM containers Our goal for the libpmemobj C++ bindings is to create a friendly and less error prone API for persistent memory programming. Even with persistent memory pool allocators, convenient interface for creating and managing transactions, auto-snapshotting class templates and smart persistent pointers, designing an application with persistent memory usage…
November 02, 2018 – C++ persistent containers - array
Introduction Until now, our C++ bindings were missing one important component - persistent containers. In 1.5 release we have introduced the first one - pmem::obj::array. This container is currently placed in experimental namespace and folder (this means that both API and layout can change). It has almost the same functionality…
October 22, 2018 – New release of PMDK
We’ve been very quiet on this blog as of late, mostly because of the amount of work that we needed to put into our very ambitiously planned 1.5 release. But we’ve made it, and there’s finally time to get back to discussing the technical minutiae of our work. In this…
June 25, 2018 – Running FIO with pmem engines
When we, the PMDK team, want to check performance of our library, either to see if there was any regression or if our tweaks did a good job, we run benchmarks. One of them is FIO. It helps us simulate synthetic traffic of reads and writes to a pmem device….
May 15, 2018 – Using Persistent Memory Devices with the Linux Device Mapper
Introduction X86/X64 systems do not typically interleave Persistent Memory Devices (also referred to as ‘modules’ or ‘DIMMs’) across sockets, so a two-socket system will have two separate interleave sets. To use these interleave sets as a single device requires using a software device mapper or volume manager. This article focuses…
December 27, 2017 – Benchmarking with different storage engines using pmemkv
We’re closing out 2017 with two big improvements to pmemkv: support for multiple storage engines, and an improved benchmarking utility based on db_bench. These changes set the stage for some interesting experiments to come next year, as we continue to add new features and tune performance of pmemkv and its…
December 11, 2017 – Announcing the Persistent Memory Development Kit
This is to announce a name change: The NVML project is now known as PMDK, the Persistent Memory Development Kit. Why the name change? The old name, NVML, made it sound like the project produced a single library that applied to Non-Volatile Memory. In reality, the project currently supports ten…
October 05, 2017 – Progress Report Q3 2017
The last quarter was rather… peaceful. But nevertheless there were a few noteworthy things. FreeBSD & ARM We always asserted that our library is multi-platform and hardware agnostic… as long as your platform is a recent distribution of Linux (or Windows) on x86 hardware :) Two things happened that intend…
July 10, 2017 – Using Standard Library Containers with Persistent Memory
Introduction Somewhere along the road, when we were doing the C++ bindings for libpmemobj, we found the need for some kind of containers. We were faced with two viable solutions: write everything from scratch or adapt an existing implementation. The obvious choice was NOT to implement from scratch. We would…
July 04, 2017 – Progress Report Q2 2017
It’s summer already (at least in my hemisphere) ! :) And it’s time for the next progress report. Control interface After a very long in the oven, we’ve finally finalized and merged the CTL API which allows for introspection and modification of the internal state of the library. This feature…
April 03, 2017 – Apache Kudu Persistent Memory Enabled Block Cache
Using NVM Libraries To Add Persistent Memory Support to Apache Kudu Block Cache An early project done with the NVM libraries was adding persistent memory support, both volatile and persistent mode, into the Apache Kudu storage engine block cache. This project required modification of existing code. Apache Kudu: http://github.com/apache/kudu My…
March 29, 2017 – Progress Report Q1 2017
It’s been three months already since the last time I wrote something ;) Time really flies by quickly when you are doing interesting stuff. We’ve been very busy with lots of improvements to the library as well as A LOT of tiny fixes: over the last three months the team…
March 09, 2017 – Zero-copy leaf splits in pmemkv
In a B+ tree, splitting a full leaf into two leaves is one of its slowest operations, but pmemkv optimizes this using a zero-copy strategy. Rather then copying any key/value data between full and new leaf, pmemkv splits leaves by swapping persistent structures in place. This minimizes write amplification and…
February 21, 2017 – Introducing pmemkv
We’ve blogged before about building and optimizing key-value stores for persistent memory, and we’re excited to put these ideas to the test in a more formal way. Our new pmemkv project is an open-source key-value store that is optimized for read-heavy workloads on persistent memory. Compared with key-value stores based…
January 25, 2017 – What’s coming in libpmemobj
In my last post I’ve made a promise to share our plans for the near future. So here it is: 4 ideas that we are planning to ship with the upcoming version of libpmemobj. Please note that most of our plans related to libpmemobj are available on our github issues…
January 23, 2017 – Modeling strings with libpmemobj C++ bindings
C++ developers using libpmemobj have more than one option for modeling strings, depending on the size of the strings and whether they are fixed or varying in length. In this post we’ll review the representations that work, known variations to avoid, and finally present a persistent string class that implements…
December 21, 2016 – PMDK for Windows
Throughout 2016 a team of engineers from Microsoft, Intel, HPE and HPI have been working to port the PMDK project to Windows and we are happy to announce that Technical Preview release is now available! Our main goal in porting the library was to make it as easy as possible…
December 20, 2016 – libpmemobj - a year in review
It’s been a while since the last post on our blog, but we’ve been busy with the recently released 1.2 version of the library. It comes packed with improvements all throughout the code base and it also brings a handful of new features that we hope will end up being…
June 02, 2016 – C++ bindings for libpmemobj (epilogue) - converting existing applications
During the development of the C++ bindings, we wrote a couple of examples and even more tests. But these are new applications written from scratch to understand persistence. While this approach is OK for newly developed apps, there is a lot of existing code out there that is not designed…
May 31, 2016 – C++ bindings for libpmemobj (part 7) - synchronization primitives
To finish off the C++ bindings to libpmemobj blog marathon, I will introduce to you the synchronization mechanisms we implemented. They are mostly C++11-like implementations of different kinds of mutexes and the condition variable. They satisfy their respective concepts (Mutex, SharedMutex and so on), the difference is that they are…
May 25, 2016 – C++ bindings for libpmemobj (part 6) - transactions
As I mentioned in my previous blog post, transactions are the heart of libpmemobj. That is why we had to take utmost care while designing their C++ versions, so that they are as easy to use as possible. There are, however, a couple of compromises we had to make due…
May 19, 2016 – C++ bindings for libpmemobj (part 5) - make_persistent
One of the most important features of the C++ bindings to libpmemobj is the persistent_ptr smart pointer template. While using it is fairly straightforward, the allocation and object construction with the use of the C API is hard to get right. So like it’s C++ standard’s counterparts, it needed an…
May 10, 2016 – C++ bindings for libpmemobj (part 4) - pool handle wrapper
One of the necessary steps in developing the C++ libpmemobj bindings was the introduction of an abstraction of the C pool handle. We decided to do a very simple hierarchy where the pool template inherits from a generic pool_base. This was necessary to be able to have functions/methods which do…
February 25, 2016 – Persistent allocator design - fragmentation
Implementing a memory allocator is a balance between numerous properties with the two most important being time and space constraints. Making the malloc/free routines reasonably fast is a must for the implementation to be considered usable at all. The algorithm also mustn’t waste excessive amounts of memory. During development of…
February 22, 2016 – How to emulate Persistent Memory
Data allocated with PMDK is put to the virtual memory address space, and concrete ranges are relying on result of mmap(2) operation performed on the user defined files. Such files can exist on any storage media, however data consistency assurance embedded within PMDK requires frequent synchronisation of data that is…
January 12, 2016 – C++ bindings for libpmemobj (part 3) - persistent queue example
The best way to learn to code is usually by implementing an example. We are going to be creating a linked-list based queue data structure using the the pmem::obj::p and pmem::obj::persistent_ptr classes and libpmemobj C API. But first, a little bit of CS 101 :) Linked-list queue Queue is a…
January 12, 2016 – C++ bindings for libpmemobj (part 2) - persistent smart pointer
In our C API the programmer has to deal with custom pointers represented by the PMEMoid structure. Thanks to some macro magic we made it so that those PMEMoids are somewhat usable. C++ allows us to evolve this concept. pmem::obj::persistent_ptr Almost everyone who ever touched a C++ code knows the…
January 12, 2016 – C++ bindings for libpmemobj (part 1) - pmem resident variables
One of the biggest hurdles and error prone things about our C API is that the user has to manually keep track of modifications to the persistent memory resident variables while in a transaction. A special semi-transparent template property class has been implemented to automatically add variable modifications to the…
January 12, 2016 – C++ bindings for libpmemobj (part 0)
Our goal for the C pmemobj library was to make a fully featured implementation of persistent memory programming model without modifying the compiler. It’s meant for authors of very low-level system software and language creators. It’s not particularly pretty nor easy to use. The amount of macros, as well as…
December 15, 2015 – Performance improvements
I would like to inform you about the performance improvements that have been going on in PMDK and libpmemobj in particular. We have not been standing still and we are trying out a couple of ideas on how to make our libraries even faster. Some of the improvements are smaller,…
November 23, 2015 – An introduction to replication
Replication is a means for raising the reliability of your pmemobj based applications. You can basically think of it as RAID 1 within PMDK. What happens is, when you write to your pool using the pmemobj_* (memcpy, persist, and so on) primitives, it gets copied to your replicas. Yes, you…
October 20, 2015 – Evaluation of a better object container
During performance evaluation of our library, I asked myself a following question: Which data structure has computational complexity of “insert at end” and “remove given element” operations no worse than a doubly-linked list, but with a smaller constant? The point of that mental exercise was to come up with a…
September 29, 2015 – pmemobjfs - The simple FUSE based on libpmemobj
How to use it The sources of the pmemobjfs file system are available here. Please refer to README file for instructions on how to create a file system layout and mount it. NOTE: This is just an example implementation of file system in user space using the libpmemobj library and…
September 16, 2015 – Challenges of multi-threaded transactions
Our library currently does not support threads cooperating (writing) within a single transaction. It does shift a lot of work from the library onto the user who now has to think about different parallelization solutions. This was a conscious decision with iterative approach to creating the library in mind. It…
September 10, 2015 – KV-store improved & measured
As promised in the previous post about the kv-store implementation I’m back with new results after implementing the optimizations I devised a month ago. As a bonus I implemented a red-black tree to have a fair comparison between two data structures that allocate similar number of nodes. tl;dr: I was…
July 31, 2015 – Transactional key-value store using libpmemobj - DIY
Our library often gets compared to NoSQL databases because it stores things on storage in unstructured manner. Which is true, but, when you think about it, the pmemobj library is not technically a database, but can be used to implement one - like the MySQL storage engine example. In this…
July 20, 2015 – An introduction to pmemcheck (part 2) - transactions
In my previous blog post I described the key features of the new persistent memory analysis tool we created - pmemcheck. You should now be aware of the main pitfalls of persistent memory programming and of ways pmemcheck informs you about possible misuses of PMEM. We should now dive into…
July 17, 2015 – An introduction to pmemcheck (part 1) - basics
As you probably noticed from the previous posts, persistent memory programming isn’t really that easy. There are a couple of things you have to consider - data consistency being the most important one. The contemporary x86_64 architecture supports at most 8-byte atomic stores. You probably know by now, that by…
June 19, 2015 – An introduction to pmemobj (part 7) - persistent lists
The pmemobj library provides non-transactional persistent atomic circular doubly-linked lists (or NTPACDLL for short) API with an interface familiar to anyone who have ever included sys/queue.h header file - it’s in fact so similar that I considered not writing this post at all, you can just search the web for…
June 18, 2015 – An introduction to pmemobj (part 6) - threading
All of the pmemobj library functions are thread-safe, with following two exceptions: pool management functions (open, close and friends) and pmemobj_root when providing different sizes in different threads - so as long as you are using this function the way it’s meant to be used you don’t have to worry…
June 18, 2015 – An introduction to pmemobj (part 5) - atomic dynamic memory allocation
In the previous post I talked about using transactions for allocating new objects, which is fine and is the most similar approach to the standard POSIX way. But it does add an overhead of maintaining an undo log of changes. A more optimal memory management can be achieved using the…
June 17, 2015 – An introduction to pmemobj (part 4) - transactional dynamic memory allocation
This is a topic I intentionally avoided not to introduce too much complexity too fast. The pmemobj library contains an implemented from scratch memory allocator, that was designed with persistent memory in mind. There are two separate APIs: non-transactional and transactional. Transactional allocations Let’s start with a simple snippet of…
June 16, 2015 – An introduction to pmemobj (part 3) - types
In all of the previous post the code snippets and examples had persistent pointers (PMEMoid) without any type information - they were simple C structures. Very early in the development of the library we discovered that using something like that was extremely error-prone and generally difficult. That’s why considerable effort…
June 15, 2015 – An introduction to pmemobj (part 2) - transactions
By now you should be fairly familiar with the basics persistent memory programming, but to make sure the application is always in a consistent state you had to rely on your own solutions and tricks - like the length of a buffer in the previous example. Now, we will learn…
June 13, 2015 – An introduction to pmemobj (part 1) - accessing the persistent memory
In the previous post, you learned a little bit about the general concept of the persistent memory programming model, now it’s time to start the coding ;) Memory pools If you’ve read the overview you know that persistent memory is exposed by the OS as memory-mapped files, we call them…
June 12, 2015 – An introduction to pmemobj (part 0) - new programming model
The aim of this tutorial series is to introduce you to programming with persistent, byte-addressable memory using the pmemobj library. We will go over all the available features, implement an example application and learn something about the inner workings of libpmemobj. If you haven’t read the NVM Library overview I…
June 11, 2015 – Type safety macros in libpmemobj
The PMEMoid plays the role of a persistent pointer in a pmemobj pool. It consist of a shortened UUID of the pool which the object comes from and an offset relative to the beginning of the pool: 1 2 3 4 typedef struct pmemoid { uint64_t pool_uuid_lo; uint64_t off; }…
June 02, 2015 – Implementing (simple) MySQL storage engine with libpmemobj
The focus of the pmemobj library, like the name suggests, is storing objects on a persistent medium. A different, but very common, approach of doing exactly the same is to use a database with a specialized interface to manipulate the collection of data. MySQL is one such database, it processes…
September 23, 2014 – Using the Block Translation Table for sector atomicity
Persistent memory based storage is able to perform IO at byte (or more accurately, cache line) granularity. However, we often want to expose such storage as traditional block devices. The block drivers for persistent memory will do exactly this. However, they do not provide any atomicity guarantees. Traditional SSDs typically…
September 09, 2014 – Git Workflow
Now that we’ve created the GitHub Repository for the PMDK, here’s a more detailed description of the git workflow we’ve chosen. The basic idea is: Current development happens on the master branch Releases are created periodically by tagging After a major release, a stable-1.x branch is created. All bug fixes…
September 01, 2014 – NVM Library Overview
[Edit on 2017-12-11: In the years since this entry was written, this work has evolved into PMDK.] Why are we building an NVM Library? Where does it live? How does it work? This blog entry provides some answers, which refer to this picture showing the overall library architecture: Why? The…
August 29, 2014 – Linux Examples
[Edit on 2017-12-11: The Linux examples are outdated now, look at PMDK instead.] The basic architecture for exposing persistent memory gives applications a very raw type of access. Applications can load/store directly to the persistence, but then what. What are the interesting problems facing an application developer and what would…
August 27, 2014 – Crawl, Walk, Run…
If you can’t fly then run, if you can’t run then walk, if you can’t walk then crawl, but whatever you do you have to keep moving forward. Martin Luther King Jr. This project, as well as the support for persistent memory in various operating systems, can be thought of…
August 26, 2014 – References
Some very interesting research has been happening in the area of Persistent Memory and more is emerging. While the examples provided here are meant as introductory and simple, some publications cover the topic in much more depth and include complete transaction systems, compiler/language enhancements, etc. Here are some of the…
August 25, 2014 – Creating pmem.io
The pmem project in GitHub has been created as an open source project focused on persistent memory programming. Everything on this web site and the associated GitHub repositories is open source under the “three-clause” BSD license. Some educational Linux examples are included, which demonstrate the SNIA NVM programming model and…