Open in app

Sign in

Write

Sign in

Enable Intel NVDIMM on Ubuntu

Woodrow Shen
5 min readNov 26, 2018

Intel announced their NVDIMM product series “Optane DC Persistent Memory” in May, which introduced a new class between DRAM (Memory) and SSD (Storage). For OS-level perspective, some distributions[1] already supported NVDIMM since Linux Kernel 4.2 has patchworks for that.

In this article, it illustrates how to enable Intel NVDIMM on Ubuntu based system with given instructions to achieve the enablement quickly. However, the HW configuration for NVDIMM relies on platform specific, so this work won’t be addressed here. The test environment is revealed as example:

  • Intel Cascade Lake (Must)
  • Memory configuration: 1*128 GB NVDIMM + 2*8 GB DRAM on CPU0
  • Vanilla Ubuntu 18.04

Assuming the machine with Intel NVDIMM module can boot-up without particular issues, there are three approaches to apply NVDIMM:

  • Memory mode (MM)
  • App Direct mode (AD)
  • Mixed mode

Memory mode will be data volatile, which means it’s equivalent to memory and almost has competitive latencies compared to DRAM. App Direct is 100% capacity to be used for storage and its management is quite more complicated than memory. Mixed mode is co-existence of MM and AD mode, by given arbitrary percentage for them. For example, if there is one 128 GB module, the administrator can configure 50:50 for MM:AD mode. So ideally that’s 64 GB for memory and 64 GB for AD storage. Be aware that original DRAM modules in the system are also treated as WB cache controlled by processors, no matter which mode is used.

Intel NVDIMM SW stack implements several aspects to enable NVDIMM:

  • nvdimm driver[2]
  • Direct Access (DAX) support with file system[3]
  • Block Translation Table (BTT)[4]
  • UEFI Human Interface Infrastructure (HII) Firmware
  • ndctl[5]: Utility library for managing the libnvdimm (non-volatile memory device) sub-system in the Linux kernel
  • ipmctl[6]: Utility for configuring and managing Intel Optane DC persistent memory modules

Basically, the kernel 4.15 on Ubuntu 18.04 already covers nvdimm driver, DAX and BTT features. The Intel DCPMEM SW for UEFI HII depends on vendor specific, and it needs confirmation to make sure if it’s integrated with BIOS or not. ndctl package is in the universe of Ubuntu archive so it’s easy to be installed by apt. For ipmctl package, it can be installed from my PPA:

sudo add-apt-repository ppa:woodrow-shen/ppa
sudo apt-get update
sudo apt install -y ipmctl libsafec3

Procedure to enable AD mode

This section won't describe UEFI HII approach as it's a vendor specific.

Once ndctl and ipmctl installed, we can use ipmctl to show the information about nvdimm in the system:

$ sudo ipmctl show -dimm
DimmID    Capacity    HealthState    ActionRequired    LockState    FWVersion
0x0020    125.7 GiB    Healthy     0     Disabled    01.00.00.4351

Also, ipmctl can show information about nvdimm resource allocation:

$ sudo ipmctl show -memoryresourcesCapacity=125.7 GiB
MemoryCapacity=125.0 GiB
AppDirectCapacity=0.0 GiB
UnconfiguredCapacity=0.0 GiB
InaccessibleCapacity=0.0 GiB
ReservedCapacity=0.7 GiB

From above information, it's obvious to point out the current configuration is Memory mode, and the next step is to enable App Direct with full capacity. Before that, let's make sure if nvdimm has a capability to support AD or not. Likewise, ipmctl is used to show its capabilities:

$ sudo ipmctl show -a -system -capabilities
PlatformConfigSupported=1
Alignment=1.0 GiB
AllowedVolatileMode=Memory Mode
CurrentVolatileMode=Memory Mode
AllowedAppDirectMode=App Direct
ModesSupported=1LM, Memory Mode, App Direct
SupportedAppDirectSettings=x1 (ByOne), x2 - 4KB iMC x 4KB Channel (4KB_4KB), x3 - 4KB iMC x 4KB Channel (4KB_4KB), x4 - 4KB iMC x 4KB Channel (4KB_4KB), x6 - 4KB iMC x 4KB Channel (4KB_4KB)
RecommendedAppDirectSettings=x1 (ByOne), x2 - 4KB iMC x 4KB Channel (4KB_4KB), x3 - 4KB iMC x 4KB Channel (4KB_4KB), x4 - 4KB iMC x 4KB Channel (4KB_4KB), x6 - 4KB iMC x 4KB Channel (4KB_4KB)
MinNamespaceSize=1.0 GiB
AppDirectMirrorSupported=0
DimmSpareSupported=0
AppDirectMigrationSupported=0
RenameNamespaceSupported=1
GrowAppDirectNamespaceSupported=0
ShrinkAppDirectNamespaceSupported=0
InitiateScrubSupported=0
AdrSupported=1
EraseDeviceDataSupported=0
EnableDeviceSecuritySupported=0
DisableDeviceSecuritySupported=1
UnlockDeviceSecuritySupported=0
FreezeDeviceSecuritySupported=0
ChangeDevicePassphraseSupported=0

In order to make use nvdimm with AD mode, Intel ipmctl takes a responsibility for management but ndctl also tackles with namespace management. The conceptional workflow of creating a MM/AD mode is as below:

  • Create a memory allocation goal
  • Reboot system
  • Create a namespace for specific regions

To create a goal with 100% AD, just issue the command:

$ sudo ipmctl create -goal
The following configuration will be applied:SocketID    DimmID    MemorySize    AppDirect1Size    AppDirect2Size
0x0000     0x0020    0.0 GiB     125.0 GiB    0.0 GiB
Do you want to continue? [y/n] y
Created following region configuration goal
SocketID    DimmID    MemorySize    AppDirect1Size    AppDirect2Size
0x0000     0x0020    0.0 GiB     125.0 GiB    0.0 GiB
A reboot is required to process new memory allocation goals.

Before system reboots, the goal information can be printed by $ sudo ipmctl show -a -goal

After reboot, use ipmctl again to confirm if the goal was executed successfully:

$ sudo ipmctl show -memoryresourcesCapacity=125.7 GiB
MemoryCapacity=0.0 GiB
AppDirectCapacity=125.0 GiB
UnconfiguredCapacity=0.6 GiB
InaccessibleCapacity=0.0 GiB
ReservedCapacity=0.0 GiB

Assuming the goal worked correctly, and then creating a namespace for goal is as below:

$ sudo ndctl create-namespace
{
  "dev":"namespace0.0",
  "mode":"fsdax",
  "map":"dev",
  "size":"123.04 GiB (132.12 GB)",
  "uuid":"c3991a7f-deb1-4d0f-8379-80047dc63821",
  "raw_uuid":"8212ffc1-8f66-4ff7-bb22-198160e71170",
  "sector_size":512,
  "blockdev":"pmem0",
  "numa_node":0
}

So basically the namespace function will create a blockdev node (e.g. /dev/pmem0) responding to the specific region. In this case, a namespace is generated with full capacity of nvdimm. Furthermore, using additional parameter “-s” is to specify the requested region per command so that the multiple nodes would be generated simultaneously.

Now /dev/pmem0 exists in the system, the next step is to make everything as usual. For example, partitioning pmem0 via parted could be

$ parted -s -a optimal /dev/pmem0 \
        mklabel gpt -- \
        mkpart primary ext4 1MiB 4GiB \
        mkpart primary xfs 4GiB 12GiB \
        mkpart primary btrfs 12GiB -1MiB \
        print

Or, make a specific filesystem on /dev/pmem0 easily:

$ sudo mkfs.xfs /dev/pmem0

For mounting file system, ext4 and xfs support DAX. To get the benefit from it, the extra option needs to be added in the mount:

$ mkdir xfs-pmem0
$ sudo mount -o dax /dev/pmem0 xfs-pmem0

If /dev/pmem0 is successfully mounted, the dmesg should give the following messages:

$ mount | grep dax
/dev/pmem0 on /home/u/xfs-pmem0 type xfs (rw,relatime,attr2,dax,inode64,noquota)$ dmesg | grep pmem
[92747.084077] XFS (pmem0): Mounting V5 Filesystem
[92747.087630] XFS (pmem0): Ending clean mount$ dmesg | grep DAX
[ 1580.768764] XFS (pmem0): DAX enabled. Warning: EXPERIMENTAL, use at your own risk

Mixed mode

The mixed mode is still painless to manipulate ipmctl to give the specific percentage for memory mode (AD is just opposite value). For example, the configuration for MM:AD is set to 50:50, and the command will be issued by

$ ipmctl create -goal memorymode=50 persistentmemorytype=appdirect
$ ipmctl create -goal memorymode=50 persistentmemorytype=appdirectnotinterleaved

Keep it moving

The fundamental basis for nvdimm was addressed here, and a enablement for Intel NVDIMM on Ubuntu also gave the simple instructions to reproduce these functions. For more information about ndctl and ipmctl, the man page of them should be referred to understand the meaning behind each command. However, taking an advantage for persistent memory is from applications. In order to allow applications access persistent memory as memory-mapped files, the PMDK[7] libraries are provided. It expected that storage-based applications will be pushing this work forward in the future.

[1] https://nvdimm.wiki.kernel.org

[2] https://git.kernel.org/pub/scm/linux/kernel/git/nvdimm/nvdimm.git

[3] https://www.kernel.org/doc/Documentation/filesystems/dax.txt

[4] https://www.kernel.org/doc/Documentation/nvdimm/btt.txt

[5] https://github.com/pmem/ndctl

[6] https://github.com/intel/ipmctl

[7] http://pmem.io

Ubuntu

--

--

Woodrow Shen

Written by Woodrow Shen

5 Followers

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams