Installing SAP HANA, express edition with Docker - persistence on zfs

[lbe]

I tried to install the HANA Express Docker on a Debian 9 system where all storage is zfs. The creation of the docker failed when creating the persistence layer. See below for the actual output from the failure.

Suspecting that the problem was with lack of support for ZFS, I created a sparse image file on the zfs system, mounted it, changed the ownership of the mount point to hxeadm.sapsys (12000.79) and the docker creation completed.

The commands to create and mount the file are:

dd if=/dev/zero of=HANA.img bs=128k seek=819200 count=0
mkfs.ext4 HANA.img
mount HANA.img HANA
chown hxeadm.sapsys HANA

This creates and mounts a 100G sparse file, I chose bs based upon zfs recordsize on underlying filesystem

I suggest documentation to show what the filesystems are supported as persistence layers. I was unable to find this information from my searches.

Cheers,

lbe

Error Message on zfs:

Initializing persistence for first start of HANA (system db) ...
nameserver localhost:39001 not responding.
creating persistence ...
assign failed with persistence startup error. exception 1: no.2000004 (Basis/IO/FileAccess/impl/FileDescriptors.cpp:328)
Cannot open file ""/datavolume_0000.dat" ((closed, mode= RW, access= rw-rw-r--, flags= DIRECT|LOCK), factory= (root= "/hana/mounts/data/mnt00001/hdb00001/" (access= rw-rw-r--, flags= AUTOCREATE_DIRECTORY, usage= DATA, fs= UNKNOWN (zfs), config= (async_write_submit_active=on,async_write_submit_blocks=all,async_read_submit=on,num_submit_queues=1,num_completion_queues=1,size_kernel_io_queue=512,max_parallel_io_requests=64,min_submit_batch_size=16,max_submit_batch_size=64))", rc=22: Invalid argument
exception throw location:
1: 0x00007fd814b03b5b in FileAccess::FileAccessor::getAccessRights(ltt::basic_string<char, ltt::char_traits, ltt::integer_constant<64> > const&) const+0x1757 (libhdbbasis.so)
2: 0x00007fd814b263a3 in FileAccess::IoThreadHandler::~IoThreadHandler()+0x34f0 (libhdbbasis.so)
3: 0x00007fd814b268b8 in FileAccess::IoThreadHandler::~IoThreadHandler()+0x3a04 (libhdbbasis.so)
4: 0x00007fd814b331eb in FileAccess::DiskFullEvent::printEvent(ltt::basic_ostream<char, ltt::char_traits >&) const+0x5c97 (libhdbbasis.so)
5: 0x00007fd828e74936 in PageAccess::ReclaimDataVolumeTraceCallback::notifyFail(unsigned int, char const*)+0x8662 (libhdbdataaccess.so)
6: 0x00007fd828e76675 in PageAccess::DataVolume::createInstance(ltt::allocator&, Configuration::ConfigurationHandler&, Diagnose::StatisticsData const*, PageAccess::PageIO const&, PageAccess::ExtendedDataVolumeInfo const&, bool, bool)+0x141 (libhdbdataaccess.so)
7: 0x00007fd828e7e350 in PageAccess::DataVolumesSPI::forEachDataVolume(PageAccess::DataVolumeWriteFunctor&)+0xa00 (libhdbdataaccess.so)
8: 0x00007fd828e7f723 in PageAccess::DataVolumesSPI::forEachDataVolume(PageAccess::DataVolumeWriteFunctor&)+0x1dd0 (libhdbdataaccess.so)
9: 0x00007fd828e816d1 in PageAccess::DataVolumesSPI::forDataVolume(unsigned int, PageAccess::DataVolumeReadFunctor&) const+0xd30 (libhdbdataaccess.so)
10: 0x00007fd828e81d32 in PageAccess::DataVolumesSPI::createInstance(ltt::allocator&, Configuration::ConfigurationHandler&, Diagnose::StatisticsData const*, PageAccess::PageIO const&, PageAccess::ExtendedDataVolumesInfo const&)+0xb0 (libhdbdataaccess.so)
11: 0x00007fd828ec0345 in PageAccess::PageIOSPI::collectAndCheckPages(unsigned long&, unsigned long&, unsigned long*, unsigned long&, unsigned long*, unsigned long&, unsigned long*, ltt::vectorPageAccess::PhysPageNo, unsigned long&, unsigned long, ltt::vectorPageAccess::PhysPageNo) const+0xd1 (libhdbdataaccess.so)
12: 0x00007fd828ec400a in PageAccess::PageIO::createInstance(ltt::allocator&, Configuration::ConfigurationHandler&, Diagnose::StatisticsData const, PageAccess::ExtendedDataVolumesInfo const&)+0xb6 (libhdbdataaccess.so)
13: 0x00007fd828b5dfde in DataAccess::PersistenceManagerImpl::resetAfterIncompleteRecovery(DataAccess::SavepointVersion, bool)+0x169a (libhdbdataaccess.so)
14: 0x00007fd828b5ed71 in DataAccess::PersistenceManagerImpl::resetAfterIncompleteRecovery(DataAccess::SavepointVersion, bool)+0x2430 (libhdbdataaccess.so)
15: 0x00007fd828b5f12d in DataAccess::PersistenceManagerImpl::resetAfterIncompleteRecovery(DataAccess::SavepointVersion, bool)+0x27e9 (libhdbdataaccess.so)
16: 0x00007fd828b34f2b in DataAccess::PersistenceManager::create(ltt::refcounted_handleDataAccess::PersistenceConfiguration const&)+0x7 (libhdbdataaccess.so)
17: 0x00007fd82b676572 in PersistenceLayer::PersistenceSystem::initialize(NameServer::ServiceStartInfo const&, bool, PersistenceLayer::PERSISTENCE_MODE)+0x780 (libhdbpersistence.so)
18: 0x00007fd82b6b5b06 in PersistenceLayer::PersistenceFactory::initPersistence(PersistenceLayer::PERSISTENCE_MODE, DataAccess::TransactionCallback*, NameServer::ServiceStartInfo&, ltt::refcounted_handleTransactionManager::TransactionControlBlockFactory&, bool, ltt::smartptr_handleDataAccess::DisasterRecoverySecondaryCallback&)+0x3d2 (libhdbpersistence.so)
19: 0x00007fd83160065b in PersistenceController::startup(PersistenceLayer::PERSISTENCE_MODE, NameServer::ServiceStartInfo*, bool, DataAccess::TablePreloadWriteCallback*, DataAccess::TablePreloadReadCallback*, Backup::RecoverCbc_Federation*)+0x2157 (libhdblogger.so)
20: 0x00007fd830e99808 in NameServer::Topology::initPersistence(NameServer::ServiceStartInfo&, bool, bool, TREX_ERROR::TRexError*, bool, ltt_adp::basic_string<char, ltt::char_traits, ltt::integral_constant<bool, true> >, NameServer::ServiceStartInfo::RequestAction)+0x254 (libhdbns.so)
21: 0x00007fd830e9284a in NameServer::Topology::initPersistence(bool, bool, TREX_ERROR::TRexError*, bool, ltt_adp::basic_string<char, ltt::char_traits, ltt::integral_constant<bool, true> >, NameServer::ServiceStartInfo::RequestAction)+0x2e6 (libhdbns.so)
22: 0x000055fe12cc0314 in NameServer::HADRProviderManager::~HADRProviderManager()+0xb2c0 (hdbnsutil)
23: 0x000055fe12ccaf20 in config::IniFile::setValue(ltt_adp::basic_string<char, ltt::char_traits, ltt::integral_constant<bool, true> > const&, ltt_adp::basic_string<char, ltt::char_traits, ltt::integral_constant<bool, true> > const&, char const*)+0x79b0 (hdbnsutil)
24: 0x000055fe12cf9829 in NameServer::ServerInfo::operator=(NameServer::ServerInfo const&)+0x98f5 (hdbnsutil)
25: 0x000055fe12c59d20 in +0x0 (hdbnsutil)
26: 0x00007fd814da2034 in mainInitialPhase+0x2b40 (libhdbbasis.so)

stopping service...
error: Unknown persistence layer error
failed. trace file nameserver_localhost.00000.000.trc may contain more error details.

[aplata-sap]

We will add storage recommendations in the next release.

[nthu9280]

@lbe

Thanks for the hint on the ZFS.
I followed the above steps but still experiencing the issue. The minor difference on how I mounted

---

dd if=/dev/zero of=HANA.img bs=128k seek=1638400 count=0
mkfs.ext4 HANA.img
mount HANA.img /mnt/hxe1data
chown hxeadm.sapsys /mnt/hxe1data
chmod 666 /mnt/hxe1data

---

I'm using the latest release if that makes a difference
store/saplabs/hanaexpressxsa:2.00.033.00.20180925.2
The base OS is Ubuntu 18.04.1 LTS
Docker -
Server Version: 18.06.1-ce

Snippet of error log:

---

Change hardware key...
nameserver hxehost:39001 not responding.
Opening persistence ...
assign failed with persistence startup error. exception 1: no.2000004 (Basis/IO/FileAccess/impl/FileDescriptors.cpp:334)
Cannot open file ""/datavolume_0000.dat" ((closed, mode= RW, access= rw-rw-r--, flags= DIRECT|MUST_EXIST|LOCK), factory= (root= "/hana/shared/data/HXE/mnt00001/hdb00001/" (access= rw-rw-r--, flags= AUTOCREATE_DIRECTORY, usage= DATA, fs= UNKNOWN (zfs), config= (async_write_submit_active=on,async_write_submit_blocks=all,async_read_submit=on,num_submit_queues=1,num_completion_queues=1,size_kernel_io_queue=512,max_parallel_io_requests=64,min_submit_batch_size=16,max_submit_batch_size=64))", rc=22: Invalid argument
exception throw location:

 stopping service...
error: Unknown persistence layer error
failed. trace file nameserver_hxehost.00000.000.trc may contain more error details.
~~~

*****

[aplata-sap]

@TerryNing can you help?

[songrui01]

Fow HXE docker full image: hanaexpressxsa, we recommend the devicemapper storage driver .

https://docs.docker.com/storage/storagedriver/select-storage-driver/#docker-ee-and-cs-engine

Docker EE and CS-Engine
For Docker EE and CS-Engine, the definitive resource for which storage drivers are supported is the Product compatibility matrix. To get commercial support from Docker, you must use a supported configuration.

Docker CE
For Docker CE, only some configurations are tested, and your operating system’s kernel may not support every storage driver. In general, the following configurations work on recent versions of the Linux distribution:
Linux distribution
Recommended storage drivers
Docker CE on Ubuntu aufs, devicemapper, overlay2 (Ubuntu 14.04.4 or later, 16.04 or later), overlay, zfs, vfs
Docker CE on Debian aufs, devicemapper, overlay2 (Debian Stretch), overlay, vfs
Docker CE on CentOS devicemapper, vfs
Docker CE on Fedora devicemapper, overlay2 (Fedora 26 or later, experimental), overlay (experimental), vfs

The detail steps:
Docker installation for HXE.

1.Install docker CE.
https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-docker-ce-1

2.Stop Docker
$ sudo systemctl stop docker

3.Manage Storage System
If your host file system is xfs, you can recommend the storage driver through the devicemapper in Docker. Add -s devicemapper to the DOCKER_OPTS property in the docker folder.
The minimum recommended storage size is 50G. For example:
DOCKER_OPTS=“-s devicemapper —storage-opt dm.basesize=100G”
For Ubuntu and Debian, the DOCKER_OPTS property can be found at /etc/default/docker.
For Fedora, SuSE, and Centos, the DOCKER_OPTS property can be found at /etc/sysconfig/docker.

4.Increase Docker container size limit
sudo dockerd —storage-opt dm.basesize=100G

5.Edit /etc/docker/daemon.json. If it does not yet exist, create it. Assuming that the file was empty, add the following contents.
{
“storage-driver”: “devicemapper”
}

6.Edit the host /etc/sysctl.conf file
fs.file-max=20000000
fs.aio-max-nr=262144
vm.memory_failure_early_kill=1
vm.max_map_count=135217728
net.ipv4.ip_local_port_range=60000 65535
Verify the changes by running: sudo /sbin/sysctl -p

7.Start Docker
$ sudo systemctl start docker
Verify that the daemon is using the devicemapper storage driver. Use the docker info command and look for Storage Driver.
docker info |grep Base

8.ADD your docker user as sudo group
gpasswd -a your_user docker

Note: For HXE docker full images, we have to execute steps 3/4/5 (use devicemapper storage).

[nthu9280]

@songrui01

Thanks for the recapping the instructions.

I have a working docker install with ZFS and have been using for learning Hadoop sandbox, machine learning and other systems.
The HANA XSA docker image is failing to install.
Per docker's documentation:
https://docs.docker.com/storage/storagedriver/select-storage-driver/

recommended storage driver order of preference
zfs (if supported by underlying OS)
overlay2
devicemapper is last.

Guess I'll just create ext4 FS over zfs and move my /var/lib/docker.

[songrui01]

@nkthupili1,
SAP HANA, express edition with Docker didn't test on all storage so far, such as ZFS.
As i know, devicemapper works for both of HXE server only and full image.

[cocopapagal]

“storage-driver”: “devicemapper”

Hello,

Seems that devicemapper storage driver is deprecated. #1424
As a matter of fact, it cannot be used with the latest Docker CE Version 18.09. #452

Is there any solution for overlay2?
Thank you

[entmike]

Thank you @lbe for posting this. The HXE docker tutorial was not ever updated to talk about supported filesystems but I managed to Google my way to this page thanks to your post.

[DutchKevv]

It doesn't work on the newest docker version. Anyone found a solution?