Proof of Concept: Docker with BoostFS

Docker: it’s where all the cool kids are playing these days (or so I’m told). It’s certainly got some good functionality and offers a lightweight approach to spinning up workloads compared to running a full virtual machine (VMware, Hyper-V, KVM or anything else, really, for that matter).

I recently repurposed one of my servers at home as a dedicated development/test server, giving me the opportunity to play around with Docker or anything else without occupying CPU cycles from my main VMware lab environment, and since I usually learn best by giving myself an actual task to perform, I thought, can I get BoostFS running in a Docker container?

My environment is:

• CentOS 7.6 Server as my Docker server
• CentOS Docker Container
• DDVE running DDOS 6.2.0.5
• BoostFS 1.3

Initially I plowed right ahead and hit an obvious challenge: BoostFS uses the FUSE library, and within a Docker container, you can’t just load kernel modules. So, how does one get around this?

The process for getting around this is the following:

1. Setup the Data Domain Mtree/Storage Unit as you would for BoostFS. (Here’s an example where I’d set it up previously.)
2. Add the fuse and fuse-libs package to your Docker server. (Make sure to activate it via a modprobe fuse, of course.)
3. Check out a CentOS docker image with permissions to access the FUSE device.
4. Deploy BoostFS within the Container
5. Setup the BoostFS configuration, and lockbox.
6. Mount the BoostFS storage unit from the DDVE to confirm it works.
7. Unmount the BoostFS storage unit.
8. Exit the Docker instance and save the image for easier access later.

Now, there’s some extra components to the above, but I’ll get to them as I go through. I’m going to assume if you wanted to read this, you’d be comfortable with items (1) and (2), so I’m jumping straight to item (3) for my walkthrough.

Getting a Docker Container Running

OK, first step is to get a Docker container running. Normally you’d d something like:

$ docker run -it centos

But in this case, we need to give our container a bit more access. Of course, I know this changes the security profile, but this is a proof of concept.

So, the command we run instead is:

pmdg@abydos 
 $ docker run -it --privileged --cap-add SYS_ADMIN --cap-add MKNOD --device /dev/fuse centos

Deploy BoostFS Within the Container

At this point you get a prompt inside your container, with the container ID as the hostname. The next step is to get the BoostFS package added to the container using the docker cp command. For me, that was:

pmdg@abydos 
 $ docker cp DDBoostFS-1.3.0.0-607940.rhel.x86_64.rpm b6ad7a179778:/tmp

You’ll want to install the DDBoostFS Plugin now, but Docker will still want the FUSE base package and library installed as well, so I fall back on the old yum localinstall technique for that, viz.:

root@b6ad7a179778 $ yum localinstall --nogpgcheck /tmp/DDBoostFS-1.3.0.0-607940.rhel.x86_64.rpm 
 Loaded plugins: fastestmirror, ovl
 Examining /tmp/DDBoostFS-1.3.0.0-607940.rhel.x86_64.rpm: ddboostfs-1.3.0.0-607940.x86_64
 Marking /tmp/DDBoostFS-1.3.0.0-607940.rhel.x86_64.rpm to be installed
 Resolving Dependencies
 --> Running transaction check
 ---> Package ddboostfs.x86_64 0:1.3.0.0-607940 will be installed
 --> Processing Dependency: fuse >= 2.8 for package: ddboostfs-1.3.0.0-607940.x86_64
 Determining fastest mirrors
...snip...

Installed:
   ddboostfs.x86_64 0:1.3.0.0-607940                                             

Dependency Installed:
   fuse.x86_64 0:2.9.2-11.el7           fuse-libs.x86_64 0:2.9.2-11.el7          
   which.x86_64 0:2.20-7.el7           

Complete!
 root@b6ad7a179778 $ 

Setup BoostFS Configuration and Lockbox

Now we need to create a BoostFS configuration file. (I’ve also added BoostFS’s executable path, /opt/emc/boostfs/bin, to my PATH.)

My container is going to write to a Data Domain called neutronium using a storage unit called BoostDocker, so the configuration file looks like the following:

root@b6ad7a179778 $ cat /opt/emc/boostfs/etc/boostfs.conf 
[global]
 data-domain-system=neutronium.turbamentis.int
 storage-unit=BoostDocker
 lockbox-path=/opt/emc/boostfs/lockbox/boostfs.lockbox
 log-enabled=true
 log-level=info
 log-dir=/opt/emc/boostfs/log
 log-file=output.log
 log-maxsize=100
 log-rotate-num=4

[/container/datadomain]
 data-domain-system=neutronium.turbamentis.int
 storage-unit=BoostDocker
 security=lockbox
 mtboost-enabled=true
 mtboost-threads=16
 max-connections=128

With the configuration file created, the next step is the lockbox:

root@b6ad7a179778 $ boostfs lockbox set -u boost -d neutronium.turbamentis.int -s BoostDocker
 Enter storage unit user password: 
 Enter storage unit user password again to confirm: 
 Lockbox entry set

OK, now we’re getting closer.

Mount the BoostFS Storage Unit

With a lockbox and configuration file setup, we’re right to create the mount-point, and mount the BoostFS filesystem. First, creating the mount point:

root@b6ad7a179778 $ mkdir -p /container/datadomain

Now, to get access to a deduplicating mount-point:

root@b6ad7a179778 $ boostfs mount /container/datadomain

 mount: Mounting neutronium.turbamentis.int:BoostDocker on /container/datadomain

We can see the mounted BoostFS filesystem via a “df” command:

root@b6ad7a179778 $ df -h
 Filesystem                      Size  Used Avail Use% Mounted on
 overlay                          50G  5.2G   45G  11% /
 tmpfs                            64M     0   64M   0% /dev
 tmpfs                           3.8G     0  3.8G   0% /sys/fs/cgroup
 /dev/mapper/centos_abydos-root   50G  5.2G   45G  11% /etc/hosts
 shm                              64M     0   64M   0% /dev/shm
 boostfs                         354G   54G  300G  16% /container/datadomain

A quick test of writing data to the BoostFS filesystem:

root@b6ad7a179778 $ du -hs /usr/share
 37M    /usr/share
 root@b6ad7a179778 $ tar cf /container/datadomain/share.tar /usr/share
 tar: Removing leading /' from member names
tar: Removing leading/' from hard link targets
 root@b6ad7a179778 $ ls -l /container/datadomain/share.tar
 -rw-r--r-- 1 root root 34007040 Mar  5 04:44 /container/datadomain/share.tar

Unmount the BoostFS Filesystem

To unmount the filesystem ahead of exiting the container, just execute:

root@b6ad7a179778 $ boostfs unmount /container/datadomain
root@b6ad7a179778 $ df -h
 Filesystem                      Size  Used Avail Use% Mounted on
 overlay                          50G  5.2G   45G  11% /
 tmpfs                            64M     0   64M   0% /dev
 tmpfs                           3.8G     0  3.8G   0% /sys/fs/cgroup
 /dev/mapper/centos_abydos-root   50G  5.2G   45G  11% /etc/hosts
 shm                              64M     0   64M   0% /dev/shm

Exit the Docker Image, Save the Container Template

OK, now we’re ready to exit the Docker image and save the container template. For me, that worked as follows:

root@b6ad7a179778 $ exit
 exit

pmdg@abydos
$ docker commit -m "Added FUSE & BoostFS" -a "Preston de Guise" b6ad7a179778 centos/boostfs
 sha256:96f67ea69027ae7d85904a0960f8266d4437ba4da60209f1a3c122bf000644ac

Using the BoostFS Container

Obviously the point of creating a container template is to be able to make use of it. To do this, we have to:

1. Use docker to create a new container with the same settings as before.
2. Delete the lockbox information and recreate it. Why? The lockbox information is tied to the docker container we’d spun up for the creation of the template. But our new docker instance will get a different container ID – a different hostname from the DD perspective, and we’ll need to start from scratch.
3. Mount the BoostFS storage unit.

So, here goes!

First, getting the docker container running based on our new centos/boostfs template:

pmdg@abydos
$ docker run -it --privileged --cap-add SYS_ADMIN --cap-add MKNOD --device /dev/fuse centos/boostfs

Within the container, we delete and recreate the lockbox information:

[root@8a2861cd001a /]# rm /opt/emc/boostfs/lockbox/*

[root@8a2861cd001a /]# boostfs lockbox set -u boost -d neutronium.turbamentis.int -s BoostDocker
 Enter storage unit user password: 
 Enter storage unit user password again to confirm: 
 Lockbox entry set

Now, we can mount the BoostFS filesystem and access whatever we’d previously saved:

[root@8a2861cd001a /]# boostfs mount /container/datadomain

 mount: Mounting neutronium.turbamentis.int:BoostDocker on /container/datadomain
 [root@8a2861cd001a /]# ls -l /container/datadomain
 total 2076
 -rw-r--r-- 1 root root 34007040 Mar  5 04:44 share.tar

Wrapping Up

I’m not pretending to know a lot about Docker with the above, but for me the next logical step in using Containers is to make sure when they do have state (and I know there’s a lot of arguments that they should, as much as possible, be stateless – but there’s going to be arguments that there’s always exceptions), you can do a storage and network efficient operation to protect that state data. One way to do that without going through a full agent/client configuration within something like Avamar or NetWorker (and dealing with obvious client hostname issues) is to use BoostFS instead, which is what I wanted to test out here.