Running unmodified Linux executables on OSv

From the beginning, OSv has been designed to implement a subset of Linux flavor of POSIX API. But until the release 0.54.0 most Linux applications had to be re-compiled from source as shared libraries or some, like Java, rely on OSv version of /usr/bin/java wrapper to run. This meant one could NOT run a Linux executable "as is". In other words, OSv has always been Linux-compatible at source level but not at binary level.

Starting with the release 0.54.0, it should be in general possible to run unmodified Linux position-independent executables (so-called "PIEs") and position-dependent executables "as-is" as long as they do not use "fork/execve" or other unsupported Linux API. It means that very often one can take a binary from Linux host and run it on OSv without having to locate the source code on the Internet and build it as a shared library. At this moment OSv still does not support statically-linked executables as described here.

So how can one run arbitrary Linux executable on OSv? In high-level it as simple as locating and adding the relevant executable file and any related files onto the filesystem that is part of an OSv image and running that image on the hypervisor of choice. Creating an OSv image requires fusing its kernel and application files together by either using Capstan or the Python-based build scripts. The former involves collecting all files to be added into a directory or using a template and is described in more detail here and there. The latter, which most of this Wiki focuses on, involves using fairly new shell script manifest_from_host.sh and build that consumes a "manifest" produced by former one and delegates to number of python scripts to build a final image.

The script manifest_from_host.sh in essence automates process of locating an executable file and any shared library files it depends on through DT_NEEDED section in the executable ELF and outputs the list of files in the "manifest" format described pretty well here. Some of the dependencies like ld-linux-x86-64.so.2, libc.so.6 or others as listed here are built-in into OSv kernel and get automatically filtered out. Also please note that manifest_from_host.sh has no way to know which non-binary files like configuration ones need to be added as well. Similarly it can not find any shared libraries that would be loaded through dlopen() call. Finally in theory one can produce manifest file by hand and feed it to build script as well.

Here is the usage information with examples of how manifest_from_host.sh can be used:

Produce manifest referencing files on the host filesystem

Usage: manifest_from_host.sh [options] <ELF file> | <directory path> [<subdirectory path>]

Options:
  -l              Look for a shared library
  -r              Resolve all SO dependencies in directory
  -R              Make guest root path match the host, otherwise assume '/'; applies with directory path input
  -h              Show this help output
  -w              Write output to ./build/last/append.manifest

Examples:
  ./scripts/manifest_from_host.sh ls                 # Create manifest for 'ls' executable
  ./scripts/manifest_from_host.sh -r /some/directory # Create manifest out of the files in the directory
  ./scripts/manifest_from_host.sh -l libz.so.1       # Create manifest for libz.so.1 library
  ./scripts/manifest_from_host.sh -w ls && \
  ./script/build --append-manifest                   # Create manifest for 'ls' executable

The manifest produced for ls utility would look like this:

# Position Dependent Executable
/ls: /bin/ls
# --------------------
# Dependencies
# --------------------
/usr/lib/libpcre2-8.so.0: /usr/lib/x86_64-linux-gnu/libpcre2-8.so.0
# --------------------

Normally the next step would be to feed the manifest file produced by manifest_from_host.sh and saved under build/{arch}/append.manifest to the build script. Typically the build script would receive a list of predefined modules or apps through the "image" parameter, but in this case we would pass --append-manifest parameter as a way to indicate to read files from the append.manifest and concatenate into final build/{arch}/usr.manifest. As you can see it is also possible to use build with image parameter to supply additional modules or apps. Finally one can manually craft append.manifest by hand.

Pleas also note that manifest_from_host.sh is used quite extensively to produce many predefined modules and apps. Some example include:

• module openssl
• module cli
• apps that end with -from-host

Building and running Linux applications from host has been quite extensively tested on on Ubuntu, Fedora and Alpine Linux with Docker.