Managing your runtime environments

When klee run is executed, a container is created and a process is started within that container. The process runs in an isolated environment with its own file system, its own networking, and its own isolated process tree separate from the host.

In FreeBSD terminolgy, the process is ‘jailed’ (using the jail kernel functionalty) and it can be managed using native FreeBSD tooling. Some of these is being used by Kleene under the hood, but some of the tools can be relevant to use alongside Kleene as well.

The degree of isolation can be adjusted in many ways, and will be discussed in the following sections.

Creating and running containers

The most common way of using containers with the client is to use the command klee run. This command combines klee container create and klee container start. Therefore, running

$ klee container create FreeBSD echo Hello!
433b69dfc748
$ klee container start 433b
created execution instance 1671b5124e42
Hello!

executable 1671b5124e42 and its container exited with exit-code 0

and

$ klee run FreeBSD:testing echo Hello!
24644efab9a3
created execution instance f2f4d2ef6c26
Hello!

executable f2f4d2ef6c26 and its container exited with exit-code 0

are equivalent. A few things to notice with these two examples:

• In both cases, Kleene generated a unique ID for the container when it was created, and printed it to the user on the first line. Since no name was set with --name, Kleene autogenerated one for the container, but did not print it. Run klee lsc -a to see the names of the containers.

• Kleene also created an execution instance when the container is started. An execution instance is a ephemeral object in Kleene that represents a spawned process. A running container can have more than one execution instance and the user can create more with Klee using klee exec.

• When starting the container manually, it is enough to refer to the container using an initial segment of the ID, as long as it is unique across all container ID’s. You can also refer to a container by it’s name, however, then the entire name must be used.

• Lastly, the containers have connectivity using the host network driver, since no network driver was specified (with --network-driver/-l) when the containers were created. It is recommended to use networks instead, and an easy way to get started with networks is by going through the Get started guide. Many more details are given in the networking section.

Detached vs. foreground

By default, a container runs in the foreground and relays STDOUT and STDERR from the process to the terminal:

$ klee run FreeBSD:test echo Hello!
8752dee80656
created execution instance 7726cb9f5299
Hello!

executable 7726cb9f5299 and its container exited with exit-code 0

However, it can also run in the background using the --detach flag:

$ klee run --detach FreeBSD:test echo Hello!
4a687b420ed
created execution instance acba96f0411c

Here, Kleene started the container, and exits back to the terminal. In this case, the container stops immediatly after echo has printed ‘Hello!’ but if it was a long-running process it would continue to run in the background. This is exemplified in the following.

Running containers in the foreground

When running containers in the foreground, klee run starts the process and exits when the process does.

$ klee run FreeBSD:testing
a33acc099268
created execution instance 94455a51098c
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
32-bit compatibility ldconfig path: /usr/lib32
Updating motd:.
Creating and/or trimming log files.
Clearing /tmp (X related).
Updating /var/run/os-release done.
Starting syslogd.
/etc/rc: WARNING: failed to start syslogd
Starting sendmail_submit.
554 5.3.0 host "localhost" unknown: Protocol not supported
/etc/rc: WARNING: failed to start sendmail_submit
Starting sendmail_msp_queue.
Starting cron.

Tue Feb 13 13:04:21 UTC 2024

94455a51098c has exited with exit-code 0

Notice the last line compared to the previous examples: It only says that the execution instance has exited, and there is no mention of the container because it is stille running. Check with klee lsc. If the process that startede the container exists but it has spawned other processes that are still running, the container will continue to run in the background.

Note

When running a containerized process in the foreground and exiting Klee with ctrl+c, Kleened responds by terminating the jail(8) process and not any subprocesses. Therefore, the jail might still be running even though the attached session is closed. For instance, klee exec my-container /bin/sh -c "nc -l 4000" keeps running after Klee exits. Shut it down by killing the proces with kill from either host/container or, alternatively, stop the container entirely with klee stop my-container.

Accessing a running container

Since containers usually have a full-blown userland they can be accessed and managed with many of the same tools as the host, albeit in a restricted environment.

There are two different ways of doing this. Either by using klee or using FreeBSD’s native tool jexec(8).

Using klee

Klee’s commands for starting containers/execution instances, such as klee run and klee exec supports interactive sessions:

• With the -i/--interactive flag, Kleene redirects terminal STDIN to STDIN of the containerized process.

• With the -t/--tty flag Kleene allocates a pseudo-TTY which is needed in order for certain interactive applications such as sh, bash etc. to work properly.

For interactive processes (like a shell), use -i -t together in order to allocate a tty for the container process. -i -t is usually written -it.

The advantage of using the Kleene client is that it wraps the interactive session with relevant information such container/execution instance ID’s an whether the container terminated or just the execution instance. It can also run on a remote machine instead of the Kleene host. It is still experiemental, however, and it communicates through a websocket, which incurs some overhead.

Using jexec

FreeBSD comes with its own tool for executing commands inside a container/jail. It is straightforward to use and it works seamlessly with Kleene containers. You can use the JID or the (entire) container ID to specify the container that the command should run in. See jexec(8) for details.

The advantage of using jexec instead of Klee is that it is simple and fast (written in C). The drawback is that jexec only works on the host.

Image identification

When containers are created, they are created from an image which can be specified using this compressed form:

IMAGE_ID|IMAGE_NAME[:tag][@snapshot]

which is explained in detail in the documentation of klee container create. It is also possible to create images from build snapshots as described here, which can be useful during image development.

Overriding image defaults

An image can be equipped with parameters that also affects the runtime environment, i.e., the container. These can be overwritten when creating/running containers.

For instance, with klee run [OPTIONS] IMAGE [COMMAND]... the Dockerfile instructions can be overwritten by Klee in the following way:

In order to see what default values comes with the image, use klee image inspect.