Spacedfile

June 2, 2016

The Spacedfile was initially developed to provide support for Docker images. To date, building recipes from a Spacedfile has become the recommended way of building recipes.

Spacedecode can build recipes automatically by reading the instructions from a Spacedfile. A Spacedfile is a text document that contains all the commands a user could call on the command line to assemble a recipe. Using spaced recipe build users can create an automated build that executes several command-line instructions in succession.

This page describes the commands you can use in a Spacedfile.

Usage

The spaced recipe build command builds a recipe from a Spacedfile and a context. The build’s context is the files at a specified location PATH or URL. The PATH is a directory on your local filesystem. The URL is a the location of a Git repository.

A context is processed recursively. So, a PATH includes any subdirectories and the URL includes the repository and its submodules. A simple build command that uses the current directory as context:

$ spaced recipe build .
Uploading context to the SpaceD daemon: 33 kB
...

The build is run by the Spacedecode daemon, not by the CLI. The first thing a build process does is send the entire context (recursively) to the daemon. In most cases, it’s best to start with an empty directory as context and keep your Spacedfile in that directory. Add only the files needed for building the Spacedfile.

Warning: Do not use your root directory, /, as the PATH as it causes the build to transfer the entire contents of your hard drive to the Spacedecode daemon.

To use a file in the build context, the Spacedfile refers to the file specified in an instruction, for example, a COPY instruction.

Traditionally, the Spacedfile is called Spacedfile and located in the root of the context. You use the -f flag with spaced recipe build to point to a Spacedfile anywhere in your file system.

$ spaced recipe build -f /path/to/a/Spacedfile .

You can specify a namespace and the name to label the resulting recipe by passing the parameters --name (or -n) and --namespace (or -s):

$ spaced recipe build -s sithembiso -n myrecipe .

The Spacedecode daemon runs the instructions in the Spacedecode one-by-one, committing the result of each instruction and outputting the commit ID of each step. Once finished you will see the following:

...
Generating recipe...

Cleaning up...
Build was completed successfully

The Spacedecode daemon will automatically clean up the context you uploaded.

Note that each instruction is run independently, and causes a new layer to be created - so RUN cd /tmp will not have any effect on the next instructions.

Whenever possible, Spacedecode will re-use the cache, to accelerate the spaced recipe build process significantly. This is enabled by default, and you can disable it by passing --nocache or -x:

$ spaced recipe build -s sithembiso -n myrecipe .
Uploading context to the SpaceD daemon: 33 kB

Step 1: FROM debian:wheezy

Step 2: MAINTAINER Spacedecode <future@spacedecode.com>
----> From cache
----> 97b0aaff84f4dc091a1d0a0ce6b005e9
...

When you’re done with your build, you’re ready to look into pushing the recipe to Spacedecode.

Format

Here is the format of the Spacedfile:

# Comment
INSTRUCTION arguments

Spacedecode runs instructions in a Spacdfile in order. The first instruction must be `FROM` in order to specify the base recipe from which you are building.

Spacedecode treats lines that begin with # as a comment. Line continuation characters are not supported in comments.

Environment replacement

Environment variables (declared with the ENV statement) can also be used in certain instructions as variables to be interpreted by the Spacedfile. Escapes are also handled for including variable-like syntax into a statement literally.

Environment variables are notated in the Spacedfile either with $variable_name or ${variable_name}. They are treated equivalently and the brace syntax is typically used to address issues with variable names with no whitespace, like ${foo}_bar.

Example (parsed representation is displayed after the #):

FROM ubuntu
ENV foo /bar
WORKDIR ${foo}   # WORKDIR /bar
ADD . $foo       # ADD . /bar
COPY \$foo /quux # COPY $foo /quux

Environment variables are supported by the following list of instructions in the Dockerfile:

  • ADD
  • COPY
  • ENV
  • WORKDIR
  • VOLUME

Environment variable substitution will use the same value for each variable throughout the entire command. In other words, in this example:

ENV abc=hello
ENV abc=bye def=$abc
ENV ghi=$abc

will result in def having a value of hello, not bye. However, ghi will have a value of bye because it is not part of the same command that set abc to bye.

FROM

FROM <namespace>/<recipe>

Or

FROM <namespace>/<recipe>:<tag>

Or

FROM <recipe> # if it's an official recipe

The FROM instruction sets the base recipe for subsequent instructions. As such, a valid Spacedfile must have FROM as its first instruction. The <recipe> can be any recipe available either locally or remotely (as long as it is reachable by the current user).

  • FROM must be the first non-comment instruction in the Spacedfile.

  • The tag value is optional. If you omit it, the builder assumes a latest by default. The builder returns an error if it cannot match the tag value.

MAINTAINER

MAINTAINER <name> "<email>"

The MAINTAINER instruction allows you to set the Maintainer field of the generated recipe.

RUN

RUN has 2 forms:

  • RUN <command> (shell form, the command is run in a shell - /bin/sh -c)
  • RUN ["executable", "param1", "param2"] (exec form)

The RUN instruction will execute any commands in a new layer on top of the current recipe and commit the results.

The exec form makes it possible to avoid shell string munging, and to RUN commands using a base image that does not contain /bin/sh.

In the shell form you can use a \ (backslash) to continue a single RUN instruction onto the next line. For example, consider these two lines:

RUN /bin/bash -c 'source $HOME/.bashrc ;\
echo $HOME'

Together they are equivalent to this single line:

RUN /bin/bash -c 'source $HOME/.bashrc ; echo $HOME'

Note: To use a different shell, other than ‘/bin/sh’, use the exec form passing in the desired shell. For example, RUN ["/bin/bash", "-c", "echo hello"]

Note: The exec form is parsed as a JSON array, which means that you must use double-quotes (“) around words not single-quotes (‘).

Note: Unlike the shell form, the exec form does not invoke a command shell. This means that normal shell processing does not happen. For example, RUN [ "echo", "$HOME" ] will not do variable substitution on $HOME. If you want shell processing then either use the shell form or execute a shell directly, for example: RUN [ "sh", "-c", "echo $HOME" ].

Note: In the JSON form, it is necessary to escape backslashes. This is particularly relevant on Windows where the backslash is the path seperator. The following line would otherwise be treated as shell form due to not being valid JSON, and fail in an unexpected way: RUN ["c:\windows\system32\tasklist.exe"] The correct syntax for this example is: RUN ["c:\\windows\\system32\\tasklist.exe"]

The cache for RUN instructions isn’t invalidated automatically during the next build. The cache for an instruction like RUN apt-get dist-upgrade -y will be reused during the next build. The cache for RUN instructions can be invalidated by using the --no-cache flag, for example spaced recipe build --no-cache.

The cache for RUN instructions can be invalidated by ADD instructions. See below for details.

CMD

The CMD instruction has three forms:

  • CMD ["executable","param1","param2"] (exec form, this is the preferred form)
  • CMD ["param1","param2"] (as default parameters to ENTRYPOINT)
  • CMD command param1 param2 (shell form)

There can only be one CMD instruction in a Spacedfile. If you list more than one CMD then only the last CMD will take effect.

The main purpose of a CMD is to provide defaults for an executing vessel. These defaults can include an executable, or they can omit the executable, in which case you must specify an ENTRYPOINT instruction as well.

Note: If CMD is used to provide default arguments for the ENTRYPOINT instruction, both the CMD and ENTRYPOINT instructions should be specified with the JSON array format.

Note: The exec form is parsed as a JSON array, which means that you must use double-quotes (“) around words not single-quotes (‘).

When used in the shell or exec formats, the CMD instruction sets the command to be executed when running the recipe.

If you use the shell form of the CMD, then the <command> will execute in /bin/sh -c:

FROM ubuntu
CMD echo "This is a test." | wc -

If you want to run your <command> without a shell then you must express the command as a JSON array and give the full path to the executable. This array form is the preferred format of CMD. Any additional parameters must be individually expressed as strings in the array:

FROM ubuntu
CMD ["/usr/bin/wc","--help"]

If you would like your vessel to run the same executable every time, then you should consider using ENTRYPOINT in combination with CMD. See ENTRYPOINT.

Note: don’t confuse RUN with CMD. RUN actually runs a command and commits the result (during the build process); CMD does not execute anything at build time, but specifies the intended command for the recipe.

EXPOSE

EXPOSE <port> [<port>...]

The EXPOSE instruction informs Spacedecode that the vessel listens on the specified network ports at runtime.

ENV

ENV <key> <value>
ENV <key>=<value> ...

The ENV instruction sets the environment variable <key> to the value <value>. This value will be in the environment of all “descendant” Spacedfile commands and can be replaced inline in many as well.

The ENV instruction has two forms. The first form, ENV <key> <value>, will set a single variable to a value. The entire string after the first space will be treated as the <value> - including characters such as spaces and quotes.

The second form, ENV <key>=<value> ..., allows for multiple variables to be set at one time. Notice that the second form uses the equals sign (=) in the syntax, while the first form does not. Like command line parsing, quotes and backslashes can be used to include spaces within values.

For example:

ENV myName="John Doe" myDog=Rex\ The\ Dog \
    myCat=fluffy

and

ENV myName John Doe
ENV myDog Rex The Dog
ENV myCat fluffy

will yield the same net results in the final vessel, but the first form is preferred because it produces a single cache layer.

The environment variables set using ENV will persist when a vessel is run from the resulting recipe. You can view the values in the vessel.yml file.

Note: Environment persistence can cause unexpected side effects. For example, setting ENV DEBIAN_FRONTEND noninteractive may confuse apt-get users on a Debian-based recipe. To set a value for a single command, use RUN <key>=<value> <command>.

ADD

ADD has two forms:

  • ADD <src> <dest>
  • ADD ["<src>",... "<dest>"] (this form is required for paths containing whitespace)

The ADD instruction copies new files, directories or remote file URLs from <src> and adds them to the filesystem of the build vessel at the path <dest>.

Multiple <src> resource may be specified but if they are files or directories then they must be relative to the source directory that is being built (the context of the build).

The <dest> is an absolute path into which the source will be copied inside the destination container.

ADD test relativeDir/          # adds "test" to `WORKDIR`/relativeDir/
ADD test /absoluteDir/         # adds "test" to /absoluteDir/

All new files and directories are created with a UID and GID of 0.

Note: If your URL files are protected using authentication, you will need to use RUN wget, RUN curl or use another tool from within the vessel as the ADD instruction does not support authentication.

Note: The first encountered ADD instruction will invalidate the cache for all following instructions from the Spacedfile if the contents of <src> have changed. This includes invalidating the cache for RUN instructions.

ADD obeys the following rules:

  • The <src> path must be inside the context of the build; you cannot ADD ../something /something, because the first step of a spaced recipe build is to send the context directory (and subdirectories) to the spaced daemon.

  • If <src> is a URL and <dest> does not end with a trailing slash, then a file is downloaded from the URL and copied to <dest>.

  • If <src> is a URL and <dest> does end with a trailing slash, then the filename is inferred from the URL and the file is downloaded to <dest>/<filename>. For instance, ADD http://example.com/foobar / would create the file /foobar. The URL must have a nontrivial path so that an appropriate filename can be discovered in this case (http://example.com will not work).

  • If <src> is a directory, the entire contents of the directory are copied, including filesystem metadata.

Note: The directory itself is not copied, just its contents.

  • If <src> is a local tar archive in a recognized compression format (identity, gzip, bzip2 or xz) then it is unpacked as a directory. Resources from remote URLs are not decompressed. When a directory is copied or unpacked, it has the same behavior as tar -x: the result is the union of:

    1. Whatever existed at the destination path and
    2. The contents of the source tree, with conflicts resolved in favor of “2.” on a file-by-file basis.

Note: Whether a file is identified as a recognized compression format or not is done solely based on the contents of the file, not the name of the file. For example, if an empty file happens to end with .tar.gz this will not be recognized as a compressed file and will not generate any kind of decompression error message, rather the file will simply be copied to the destination.

  • If <src> is any other kind of file, it is copied individually along with its metadata. In this case, if <dest> ends with a trailing slash /, it will be considered a directory and the contents of <src> will be written at <dest>/base(<src>).

  • If <dest> does not end with a trailing slash, it will be considered a regular file and the contents of <src> will be written at <dest>.

  • If <dest> doesn’t exist, it is created along with all missing directories in its path.

COPY

COPY has two forms:

  • COPY <src> <dest>
  • COPY ["<src>",... "<dest>"] (this form is required for paths containing whitespace)

The COPY instruction copies new files or directories from <src> and adds them to the filesystem of the vessel at the path <dest>.

Multiple <src> resource may be specified but they must be relative to the source directory that is being built (the context of the build).

The <dest> is an absolute path, or a path relative to WORKDIR, into which the source will be copied inside the destination vessel.

COPY test relativeDir/   # adds "test" to `WORKDIR`/relativeDir/
COPY test /absoluteDir/  # adds "test" to /absoluteDir/

All new files and directories are created with a UID and GID of 0.

COPY obeys the following rules:

  • The <src> path must be inside the context of the build; you cannot COPY ../something /something, because the first step of a spaced recipe build is to send the context directory (and subdirectories) to the docker daemon.

  • If <src> is a directory, the entire contents of the directory are copied, including filesystem metadata.

Note: The directory itself is not copied, just its contents.

  • If <src> is any other kind of file, it is copied individually along with its metadata. In this case, if <dest> ends with a trailing slash /, it will be considered a directory and the contents of <src> will be written at <dest>/base(<src>).

  • If multiple <src> resources are specified, either directly or due to the use of a wildcard, then <dest> must be a directory, and it must end with a slash /.

  • If <dest> does not end with a trailing slash, it will be considered a regular file and the contents of <src> will be written at <dest>.

  • If <dest> doesn’t exist, it is created along with all missing directories in its path.

ENTRYPOINT

ENTRYPOINT has two forms:

  • ENTRYPOINT ["executable", "param1", "param2"] (exec form, preferred)
  • ENTRYPOINT command param1 param2 (shell form)

An ENTRYPOINT allows you to configure a vessel to have a default command.

Exec form ENTRYPOINT example

You can use the exec form of ENTRYPOINT to set fairly stable default commands and arguments and then use either form of CMD to set additional defaults that are more likely to be changed.

FROM ubuntu
ENTRYPOINT ["top", "-b"]
CMD ["-c"]

When you run the vessel, you will actually be running top -b -c as the main process.

To examine the result further, you can use spaced vessel exec:

$ spaced vessel exec <vessel_id> 'ps aux'
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  2.6  0.1  19752  2352 ?        Ss+  08:24   0:00 top -b -H
root         7  0.0  0.1  15572  2164 ?        R+   08:25   0:00 ps aux

And you can gracefully request top to shut down using spaced vessel stop <vessel_id>.

VOLUME

VOLUME ["/data"]

The VOLUME instruction creates a mount point with the specified name and marks it as holding externally mounted volumes from native host. The value can be a JSON array, VOLUME ["/var/log/"], or a plain string with multiple arguments, such as VOLUME /var/log or VOLUME /var/log /var/db.

The spaced vessel start command initializes the newly created volume with any data that exists at the specified location within the base image. For example, consider the following Spacedfile snippet:

FROM ubuntu
RUN mkdir /myvol
RUN echo "hello world" > /myvol/greeting
VOLUME /myvol

This Spacedfile results in a recipe that causes spaced vessel start, to create a new mount point at /myvol and copy the greeting file into the newly created volume.

Note: The list is parsed as a JSON array, which means that you must use double-quotes (“) around words not single-quotes (‘).

WORKDIR

WORKDIR /path/to/workdir

The WORKDIR instruction sets the working directory for any RUN, CMD, ENTRYPOINT, COPY and ADD instructions that follow it in the Spacedfile.

It can be used multiple times in the one Spacedfile. If a relative path is provided, it will be relative to the path of the previous WORKDIR instruction. For example:

WORKDIR /a
WORKDIR b
WORKDIR c
RUN pwd

The output of the final pwd command in this Spacedfile would be /a/b/c.

The WORKDIR instruction can resolve environment variables previously set using ENV. You can only use environment variables explicitly set in the Dockerfile. For example:

ENV DIRPATH /path
WORKDIR $DIRPATH/$DIRNAME
RUN pwd

The output of the final pwd command in this Dockerfile would be /path/$DIRNAME



comments powered by Disqus