At litl we use Docker images to package and deploy our Room for More services, using our Galaxy deployment platform. This week I spent some time looking into how we might reduce the size of our images and speed up container deployments.
Most of our services are in Go, and thanks to the fact that compiled Go binaries are mostly-statically linked by default, it’s possible to create containers with very few files within. It’s surely possible to use these techniques to create tighter containers for other languages that need more runtime support, but for this post I’m only focusing on Go apps.
The old way
We built images in a very traditional way, using a base image built on top of Ubuntu with Go 1.4.2 installed. For my examples I’ll use something similar.
Here’s a Dockerfile:
FROM golang:1.4.2
EXPOSE 1717
RUN go get github.com/joeshaw/qotd
# Don't run network servers as root in Docker
USER nobody
CMD qotd
The golang:1.4.2 base image is built on top of Debian Jessie. Let’s
build this bad boy and see how big it is.
$ docker build -t qotd .
...
Successfully built ae761b93e656
$ docker images qotd
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
qotd latest ae761b93e656 3 minutes ago 520.3 MB
Yikes. Half a gigabyte. Ok, what leads us to a container this size?
$ docker history qotd
IMAGE CREATED BY SIZE
ae761b93e656 /bin/sh -c #(nop) CMD ["/bin/sh" "-c" "qotd"] 0 B
b77d0ca3c501 /bin/sh -c #(nop) USER [nobody] 0 B
a4b2a01d3e42 /bin/sh -c go get github.com/joeshaw/qotd 3.021 MB
c24802660bfa /bin/sh -c #(nop) EXPOSE 1717/tcp 0 B
124e2127157f /bin/sh -c #(nop) COPY file:56695ddefe9b0bd83 2.481 kB
69c177f0c117 /bin/sh -c #(nop) WORKDIR /go 0 B
141b650c3281 /bin/sh -c #(nop) ENV PATH=/go/bin:/usr/src/g 0 B
8fb45e60e014 /bin/sh -c #(nop) ENV GOPATH=/go 0 B
63e9d2557cd7 /bin/sh -c mkdir -p /go/src /go/bin && chmod 0 B
b279b4aae826 /bin/sh -c #(nop) ENV PATH=/usr/src/go/bin:/u 0 B
d86979befb72 /bin/sh -c cd /usr/src/go/src && ./make.bash 97.4 MB
8ddc08289e1a /bin/sh -c curl -sSL https://golang.org/dl/go 39.69 MB
8d38711ccc0d /bin/sh -c #(nop) ENV GOLANG_VERSION=1.4.2 0 B
0f5121dd42a6 /bin/sh -c apt-get update && apt-get install 88.32 MB
607e965985c1 /bin/sh -c apt-get update && apt-get install 122.3 MB
1ff9f26f09fb /bin/sh -c apt-get update && apt-get install 44.36 MB
9a61b6b1315e /bin/sh -c #(nop) CMD ["/bin/bash"] 0 B
902b87aaaec9 /bin/sh -c #(nop) ADD file:e1dd18493a216ecd0c 125.2 MB
This is not a very lean container, with a lot of intermediate layers. To reduce the size of our containers, we did two additional steps:
(1) Every repo has a clean.sh script that is run inside the
container after it is initially built. Here’s part of a script for
one of our Ubuntu-based Go images:
apt-get purge -y software-properties-common byobu curl git htop man unzip vim \
python-dev python-pip python-virtualenv python-dev python-pip python-virtualenv \
python2.7 python2.7 libpython2.7-stdlib:amd64 libpython2.7-minimal:amd64 \
libgcc-4.8-dev:amd64 cpp-4.8 libruby1.9.1 perl-modules vim-runtime \
vim-common vim-tiny libpython3.4-stdlib:amd64 python3.4-minimal xkb-data \
xml-core libx11-data fonts-dejavu-core groff-base eject python3 locales \
python-software-properties supervisor git-core make wget cmake gcc bzr mercurial \
libglib2.0-0:amd64 libxml2:amd64
apt-get clean autoclean
apt-get autoremove -y
rm -rf /usr/local/go
rm -rf /usr/local/go1.*.linux-amd64.tar.gz
rm -rf /var/lib/{apt,dpkg,cache,log}/
rm -rf /var/{cache,log}
(2) We run Jason Wilder’s excellent
docker-squash
tool. It is especially helpful when combined with the clean.sh
script above.
These steps are time intensive. Cleaning and squashing take minutes and dominate the overall build and deploy time.
In the end, we have built a mostly-statically linked Go binary sitting alongside an entire Debian or Ubuntu operating system. We can do better.
Separating containers for building and running
There have been a handful of good blog posts about how to do this in the past, including one by Atlassian this week. Here’s another one from Xebia, and another from Codeship.
However, all these posts focus on building a completely static Go
binary. This means you eschew cgo by setting CGO_ENABLED=0 and
the benefits that go along with it. On OS X, you lose access to the
system’s SSL root CA certificates. On Linux, user.Current() from
the os/user package no longer works. And in both cases you must use
the Go DNS resolver rather than the one provided by the operating
system. If you are not testing your application with CGO_ENABLED=0
prior to building a Docker container with it then you are not testing
the code you ship.
We can use a few purpose-built base Docker images and the tricks from Jamie McCrindle’s Dockerception to build two separate Docker containers: one larger container to build our software and another smaller one to run it.
The builder
We create a Dockerfile.build, which is responsible for initializing
the build environment and building the software:
FROM golang:1.4.2
RUN go get github.com/joeshaw/qotd
COPY / Dockerfile.run
# This command outputs a tarball which can be piped into
# `docker build -f Dockerfile.run -`
CMD tar -cf - -C / Dockerfile.run -C $GOPATH/bin qotd
This container, when run, will output a tarball to standard out,
containing only our qotd binary and Dockerfile.run, used to build
the runner.
Dynamically linked binary
Notice that we did not set CGO_ENABLED=0 here, so our binary is
still dynamically linked against GNU libc:
$ ldd $GOPATH/bin/qotd
linux-vdso.so.1 (0x00007ffea6b8a000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f6e76e50000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6e76aa7000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6e7706d000)
We need to run this binary in an environment that has glibc
available to us. That means we cannot use stock BusyBox (which uses
uClibc) or Alpine (which uses musl). However, the BusyBox
distribution that ships with Ubuntu is linked against glibc, and
that’ll be the foundation for our running container.
The busybox:ubuntu-14.04 image only has a root user, but you should
never run network-facing servers as root, even in a container. Use
my joeshaw/busybox-nonroot
image
— which adds a nobody user with UID 1 — instead.
The runner
Now we create a Dockerfile.run, which is responsible for creating
the environment in which to run our app:
FROM joeshaw/busybox-nonroot
EXPOSE 1717
COPY qotd /bin/qotd
USER nobody
CMD qotd
Putting them together
First, create the builder image:
docker build -t qotd-builder -f Dockerfile.build .
Next, run the builder container, piping its output into the creation of the runner container:
docker run --rm qotd-builder | docker build -t qotd -f Dockerfile.run -
Now we have a qotd container which has the basic BusyBox
environment, plus our qotd binary. The size?
$ docker images qotd
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
qotd latest 92e7def8f105 3 minutes ago 8.611 MB
Under 9 MB. Much improved. Better still, it doesn’t require squashing, which saves us a lot of time.
Conclusion
In this example, we were able to go from a 500 MB image built from
golang:1.4.2 and containing a whole Debian installation down to a 9
MB image of just BusyBox and our binary. That’s a 98% reduction in
size.
For one of our real services at litl, we reduced the image size from
300 MB (squashed) to 25 MB and the time to build and deploy the
container from 8 minutes to 2. That time is now dominated by building
the container and software, and not by cleaning and squashing the
resulting image. We didn’t have to give up on using cgo and
glibc, as some of its features are essential to us. If you’re using
Docker to deploy services written in Go, this approach can save you a
lot of time and disk space. Good luck!