Everything at Google Runs in Containers
by Carlos Sanchez on Jun 06, 2014
Google is using containers to run everything in their clusters, starting over 2 billion of them per week.
Joe Beda, senior staff software engineer at Google Cloud Platform, gave a presentation at Gluecon about how the company is using Linux container technologies, stating that everything at Google runs in a container. The result is that Google starts over 2 billion containers per week, over 3000 started per second, not counting long-running containers.
Google is using container technologies since 2004, releasing cgroups in 2006 and just last year created the project Let Me Contain That For You (lmctfy), the open source version of Google’s container stack, which provides Linux application containers replacing LXC. These containers allow for the isolation of resources used by multiple applications running on a single machine and gives the applications the impression of running exclusively on a machine. The applications may be container-aware and thus be able to create and manage their own subcontainers.
Rohit Jnagal, software engineer at Google, commented on lmctfy:
We have lmctfy managing all of Google's resource isolation needs since 2007. So far, it was mangled into other pieces of Google infrastructure. During a redesign, we were able to separate this layer out cleanly and thought it would be fun to put it out and give back.
He also describes the main differencies between lmctfy and LXC:
Resource management API: LXC API is built for namespace support and exports cgroup support almost transparently. With lmctfy, Google tries to provide an intent-based resource configuration without users having to understand the details of cgroups, whose API may be unstable and hard to deal with.
Priority: lmctfy is built to provide support for resource sharing and for overcommitting machines with batch workloads that can run when the machine is relatively idle. All applications specify a priority and latency requirements and lmctfy manages all cgroup details to honor the requirements for each task.
Programmatic interface: lmctfy is the lowest block of app management for Google's cloud. It is built to work with other tools and programs, and much better specified and stable for building more complicated toolchains above it.
Google is also integrating containers into the Google Cloud Platform. The node container manager, open sourced by Google, is a small Python agent designed to manage a group of Docker containers according to a YAML manifest. An open preview version to manage containers running on Google Compute Engine is already available, supporting container manifests and the creation of containers at startup. The stack is based on Debian 7 and includes the Docker runtime and the node container manager.
Docker will also be able to take advantage of lmctfy since there is already an early integration for the Docker lmctfy driver, thanks to the new ability to run containers in environments other than LXC included since Docker version 0.9. |
