Trip report, Tectonic 2016.

The Tectonic Summit was a conference in New York City in December 2016, hosted by CoreOS, with the theme Enterprise Kubernetes. Mohan Kartha and Edward Vielmetti went there as a guest of Packet, a bare-metal hosting provider in the city. This report details highlights of the presentations at the event as well as accounts from attendees of what they thought was missing from the presentation and discussions.


Kubernetes is an orchestration system that runs on top of a container system like Docker. It allows an enterprise or service provider to define a system that can scale up and down according to need. Version 1.5 of Kubernetes was announced during the event, and it has extended beyond the previous version by speeding up operations so that it can support larger clusters.

The Kubernetes project is relatively young, but it draws from the experience of Google’s “Borg” program. Version 1.0 came out about two years ago, and there have been tens of thousands of contributions to the codebase since then.

Kubernetes project management; OpenStack, Docker comparison

The Kubernetes project is run by the Cloud Native Computing Foundation, which is about 50% by weight Google and the rest other cloud providers like Amazon. There was some political discussion at the event where it was clear that Kubernetes proponents made an effort to distance themselves from the OpenStack Foundation. OpenStack was what people were doing a few years ago to solve a similar set of problems, and that project has floundered because of a combination of vendor-induced complexity and a set of designs that did not reflect actual operation practice. While the Kubernetes folks were all polite to OpenStack, they were clear that they did not want to be tarred with the same brush.

The other product positioning and political interest at the event was how various organizations were dealing with Docker. That company controls the container system that is being orchestrated by Kubernetes, but also has been clawing its way up the stack acquiring other organizations that do container management and monitoring and developing additional capacities beyond the container core. Docker Swarm competes with Kubernetes, in the same way that the rkt container engine competes with [containerd] provided by Docker. The Kubernetes team would really rather have a vendor-neutral, stable, drama free container engine, rather than one that mutates all the time.

Self-driving Kubernetes: Tectonic from CoreOS

The major product announcement at the event was from CoreOS, their namesake Tectonic product. The latest release of Tectonic is said to be “self-driving”, i.e. will upgrade the version of Kubernetes on a cluster automatically. Their observation was that people who run Kubernetes spend an inordinate amount of time outside of Kubernetes upgrading systems when a new version comes out, and that there was an opportunity to remove that burden by automatically handling upgrades using a strategy much like CoreOS’s own Container Linux which automagically installs new versions when they are available.

Tectonic is now available with a free 10-node license. This is a new pricing strategy aimed at capturing the trial and test user market and offering an easier self-service onramp to getting started with the system.

Sysdig, Kubernetes monitoring and debugging tool

Kubernetes monitoring was demonstrated by Sysdig, which has a product both for individual system monitoring and for container monitoring of Kubernetes and Docker systems. There is lots of complexity in the container world, with processes moving here and there and logical systems dispersed across physical machines. sysdig starts with a robust monitoring and introspection technology that can replace lots of one-shot Linux monitoring tools (ps, top, sysstat etc) and extends that tool function all the way up into Kubernetes.

Provided for a fee is Sysdig Cloud which puts a point-and-click GUI interface on Kubernetes networking so that you get the whole picture at whatever level of detail you want. Sysdig’s CEO also presented in Chicago at Docker Chicago.

Helm, Kubernetes provisioning tool

Helm was described as “like apt-get for Kubernetes”; it promises to make deployment of systems easier by allowing a prepackaged set of commands and configurations to be distributed through a central registry or produced for your own private use. Every system needs a package manager.

Case studies: Ticketmaster, Planet

Ticketmaster was the main customer case study provided. Their system architecture support a very large, complex, heterogeneous workload with very old legacy stuff in the core of it (an old Vax, with its brain pickled into an emulator). The ticketing business model is a “self-induced DDOS attack” with very bursty loads when tickets go on sale. They use Kubernetes to schedule containers so as to handle peak loads better.

Planet was a second case study, which does global daily hi res satellite imagery. Their previous operations environment deployed VMs with Ansible, which led to a relatively slow deployment process in which every image had crazy dependencies. Docker helps contain dependency interactions. Kubernetes helps scheduling and deployment of resources as they are needed, so they can spin up and spin down resources specific to a task and not have to guess what part of the world their users are interested in today.

Notably missing from the discussion was a comprehensive look at storage. People had seemed to have sorted out at least the basics of networking and compute distribution over these large workloads, but anyone who had large storage needs had worked out something specific and purpose-built to their environment.

I had a conversation with the CTO of Quantum who is the project lead for Rook which uses an embedded Ceph filesystem. Ceph is one of the file systems that has come out of the cluster world, and Rook is an attempt to cut it down to size and simplify it so that it can be easily embedded as container storage.

No one really wants to write a file system from scratch and use it in production that same year. The most mature file systems are decades old and have a long history of correctness and performance tweaks. Of all things to worry about people are largely unwilling to put production loads on untested storage.

Kubernetes progress: development, testing, deployment

What people are using Kubernetes for: mostly for testing. The average size of a cluster is less than 3 nodes, according to a Gartner analyst. Packet has seen uptake as a Kubernetes test environment - it’s cheaper to provision bare metal than AWS or VMs for Kubernetes, and performant enough for the test environment.

ARM in the data center.

We had a good lunch discussion of ARM processors in the data center. Question: is there any way to displace Intel’s dominance of data center computing? ARM has advantages at very low end due to low power consumption, and the 4-core ARM Raspberry Pi 3 is a popular testbed for home clustering but not suitable for data center use. Packet has a 96-core ARMv8 server that we have been involved in testing, and there was lots of interest in stories around that.

The challenges of ARM in the data center: Intel dominance on software side, some small amount of fragmentation in Unix flavors, immaturity of software ports to ARMv8 (even as compared to ARMv7), unconvincing economics. One ex-Transmeta guy was very pessimistic about displacing Intel because of Intel’s manufacturing lead. On the other hand a Khosla Ventures exec was bullish on Cavium, an ARM server core vendor, and thought they had a long way to grow. He was interested in a call back in 30 - 90 days with report on uptake.

Site visit:

Packet provides bare metal as a service, either in your data center or their own. The office was very busy and bursting at the seams, with people sitting everywhere and a steady stream of vendors and partners in evidence. Packet are very nice, prompt, pleasant people to deal with, and they are expanding to new markets (Japan) with new investment (Softbank) on new servers (ARMv8).

The hosting market was explained to me this way: some companies have OpEx to spend, so they spend it on AWS; others have CapEx to spend, so they are trying to build a data center operating platform that’s friendly to using on premises. Can you run your system on other people’s capital? The Packet founders have been through this before and seem to have built a culture that knows its market.

In conclusion

Good conference, and thanks to CoreOS for organizing the event and to Packet for inviting us to participate!