Rootconf 2019 recap

Rootconf 2019 image from HasGeeks’ Flick stream

This year I had the opportunity to attend Rootconf 2019 in Bangalore, India from June 21st — 22nd. Rootconf is a community-driven two-track conference on DevOps, SRE, Infrastructure and Security. The two tracks are Security and related tutorials on day 1 and DevOps, SRE and Distributed Systems on day 2. There are other year-round editions in various cities in India like Pune, Delhi and Hyderabad. As far as the format of the conference goes, there are normal talks and also curated “Birds Of Feather” sessions for pre-selected topics. “Birds Of Feather” sessions comprise of like-minded people discussing a specific topic. This year featured a great list of speakers from various companies like Cred, Atlassian, Trusting Social, Aerospike, Moengage or Zapier.

Here are the highlights from a few talks and sessions that really stuck with me:

Shooting the trouble down to the Wireshark Lua plugin: by Shakthi Kannan

This hour and a half long talk was in the format of a hands-on workshop where the speaker first got us started with an introduction on how they use Wireshark to troubleshoot customer problems at Aerospike and how it is painful to analyze packets for not so common protocols. This was followed by a quick “show and tell” on Lua scripting and how Wireshark has built-in support for Lua. The major part of the session was a hands-on workshop on how you can build custom Wireshark plugins in Lua for packet dissection and network analysis. My key takeaway from this talk was the new knowledge of Wireshark plugins and the fact that I can extend Wireshark with custom plugins if needed.

Here are some resources if you find the topic interesting:

• Slides from the talk
• A Lua Wireshark plugin the speaker referred to
• Lua Wireshark dissectors

Deploying and Managing CSP — the Browser-side Firewall: by Lavakumar Kuppan

This was a pretty amazing talk about Content Security Policy (CSP). The speaker introduced us to what CSP is and how it can act as an effective browser-side firewall to mitigate different kinds of attacks like XSS, script injection, clickjacking, data exfiltration, etc. By introducing an HTTP response header ‘Content-Security-Policy’ it is possible to control and whitelist what domains can load or execute scripts in your web page in the browser. You can also control whether scripts could be inlined or eval’d. All violations from clients loading your web pages can be reported back to a backend that you manage, which can be hooked into some tool like Grafana for providing visibility. This talk explored various aspects of how you can control and manage CSP for your web applications from an infrastructure standpoint.

Here are some resources if you find the topic interesting:

• Watch the full talk on YouTube
• Online tool to generate CSP headers that the speaker referred to
• Platform to generate CSP for your website based on reports
• OSS csp server that you can run for your webpages to report CSP violations

Using Pod Security Policies to harden your Kubernetes cluster: by Suraj Deshmukh

This was a Kubernetes specific talk on the security track. This talk touched on the various attack vectors in a Kubernetes cluster and then dived into how Pod Security Policies (PSPs) can be used to harden our Kubernetes clusters. You can apply PSPs at the cluster level to control the security aspects of a pod spec like running in privileged mode, mounting host volumes, SELinux context, etc. PSPs gets evaluated by admission controllers. This feature is already supported on both EKS and GKE.

Here are some resources if you find the topic interesting:

• Watch the full talk on YouTube
• Kubernetes documentation on pod security policies
• A tool the speaker wrote to survey common images on Docker Hub running as root

SSH Certificates: A way to scale SSH access: by Pulkit Vaishnav

This talk about scaling ssh access was really interesting for me. At Bench, we provision short lived ephemeral credentials for database access with Vault database secrets engine and short lived OIDC auth tokens using Dex for our Kubernetes cluster access. My team mate at Bench published a great blog post on that topic a few months back if you are interested. Short lived ssh access could be something for us to look into for further hardening our infrastructure security. The speaker talked about how instead of distributing a users public key to the servers and depending on TOFU (trust on first use) for clients, we can use a certificate and public key issued by a CA server that we control and distribute that to the hosts. Similarly, for clients, the CA server can also generate an intermediate certificate by signing the public keys of the clients which can then be used for authentication. The intermediate client certs can have principals (like root or engineer for instance) to control access and can also be short lived. And if a cert gets compromised, we can always rotate the CA server certificate and key.

Here are some resources if you find the topic interesting:

• Watch the full talk on YouTube
• Netflix bless — a CA server as a lambda function mentioned by the speaker
• Bless client by Lyft

Functional programming and Nix for reproducible, immutable infrastructure: by Brian McKenna

This talk about creating immutable infrastructure with Nix was a real eye opener. The speaker talked about how the Atlassian team in Bangalore has been using Nix to build Docker images for their services and how the builds are bit-for-bit reproducible. The speaker touched on Nix features, package management and tooling. One important aspect mentioned in the talk was how easy it is because of Nix to track down which version of something as core and low level as a C library is running on a production service and might need patching. The nix-instantiate tool that generates the serialized representation of the build seems very handy in inspecting purity of the builds. Nix is now definitely on my todo list of things to explore.

Here are some resources if you find the topic interesting:

• Watch the full talk on YouTube
• Slides from the talk
• Nix — functional package manager

Site Reliability of Distributed Systems: by Piyush Verma

The talk started with a simple example application and slowly elaborated on how it can become really complex as we introduce things like high availability, reliability, idempotency, backups, restores, throttling, etc. End goal was to get people thinking of the various ways a system can fail, consider the cost of making services available and reliable and start asking the right questions while keeping business and customer needs in mind.

Here are some resources if you find the topic interesting:

• Watch the full talk on YouTube

Scale MySQL beyond limits with ProxySQL: by Ratnadeep Debnath

This talk was about how to scale database connections with ProxySQL. ProxySQL is a high performance SQL proxy with features like connection pooling, query caching, advanced routing, transaction awareness, etc. The crux of the talk was an use-case of ProxySQL at Zapier, how they have benchmarked it’s performance, how they are running the system in production and what are some pitfalls and things to watch for.

Here are some resources if you find the topic interesting:

• Watch the full talk on YouTube
• Repo for benchmarking proxySql shared by the speaker
• Sample ProxySQL configs from the speaker

Thanks to Bench for having such a great training and conference budget that helped me attend Rootconf. I got to meet a bunch of great fellow engineers, got some food for thought from the presentations, hallway discussions and “Birds Of Feather” sessions. For me, overall, it was a great experience for a couple of days there.

If you are interested in learning more about Bench Accounting or a career with our Engineering team, then please visit us at https://bench.co/careers/