Musings

Musings of a distributed systems engineer working remotely in western Canada.

Infrastructure

Common Safety Pitfalls found by Jepsen

Learning about Jepsen and doing a survey of published analyses.

When Jepsen appeared in the industry several years ago our team at PingCAP couldn't have been happier. Finally, an approachable, understandable reliability testing framework along with an enjoyable and rigorous library of analyses of popular databases.

Jepsen is an umbrella term for the software library, collection of articles, and service of the same name. Together, the project pushes distributed systems to be more accurate in their claims and more thorough in their testing.

It's been indispensable for battle-testing TiDB and TiKV, our distributed databases, as we deploy them in some of the largest technology companies in the world.


Why Benchmarking Distributed Databases Is So Hard

Exploring the complexities of benchmarking distributed systems.

If you’re an avid reader of distributed systems news like I am, you’ve probably seen your share of benchmarks. You’ve also probably stopped taking them at face value. Unfortunately, benchmarks are hard to get right, and even more unfortunately, many articles touting benchmarks are actually benchmarketing, showcasing skewed outcomes to sell products.

So why do it at all? Let’s take a look at some of the motivations of benchmarking, the common tools, and discuss a few things to keep in mind when benchmarking.


An HTTPS Terminator Box

Over the last couple days at asquera we've been on a retreat at the Landhaus Fredenwalde. It's really beautiful out here and it's given me a chance to work on a few small projects which I've been wanting to explore for awhile now.

Anyways, yesterday I set up a system that uses Ansible, Let's Encrypt, nginx, and DigitalOcean to terminate HTTP and proxy requests to arbitrary hosts. The intended use case for this is to have Github Pages sites able to be dropped onto a custom domain that is SSL enabled, but there are many other use cases which I haven't experimented with (yet).

I was, primarily, interested in exploring using Ansible and DigitalOcean. It worked out quite well http://hoverbear.org/ is running on it at the moment!


New Roots part 5, Erecting Container Infrastructure

This is the fifth part of an ongoing series on configuring a new server. In our last post we discussed and configured some basic tools. For all intensive purposes, our 'root' system is complete. What we'll be doing now is building the infrastructure to run containers the way we want to.

Before we get around to setting things up, let's describe the what we're up to.


New Roots part 4, Making it 127.0.0.1

This is the fourth of a series on taking up residence on a new server. In our last post we talked about how to set up some basic network services like ssh and configure iptables, our firewall. In this post we'll talk about making your server feel like home.

There are a grand number of things we can do at this point to make our time on the machine enjoyable. Take time to evaluate your choices though. This 'root' host is going to be our 'control seat' so it's not going to be doing much more than orcestrating virtualized environments. Once we configure these environments (next post) we'll be using these for any sort of development, deployment, or experiments.

With 'root' our goal is to make a simple set of good, sharp tools to do what we need to do. These are tools like nvim which we installed earlier. If you were following the last host you may have installed mosh as well, which makes our ssh sessions safe from network changes and comfortable against latency.


New Roots part 3, Services & Hardening

This is the third in a series of posts about getting settled into a server. First we talked about choosing a server, then we talked about installing a base OS on a dedicated server. In this post we'll discuss configuring, securing, and hardening our server.

In our last post we left our new server in a very, very minimal state. Heck, we didn't even tell it it's own name! In this post we'll talk about configuration. Throughout this process we're going to try to keep things simple and tightly knit. Through most of this guide you'll need to be using sudo or acting as root.


New Roots part 2, On the Metal

This is the second post of a series on settling into new servers. The first was about choosing a server. This post is specifically targetted at newly acquired VPS and Dedicated servers. We'll talk about installing our chosen distribution, configuring its basics, and familiarizing ourselves with the new metal.


New Roots part 1, Choosing a Server

This is the first in a series of posts where we'll discuss the process of obtaining, setting up, and settling into a new headless server. Along the way we'll install Linux, configure firewalls and web servers, set up virtual machines, tinker with system knobs, explore automation tools, and generally have a great time.

In this post, We'll discuss things to keep in mind when searching for your own hosting, things to look for, and the differences between distinctions.

This series is not intended to be a comprehensive guide. These are just the notes and rambling of a hobbyist server admin who's loved tinkering for over a decade. I'm writing these both for my own future reference, and so my readers can suggest improvements! As always, comments are welcome via email.