How Threat Stack Does DevOps (Part I): Best Practices in the Wild

This was originally posted on the Threat Stack blog - added here for continuity.

As Senior Director of Operations at Threat Stack, I am repeatedly asked one question by our customers: “How does Threat Stack ‘do’ DevOps?”

One of my long-time pet peeves has been the abuse of the term “DevOps.” You can be a DevOps engineer, you can be a Director of DevOps, you can buy DevOps tools. But when people ask me “How does Threat Stack ‘do’ DevOps?”, I imagine them saying “How do you run Technical Operations?” See, it’s my belief that people often struggle at implementing DevOps because they don’t understand the complexity of technical operations. By this I mean managing the complexity of cloud environments, distributed systems, open source and home-built applications — and engineering them all for uptime and availability for customers. This is the crux of what it means to do DevOps well.

What’s the DevOps Magic Bullet?

Many teams we speak with are part of growing organizations that are struggling to meet customer demand. At the same time, they’re trying to scale their technical operations teams to accommodate organizational growth and the increased complexity of technology.

Threat Stack has managed ever-increasing amounts of scale since we launched in 2014, growing over the years to ingest many billions of events per day from our customers — all with a small technical operations team — until recently, just two people. So it makes sense that a customer would want to know how we are using and benefiting from DevOps at our own organization.

The bad news? There is no magic bullet for doing DevOps well. Rather, we believe it requires a holistic and cross-organizational focus on three major areas:

• Engineering for rapid change
• Measuring and optimizing system health
• Making engineers accountable

But Wait: What is DevOps?

Before we dive into these three areas, we need to take a step back and define DevOps. If you ask 10 people to define the term, you’ll likely get 15 different answers. Somehow it’s 2018, but we still can’t agree on what DevOps is. The only thing that we can seem to agree on is that everyone needs to be doing it. Unfortunately, many companies want to reap the benefits of DevOps without making any changes to their organizations.

That’s why you see the term bandied about willy-nilly. You’ll see “DevOps teams,” “DevOps tools,” and “Directors of DevOps” at companies. If your brand-new DevOps team is using DevOps tools to get their work done, then you’ve achieved DevOps, right? Unfortunately, the only accomplishment is likely a new organizational silo: a team that is ignoring the rest of the company, still entrenched in their old ways.

Likewise, you can’t adopt DevOps by plugging in a couple DevOps tools. Buying a shiny new tool without changing your processes is likely going to end with zero real change in your organization. You also can’t keep using the same old tools if you want to adopt DevOps for real. Attempting to force antiquated tools into a cloud-native world will result in unnecessary pain and suffering for your teams. You need to change both how you get work done as well as the tools that you use to get that work done.

So if tools and titles won’t cut it, how does Threat Stack define DevOps? To me, DevOps is a cultural and professional movement. DevOps is a way for everyone in the engineering organization — developers and ops pros alike — to work towards a shared goal. At its core, DevOps exists to empower teams to meet the growing demands of a business.

DevOps is Essential for Change

If you want to succeed as a business, change is essential. This quote from W. Edwards Deming makes the point very clear:

“It is not necessary to change. Survival is not mandatory.” - W. Edwards Deming

In other words, if you want to stay ahead of your competitors and your customers’ needs, you must change, grow, and scale your technology. Change is how you build new features in your product. Change is how you fix bugs. Change is how you can grow as a business. The difference between a healthy business and a failing one is often how well they each adapt to change. There is always, inherently, some risk that comes with making changes. DevOps is a way to gain assurance and reduce risk by building in collaboration along the way.

To achieve change while minimizing risk, Threat Stack has built a culture over the last four years that has gotten us to where we are today, focused on the three areas I mentioned at the beginning of this post:

• Engineering for rapid change
• Measuring and optimizing system health
• Giving engineers responsibility for their code

You can call what we do DevOps or not, but the essential goal of what we are doing is enabling rapid and positive change while minimizing organizational risk.

The One Weird Trick to Doing DevOps Right

Everyone wants the one weird trick to make DevOps “happen” in their organization. The problem is that there is no one thing you can change and suddenly be “doing DevOps.” Cultural and organizational change is hard work, and anyone who is trying to sell you one weird trick is likely inexperienced or uninformed.

I joined Threat Stack almost four years ago when it had a near-greenfield environment. We were able to implement and embed DevOps best practices from day one, without having to deal with legacy systems or major cultural changes to an already-functioning organization. Yet, even under ideal conditions, successfully implementing DevOps still took active effort across the whole organization. Of course, there are still plenty of places we can improve the way we get work done, and how we provide more value to our customers. But we’ve learned a lot along the way, and we’re excited to share some of those learnings with you.

Over the course of several posts, I will share my experiences in each of the three key areas I mentioned. I’ll provide a glimpse into what it looks like when things go wrong — and when they go right. I’ll pull back the curtain and share how we do things at Threat Stack. Finally, I’ll offer tips and practical advice for applying the three principles above to your organization, so that you, too, can do DevOps well. I hope you’ll follow along!