Accountability Is Everything

One of the most important parts of any organization is the need for accountability. It’s not just for the Sales teams to be accountable for the amount of revenue they are generating for the company or the Marketing team to be accountable for the number of leads they are bringing to the organization. It’s paramount that everyone feels that they are accountable for their actions.

Many TechOps teams manage complex environments and with more people using configuration management (such as Chef, Puppet or CFEnging) there can be disastrous consequences for even the smallest of changes. At Sonian, we manage a large number of systems across four separate public clouds. That by nature creates a high level of complexity that can often be difficult for a single person to keep track of. Small changes can have big consequences and we need to mitigate that risk as much as we can (without slowing the high velocity of new code changes).

I’ve noticed that people who are held accountable for their actions are likely to exhibit the following traits:

1. They are inclusionary - Pairing and actively ask for feedback
2. They ask questions - Understating the “Why?” to get to the ideal solution
3. They understand risk, or at least will learn to understand the impact of their changes
4. They are cautious and analytical

Accountability can bring maturity to a growing organization, but it can also crush teams with overreaching process and procedure. They key is to bring just the right amount of process, slowly over time.

Another point to mention is that being cautious is not the same as being slow. You can have an organization who is cautious and still push changes with a high velocity. Building continuous integration into your workflow and expanding your test driven development are two of the many ways that can provide validation and mitigate risk.

Something very simple that we do at Sonian is to have another dev review before you push your changes. As part of our workflow we developed an application that helps to control our deploy of system code changes (roles, databags and cookbooks) to various Chef organizations. If you are curious to learn more of our Git/Chef workflow, you can check out a presentation I gave at a Chef Meetup this past summer. Within our deploy application, which we’ve named Mise en place, you can see the information that we care about when pushing new infrastructure code.

1. What Chef Organization (Dev,QA,Prod, etc…)
2. Who pushed the change (we use a knife plugin to basically push a git diff of changes)
3. What they want to push (We’ll show a Github compare link showing the diff between SHA’s)
4. Who approved the change

This application has changed the nature of how we push code out to our many chef organizations. It provides logging of actions so that we are able to track back a change if we are seeing an issue (which provides quicker remediation). And the most important part of what Mise does for us is to provide accountability and control before changes reach production. When you push your change to a “protected stack” you need to get an approval for that push from another person. That other person is now also accountable for the changes you are about to push. They should review the Git diff to validate what you are pushing will have the intended result (to the best of their ability).

This feature, having someone else “turn their key”, was influenced by a scene from War Games. In this scene, you find two officers in a nuclear missile silo moving through the procedure to launch their missiles. They both authenticate the message and the launch codes to check for validity. Finally they are both required to turn their key in order to launch.

Pushing the wrong code, or pushing a poorly written change could be the same as launching a nuclear missile at your infrastructure. Having that simple bit of accountability and review of your changes before they hit production could be the difference between a successful deploy or dealing with nuclear fallout.