DevOps implementations

In this blog I will go over the various DevOps implementations (or “levels”) I have encountered over the years. I will explain what the pros and cons of each implementation are and why I believe each level we look at is one step up from the previous one.

Introduction

There is no silver bullet for everything and I believe for software development this is also the case. So before looking at DevOps, I feel that there are situations where DevOps might not be beneficial to the project. The most important one reason for this is probably the size of the project.

If the (web) project is so small that you simply can pick any shared hosting plan and stick your app in it, by all means, go ahead.

On the other end of the spectrum, of you work in a multi billion dollar company with tens or even hundreds of teams, all building parts of a bigger application (e.g. Spotify), there’s probably going to be a platform team involved to streamline and maintain the infrastructure. I never had the pleasure of experiencing this scale though.

I am writing from experience with projects like well known nation wide brands like news papers, cellphone subscription providers, automotive services, that sort of stuff. Typically these applications are built by a single or a few teams, so anywhere from roughly 5 – 25 engineers.

Technically this is generally more than just a customer facing website, with things like API connections, backoffice software, caching layers, data processors with worker queues, etc. drawing in hundreds to tens of thousands of users per day.

So, any decently sized project should be able to benefit from this.

What is DevOps

Amazon explains this nicely on this page: DevOps is the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity. […] Under a DevOps model, development and operations teams are no longer “siloed.”

Practically this means breaking down the walls between development and operational teams, by mixing them and/or merging them together. You build it, you run it.

Level 0

Separation of dev and ops between companies

This has been the process for a very long time: the customer’s (web) application is being built either in-house or externally by e.g. a digital agency (the dev team from now on), and then is deployed at a managed hosting partner (the ops team from now on). The ops team takes care of the day to day operations and is the first contact when issues occur.

Even though both the dev and ops teams have the best intentions for the client, their goals are in direct conflict with each other: the ops team strives for stability, which means updating or otherwise modifying the platform as little as possible. Meanwhile the dev team wants to deliver as much value as possible, which means releasing the product as often as possible.

The reason behind this is simple: the dev team is driven by the customer’s wish to deliver as much value as possible, but do not experience the direct consequence of breaking / buggy / underperforming or insecure software. It is only the ops team who is on call during the night, who for this very reason rather not update at all, as to not introduce new bugs.

Despite this being a “DevOps implementation” in this blog post, don’t let it fool you, it is not. Trump clearly did a bad job because if he was ever looking for a wall to keep people separated, they don’t come any higher than this. However, I think we all have been here and it’s a great starting point for our this blog.

Pros:

• At least it works, I guess…

Cons:

• A common goal is missing.
• There’s misunderstanding and friction between dev and ops departments.
• Value is delivered slowly.
• Developers do not know the infra, and do not make optimal decisions.
• Not the best value for money for the party paying for all of this.

Level 1

Separation of dev and ops between teams

All companies and teams that I’ve worked with have run into the challenges above at one point. While some of the teams and companies sadly were in no position to do anything about this or just accepted it otherwise, some of them tried to improve on this.

One of the chosen DevOps implementations is insourcing SRE engineers to form an ops team within the company. This makes things a bit more flexible, now procedures can be simpler and from a company perspective, everyone can be assigned to chase the same outcome if need be.

Whilst this sounds great on paper, the various teams within the company still have different goals. The dev teams find the ops teams to be slow, strict and unflexible, and the ops teams find the dev teams to have little regard for quality control and stability, instead only pushing as much functionality to production as possible. In the end, the ops teams are probably still the ones on call if things go down of course.

Even though now one company is doing both dev and ops, it is still done by separate siloed teams. One teams builds it, one team runs it, and thus this is not DevOps by definition.

Pros:

• A bit more flexible than having the ops department at a different company.
• If need be, teams can come together and try to fix outages etc.
• More in-house knowledge about the platform on which the software runs.

Cons:

• It can be hard to aqcuire this knowledge within the organisation.
• A common goal is still missing between dev and ops teams.
• Misunderstanding and friction between dev and ops teams is still there.
• Unless actively cooperating, the dev teams still do not know the infra.
• Still not the best value for money for the party paying for all of this.

Level 2

Dev and ops done by different disciplines in a team

One step up from having separate dev and ops teams is throwing the teams in the blender. Every team now consists of a mix of developers and SRE’s (amongst others) and this team is solely responsible for building and running one or more projects. This can be scary at first because there is no more external party or team to hide behind if things go wrong.

At this point we are truly talking DevOps. You build it, you run it.

The teams have a shared (sprint) goal and figure out amongst themselves how to achieve this goal. The teams will also push for stability / maintenance instead of pure functionality, knowing they will be the ones getting called at night if they do not take this aspect of building software seriously.

Pros:

• The team is fully responsible for building and running the applications.
• There is a shared goal of delivering value for the money spent on the project.
• There is a shared responsibility to keep everything stable and running.
• The team has more in depth knowledge about the platform on which the software runs.

Cons:

• It can be hard to aqcuire this knowledge within the organisation.
• It can be scary at first. No more hiding behind an external ops team.
• It requires an organisational transformation.
• The possibility of SRE and development still not working together fully.

Level 3

Dev and ops done by a team of T-shaped engineers

T-shaped engineers are explained here, but the basic gist is: individuals who have deep knowledge and skills in a particular area of specialization, along with the desire and ability to make connections across different disciplines.

Teams consisting of such engineers with a mixed skillset of frontend development, backend development and operations, makes for the strongest teams in my experience. These teams can tackle any challenge that comes at them, and whilst not everyone might be evenly skilled in every area, everyone can pitch in at least on a basic level.

I regard this DevOps implementation as the holy grail.

Pros:

• The team is fully responsible for building and running the applications.
• There is a shared goal of delivering value for the money spent on the project.
• There is a shared responsibility to keep everything stable and running.
• The team has in depth knowledge on both the platform and the software.

Cons:

• Finding good T-shaped developers and form such teams can be hard.

Where to go from here

I’ve written down my own experience over the years, and the DevOps implementations I’ve encountered. This writeup has been challenged and disputed by various people, mostly SRE’s and delivery managers.

I am sure there are other ways of establishing a successful software development and delivery process out there, all I know is that at the very least embracing the DevOps culture is one of them.

It has been very successful for the teams and companies I’ve been working for and I whole-heartedly encourage every software developer to look into it.

If you are stuck in your current project / company or simply want to talk more about this, feel free to reach out to my company StevOps, and I help you out!