DevOps Flow
Several months ago, I found myself frustrated. I published a somewhat egregious blog post regarding what DevOps meant to me. I saw that frustration as an opportunity, as surely others were just as frustrated. I have ruminated on the concept of DevOps while continuing to build experience in both organisations and tooling -- turning over in my mind the reasons for which I made that original post.
There are some minor differences from my original thoughts on the subject which may appear, at first, to be contradictory. One of which is the view of DevOps as a methodology. DevOps is a methodology, but it is not yet another methodology which simply prescribes a set of practices. Instead, DevOps describes a culture -- a way to approach and solve problems. It is a new paradigm -- and yet, I am compelled to call it a methodology if for the simple fact that people know what that means. Perhaps this usage is correct. Perhaps it is not. I believe that time and the community will tell.
In this post I had hoped to cover how my thoughts on DevOps had changed over that time. Unfortunately, those thoughts have grown a bit too big for one blog post. So, in the next several months and years I hope to publish a series of blog posts and a series of always updated books (DevOps: A Manifesto and DevOps: A Workflow) which describe in greater technical and organisational detail about what DevOps means to me and, hopefully if I do it right, you as well.
DevOps. Systems as Source Code.
DevOps as a discipline is relatively new to most and as such there are few well-known best practices when it comes to actually implementing the catechism "you shall build, test and deliver your systems as you shall build, test and deliver your applications." What does that mean to apply that to systems development? What does it require? What do we not have to invent from scratch ourselves? When can we call ourselves successful in creating and implementing such a methodology?
The end goal of a successful implementation of this methodology called DevOps, like any correctly implemented methodology of production, is, of course, a working system for delivering software product to consumers and for providing information about that consumption back to those running said system. As customer demand varies we need to be able to rapidly meet those demands. Not only that, but when customer demand wanes we must release the resources we have commandeered in order to serve those requests in order to save costs. To accomplish this successfully we must have a authoritative master plan for every working module in the system because at any point a new instance in a given configuration may be required, even if there is no existing concrete example from which to spawn a copy (for instance: during a whole systems integration test). Where is our master plan?
In the application development world, that master plan is source code. There is a lot of tooling built around these collections of source code -- linters, compilers, analysis engines, continuous integration systems, test suites, debuggers and IDEs which all work together to produce an application. Methodologies work on and with the people working on this authoritative master plan. Up until recently, traditional operations departments were considered lucky or even advanced if they had a perl or shell script to set up a new machine and, if that script was really well developed, it might be idempotent enough to get things back to how they were before someone changed things by hand. Very often deployment and management of the application was nothing more than a last minute consideration. We have nothing on which a methodology can operate. We have no common concise language for communication. Systems developers are hamstrung. Where is the source code for the application as an operating module within the larger machine known as the business? All too often it doesn't exist, except, perhaps, in the minds of a dozen system administrators.
How can we reformulate our experience as a set of cooperating engineers (some with a focus on application development, others with a focus on systems integration and so on) in order to take the practices and tools with which application development has been so successful and make that success apply to the entire engineering organisation? Well, in order to apply the tool sets and best practices of software development to the problem of systems integration we'll need to come up with a unified theory that may be applied in such a way that the tools of applications software development can be applied in a manor which is compatible with the situations and scenarios for which they were developed. We need to make everything we know about systems machine readable and machine understandable. We need to make developing systems equivalent to developing software. When we do this we get our source code -- our authoritative master plan. How best to do this?
Most software is a collection of APIs tied together in interesting ways. Fortunately, in recent years, companies, groups and individuals have been building APIs for hardware and resources, both local and remote -- provisioning, configuring and managing machines, operating systems, entire virtual networks and even data centre space. They've been building frameworks and systems for organising the usage of these APIs. Those APIs and frameworks are our "flux capacitor." They allow us to leverage application development tools for systems development. They are what makes DevOps possible.
Now that we have the same tools to speak the same language, source code, truly working together on a level playing field is now possible. From this comes shared culture. There is no longer any one group known as either operations or development, there is only one engineering group. The notion that everyone on the team, or in an engineering department, is responsible for the whole product, delivered to end users is really what DevOps is about.
From shared language and responsibility comes shared understanding and better communication, in a virtuous cycle. This is what brings all the benefits for which DevOps is known to the engineering organisation and business. DevOps is not a title, it's not a tiger team and it is not a department. If you don't have a DevOps-y culture in your whole business process, then you aren't doing DevOps.
"Well," you ask, "just how does one build a DevOps culture?" That's what we're going to explore together.