InfraAsCode-IaC

Infrastructure as code

You may have created, or at least used some kind of systems automation in your career. But the DevOps approach to automation isn't just to have some automation around your systems, it's to create and manage them as dynamic assets in the first place. - We call this approach infrastructure as code.

It's where we treat infrastructure the same as we would treat code for applications. So what do we mean by that? Put simply, you write it in an IDE, save it into source control, run tests on it, build it, package it and deploy it.

Manually built and changed systems require a huge amount of work and cause a lot of trouble. The resulting systems are special snowflakes that aren't truly identical.

Differences between test and production environments plague application quality. Unexpected and unreviewed changes have been proven to be the leading cause of downtime. - Yeah, so then we layer on systems and processes to control change and discover our infrastructure. This is to prevent configuration drift, but really it just adds a lot of bloat and more work without improving quality. - That's right.

We having a saying in DevOps. Servers should be cattle, not pets. You don't handcraft them and cater to their unique needs, you mass produce them using the standard mechanism, kill them and get more. Leave the cool customizations for the back of your laptop. - You know creating your infrastructure as code doesn't mean you need to learn C or Java. You'll learn some new, system specific languages, but the real difference is in your approach to systems as a whole.

Infrastructure as code is about adapting a development approach to your infrastructure, and it's more of a cultural change than a technical one. - That's right, it's hard to wean yourself from the tendency to log into the server and change it, but you'll have to. Like fixing a code bug, you'll change the code, build it, test it, and then deploy it.

First priority had to be the implementation of a model-driven system that we would use to create and operate our systems and applications, even though it would take some more time to get going up front. We built a tool that would simply take models that developers and operations engineers collaborated on in source control, instantiate them in the Amazon and Azure clouds, deploy code to them, and dynamically hook up the services to each other. First in a preproduction environment for testing, naturally. - You know the difference was amazing. Suddenly ops became a nine to five job. We could focus more on design and implementation rather than on fire fighting and making manual changes. - And treating our infrastructure like it was code, was the secret sauce that made it work.

From code to artifacts

The task ahead is to realize that these are code, and we'll apply coding best practices to them. So, for folks that aren't developers, what does a normal code flow look like? Here's an illustration of a continuous delivery flow, where a code gets checked in,goes through successive levels of testing, and finally gets released.First, you'll want to use source control.

There are two common approaches to configuration management code. Imperative and declarative. Imperative, also known as procedural, is an approach where the commands to produce a desired state are defined and executed. Your average Bash script is imperative. You're creating a list of things to do, and the system does them. Then there's declarative, also known as functional. This is where you define the desired state, and the tool conforms the system to the model of your desired state.

Puppet manifests are declarative, as are SQL queries and makefiles. You don't describe the control flow of exactly what to do, you specify what you want done and the tool does it.Declarative is more efficient, assuming the declarative frameworksupports exactly what you want it to do. Like application code, you want to have tests for your infrastructure code and for your infrastructure.

Artifacts are what get versioned, tested, and deployed. Consider what artifacts you intend to use as the deliverables in your infrastructure code pipeline. For app code, it's usually executable binaries or JARs or similar. For infrastructure, you usually see DEBs, RPMs, or other OS packages. Docker images, AMIs, VM images, and stuff like that. You may have multiple layers of artifacts But deciding on what your artifacts are and how to manage and version them, is very important. In my shop, we build everything from OS packages, to scripts, to Java and Python applications into DEBs or Debian packages so that we can leverage that format's built in dependency management.

Then as a second layer, we build VM images and AMIs using Packer, and build Docker images directly with docker files. All this is controlled through our last seen Bamboo build system.We then use Artifactory as our artifact repository for both infrastructure and application artifacts so that our provisioning and deployment can leverage that single-source of truth. We can't stress enough the importance of creating and managing artifacts. After they're built, they never get changed, and as they're deployed from tier to tier, they're provably identical. Don't just pull code out of Git, use artifacts. But how do you know these artifacts are any good? Testing. In the next video,James will lead you through the fine points of testing your infrastructurous code, then I'll be back to talk about the second half of the deployment pipeline, where we take the artifacts and create running systems from them.

Testing your infrastructure

Testing is a programming basic, but many people still struggle with it. Testing is even more critical than ever as you increase automation and the speed of change in your systems. If there's only one thing that you should take away from this course, is that you should always test your infrastructure. We'll be diving into three key areas of testing for infrastructure, unit, integration, and security testing. Well, let's do a few definitions before we get started here. Unit testing is code-level testing and takes the smallest testable code blocks and verifies their execution. We lump in formatters and linters in this category in case you are also familiar with those. Integration testing is at the heart of infrastructure. Sure we can test code that builds the systems, however the actual build and runtime has to be validated.Integration testing focuses on testing several components together with a fully assembled and running system. Security testing is the next logical step from integration testing. In this step, we look for security tests for both positive and negative security attributes. By that we can check for expected hardening of the system or test for possible weak spots. Unit test, integration test and security tests need to be considered.

Provisioning - The process of making a server ready for operation, including hardware, OS, system services, and network connectivity

Deployment: The process of automatically deploying and upgrading application on a server.

Orchestration: The act of performing co-ordinated operations across multiple systems

Convergent: CM automation that runs to converge systems on a desired state

Idempotent: Ability to execute a CM procedure repeatedly and end up in the same state.

Immutable: Deployments that are not intended to be changed after deployment, rather the entire system is redeployed if needed. (ex docker, AMI etc)

IAAS - https://csrc.nist.gov/publications/detail/sp/800-145/final

Containers with Docker

What is a container? It's a construct based on cgroups and LXC that allows you to run segmented mini-systemson your Linux or Windows host system. These systems are smaller and more light weight and transparent than a virtual machine but are segmented and isolated more than just running an application on the host machine.Because they share the system's kernel they boot fast and are light weight but they provide isolation where you can install packages and do system configurationrestricted to just a single container. Docker's the most popular container technology but there are others like CoreOS's Rocket, or even just using LXC natively

But the real genius of containers is that they begin to blur the line between systems and applications. Why do we run systems and infrastructure?In the end, because we have to. They're a cost of doing business to be able to run our applications. But everything to do with our infrastructure is waste in the lean sense. The old model is that you construct purpose-built servers in order to run one or many applications, and once that becomes mini you have a whole world of hurt from conflicting configurations,maintenance schedules and so on. The beauty of Docker is that each container is intended to be one service, one or a very small number of processes with a minimal system environment customized to itself.

Timeline:

1998 - Virtualisation - VMware founded

2006 - Public Cloud - AWS launched

2009 - DevOps term coined

2019 - Private Cloud - OpenStack launched

2013 - Containers - Docker released

2014 - Serverless - AWS lambda released

Value vs Work Required

Public Saas >> Serverless >> Containers >> Private PaaS >> Public IaaS >> Private IaaS >> Virtualisation

Architectural Considerations

- All these tools and approaches have their pros and cons. One size doesn't fit all. First you need to understand your needs. And then pick the CM approach that fits those needs. And then, finally, you'll select one or more tools to implement that approach.

Remember, you want to get the most value for the least effort. - Yeah, it's about value. You know, serverless is a great fit for simpler applications. Most environments are full of apps that do something pretty basic, run a report, transform a file, take repeated data uploads. -

For a more complex system but one that you can code from Scratch using a microservices approach,use Docker containers. Something like Kubernetes or Amazon ECS to orchestrate them.

But what if you have a large distributed environment, like a university lab or a corporate network with heterogeneous systems and a variety of applications? In this case it's usually a good bet to use something like Chef or Puppet to make building your systems repeatable and protect against manual changes. - The convergent model is particularly strong when you have a large body of systems and you don't care which exact minute out of a 15-minute period they use to update themselves. - Okay, but what about a website with high uptime requirements? There, unless your application has been written to maintain uptime while being updated, you need orchestration to coordinate that release. - This is where Ansible or another orchestrated approach is a good fit.

what if you're using commercial instead of open-source software? Or if you're running on a Mac or Linux or Windows platform? - That's a good question, but it matters a lot less than you'd think nowadays. Docker runs on Windows, Linux, and Mac platforms and will do Windows containers. Chef and Puppet support, while mostly Unix and Linux focused 10 years ago, is strong on Windows today. - Yeah, and as for commercial software,vendors from Oracle to IBM, they're eagerly making their products available on the Cloud or in Docker containers and so on. So, don't assume that just because something's newfangled it's not going to work in your tech stack until you check. - Commercial technologies, like WebLogic, have aggressively moved to these newer platform technologies. Because the more effort their customers have to put into hosting, installing, and maintaining commercial software, the less value they realize, and therefore the less they're willing to pay for it.-

Your overall toolchain should form an optimal value stream that not only meets your needs but the needs of the other people and teams that have to interact with it. - That's right, and don't forget the little details that make your tools effectively usable for others. Integrate your tools with your Dev's IDEs, put them in the same source controland artifact repositories, make them usable on the Dev desktop as well as in the server space. Bringing both sides of the house together is what makes a DevTool or an OpsTool a DevOps tool.