DEVOPS – HOW TO CHOOSE RIGHT TOOL
CHECK THE ATTACHED WORD DOCUMENT FOR FULL DOCUMENT WITH IMAGES
1. DevOps explained
2. Stages of DevOps
Plan
Build
Continuous integration
Deploy
Operate
3. Tools
a. Docker
b. Jenkins
c. Nexus
d. Monitoring tools
i. Prometheus
ii. InfluxDB
iii. Nagios
iv.
4. Putting All Together
5. Practical example
Security Testing –
http://find-sec-bugs.github.io - Scan source code of java application. Can integrate with you Jenkins and also with IDE.
www.owasp.org - Free security tool. Hack your website – spider your site and can be used in manual security testing. Jenkins plugin available or with docker
http://sqlmap.org - Sqlmap is an open source penetration test tool that automates the process of detecting and exploiting SQL injection flaws.
www.openvas.org. - Open source vulnerability scanner and manager. (quails and nesis)
OWASP Glue Tool Project – To make security automation easier. Glue is docker container that you can put all of your tools into, out of the box it supports find set bugs, contrast, fortify and it takes all of the results and allows you to export them into json, plugin in JIRA etc. You can configure this as part of the devops release pipeline.
https://gitbub.com/OWASP/glue/blog/master/TOOLS.md
recon-ng -
https://github.com/devsecops/awesome-devsecops - Good list of devsecops tools. Trainings, presentations, tools and dashboards.
------ DEVOPS BIG PICTURE ---------
Every one is talking about Devops, is it ia new cloud what is it, is it a rebranding for the things we have before or is it a new technology or cultural thing. We will try to get to the bottom of that.
COTS Vs CUSTOM
Commercial Off the shelf Software
Organization may have different unitls like finanance BA etc, which may even run on mini IT dep which runs on cylos.
Many a times people leave after deployment – You may have a skeleton crew but all system owners like dev , testers, PM all go on to different projects. Operation remains, but it is one of the many project they maintain. Source control and probably outdated specifications are left behind but any of the know how or custom tools/ solutions are not. So this is typical is most organizations.
PAIN
Outages –
No incremental value
SLOW IT
INFIGHTING
Identify Waste –
Knowledgewaste - Operation and dev no talking. Marketing not sharing what they know to IT. This could be because of many reasons, physical barriers, reshuffling, or different organization units are not talking to each other.
Waiting Waste – for env to be created, testing to be finished, other team waiting for code to finish, approval waiting. It is a huge problem.
Over Production Waste – Asking for more resource that we need as the turn around time for more resource will take time.
Over Processing waste - Gold platting. Doing more that what is needed
Motion Waste – Long night pushing software. Repeatable tasks of deployment running through long list of deployment procedures.
Transportation Waste – Clumy way of moving codes. Moving things around
Correction Waste - Quality check at the end of the project – it should be from the start of the project.
Inventory Waste – We have code to be shipped – sitting on staging wating…
DEVOPS
Lean
· Focus on Customer Values
· Eliminate Waste
· Reduce Cycle Time
· Shared Learning
· Avoid Batching
· Theory of Constraints
“IT is the factory floor of this century” - GENE KIM
DEVOPS is a LEAN for IT
CAMS Model – Culture, Automation, Monitoring, Sharing
IF YOU CANT MEASURE YOU CANNOT IMPROVE
One interesting finding of the state of devops survey is having branches with forks with less life time before merging them to trunk and less than 3 active branches in total – all contribute to higher performance. It is also reported that lead time to deploy in high performance org is less than an hour.
Continious Integration – principles
1. Build should pass the coffee test ( ie. < 5 min)
2. Commit really small bits
3. Don’t leave the build broken( stop meeting set the build right first before anything)
4. Use a trunk based development flow. Helps rework and the pain of merging branches. ( High performing teams follow the trunk based approach)
5. Don’t allow flaky tests ( some time pass, some time fail)
6. Build should return a status, a log(of all test run) and an artifact(should be tagged with build number)
Continues Delivery:
Practice of deploying every changes like production like env and run all tests on it.
1. Build artifacts only once ( and use in all env)
2. Artifacts should be immutable (ie. It should not be allowed to be changed). CI system has only permission to write to artifact repository whereas the deployment system has only read access to the artifacts) - Why – to create trust that the build has not changed. This will also help in the audit ability, as we can trace back to the source of the build.
Flow
CODE à Version Control à CI system creates build à Build go into shell à The deploy goes to stage à Staging à testing à Ready for Prod à Deploy in Prod
3. Deployment should go to a copy of production
4. Stop deploys if a previous build fails.
5. Deployments should be idempotent
Once you planed out your pipeline. Trace a code change to answer
a. Cycle ( able to audit the change through the entire cycle)
b. Overall Cycle time - How fast you move the change to production, this is the overall cycle time
QA – Testing in continuous delivery
1. Unit Testing - Lowest level of testing for testing the code ( say add 2 +2 ) runs on developer machine.
2. Code hygiene - Linting, Code formatting ( based on the language used)
3. Integration testing - Like unit testing, but with all the integration components
4. Security testing - Simulated XSS attack test
5. Testing
a. TDD(Test Driven Development)
a. State desired ourcome of a test
b. Write code to pass test
c. repeat
b. BDD(Behaviour Driven Development)
Work with stakeholder, describe business functionality and Tests should be based on natural language description (ex when I give 2 numbers it should return the sum)
c. ATDD(Acceptance test)
This test is followed by TDD and BDD
End user perspective, Use case automation testing and testing is continuous during development.
6. Infrastructure testing – Involves starting up a host, running the test and turning it off
7. Performance testing - Basic performance should be part of every thing you do, but part of it – load test, stress test and spike test. And are good on nightly tests
CI TOOL CHAIN
1. Version Control
Github or Bitbucket
2. CI systems
Jenkins (CloudBees –commercial offerings), GoCD, Bamboo, GitLab CI and Teamcity. Continuous integration as a service - travis CI or Circle CI
3. Build
Build tools are language dependent. You simply will be going with Make/rake when you are using Ruby. These execute a consistent set of steps everytime. Or you can take workflow based approach with maven which allows you to run reproducible tests. For front end testing it is popular to use Gulp. If you are building identical machine images for multiple platforms from single source config -use Packer
a. Make/Rake
b. Maven
c. Selenium( frontend), Jasmine(test javascript code), Gulp
d. Packer ( say for building AMI for AWS) - build images.
4. Test
a. Junit for Java ( same gest with Linters and formatters)
b. GoInt or Gofmt ( for go language)
c. Rubo-Cop (for ruby)
Integration Testing – Outside in approach while testing code.
d. Robot
e. Protractor
f. Cucumber
g. Selenium ( for automating web application testing). You can use the cloud offering of selenium which is called Sauce Labs.
Infrastructure Testing
h. Kitchen CI (runs infra code on one or more platforms in isolation)
Performance Testing
i. Apache Bench
j. JMeter
Security Testing
k. Brakeman ( code inspection – open source)
l. Veracode ( paid version)
5. Artifact repository
a. Artifactory
b. Nexus (open source)
c. Dockerhub (or private registry)
d. You can also tag your artifacts and upload to S3
6. Deployment
a. Rundeck (workflow driven deployment)
b. UrbanCode (commercial)
c. ThoughtWorks (commercial)
d. Deployinator (open source)
Tools in this space change a lot, and nobody in this space has the exact pipleline. When building pipleline what is the min set of tools required to building this portion pipeline. Anything that does not add value directly to what you are building is a waste.
DESIGN FOR OPERATION
The most common infrastructure failure is due to the failure in the integration systems. We do not want to have cascading failure if there is one failure in the systems. To over come this we can adopt Circuit Breaker Pattern.
Netflix Hystrix – wrap a call to a integration point and it automatically provides circuit breaker functionality for the call. https://github.com/Netflix/Hystrix/wiki/How-it-Works
Good Manifesto for a service ready software is called the 12 Factor app. https://12factor.net/
You can adopt Factorish to easily make legay application behave like 12factor application.
https://github.com/factorish/factorish
Blogs which page objects, to server list – def are clearly written
https://martinfowler.com/bliki/
Dev comes from school and Ops comes from street
Pushing more and more highly available system is a loosing game. Don’t fall for the trap. Take the option of deliberate adversity. It is called the chaos monkey
Performance - Dev should be aware of code profilers.
APM Tools – Application Performance Management – light weight profiling across the whole architecture. You can used in production based on system, but it is more appropriate in dev systems.
APM Tools
a. AppDynamics (from Cisco)
b. Dynatrace
c. New Relic
d. CA - Wili
METRICS AND MONITORING
LEAN Approach
1. Build
2. Measure
3. Learn
4. Repeat
What to Monitor?
1. Service Performance and Uptime
2. Sofware Component Metrics – check for ports or process status. This move up a layer and instead of asking is my service working, is it asking if this particular host working.
3. System Metrics – from CPU, Memory (time series metrics)
4. Application Metrics – What your application is actually doing. How long a function call is taking, no of logins in the last hr or the account of all the errors that happened.
5. Performance
a. Linting
b. Code formatting
c. Banned function checks
6. Security –
a. system(bad tls, open ports etc)
b. appsecurity ( XSS, sqlinjection) check if they are tipped on your side
c. System security – password resets, new account creation, invalid logins
d. Anomalies – http 401, or irregular ip
But the monitoring should not be more than what is needed.
Metrics - Application Metrics, System Metrics, Infrastructure Metrics - to be collected
Microservices – Distributed architecture are harder to manage.
Devops –Driving the need to sense and respond more quickly.
LOGGING
Remember the 5w of logging
1. What happened
2. When did it happen
3. Where did it happen
4. Who was involved
5. Where did that entity come from
It can be used in Audit, troubleshooting, user-experience.
With the distribution system in place it is more important to have centralized Logging.
Principles of logging
1. Do not collect log data if you never plan to use it
2. Retain log data for as long as it is conceivable that it can be used
3. Log, but alert only on what you respond to. Clearly define log level. There should never be standard error.
4. Don’t try to make your logging more available or more secure than your production stack.
5. Logs change. – Centralized logging system should be though about
Monitoring, Metrics and logging are the 3 feedback loops that brings back operation into design. Other feedback loops include incident command system, blameless postmortem and transparent uptimes.
SRE tool chain
Most successful are the tools which most people agree to use. Look for sharing and integration while selecting your tool.
Monitoring
- Nagios or Zabbix which are available for more than a decade. Latest tools are
Sofware as Service Monitoring
- Pingdom – to system and metric monitoring
- Datadog
- Netuitive
- Ruxit
- Librato
- New Relic
- App Dynamics
Opend Source Monitoring
- Graphite
- Grafana
- Statsd
- Ganglia
- InfluxDB
- OpendTSDB
- Metrics.dropwizard.io
Container logs
- Prometheus
- sysdig
IGINGA and SENSU are so what similar to Nagios which has modern plugins.
ELK stack
Incident Resolution (SAS)
- PagerDuty
- VictorOps
- Flapjack - Open source flexible notification system.
StatusPage
- Statuspage (https://www.statuspage.io/) - showcase uptime by subscribing to the service
Orchestration -
- Rundeck
- Saltstack
- Ansible
Bash
- ./ rerun (available in github) - Take your shell script and turn them into powerful featured management utilities with no extra programming. Which has unit testing built in. Simple framework that turns loose shell scripts into modular automation.
SCRUM talks about features and nothing about bringing it to production
You build it you run it. – Amazon CEO
MICROSERVICE - it doesn’t do one thing and one thing only. I think micro services, it’s a coming together of a bunch of better practices from a number of different communities.
SOA VS Microservice
James Lewis: So, first off, I’d like to make a distinction between the service oriented architecture as a community as a whole and the implementation as I have experienced it in lots of different organizations because I think there is a distinct difference.
If you actually look back at the work people have been doing in SOA, there’s a service oriented architecture manifesto, for example. It’s actually pretty clear. It’s actually pretty sensible. I don’t think anyone can actually take much in the way of _______ then, but then if you contrast that with the implementation that I think is, well certainly my personal experience, and also the experience of many of my colleagues and of industry, I think in general there’s a massive difference between the ideal of service oriented architecture and the reality on the ground.
The reality on the ground is a 10-year $25 million program. That runs for two and a half years and gets canned that delivers no value. The reality is spending millions of dollars on pieces of kit that will solve all your integration problems and then those implementations never delivering.
So I think it’s really important to make that distinction between what the SOA community stands for and actually what the reality is.
So I think the micro services idea is really about that reality, which is why I think it’s no surprise I think it’s been a community-led or it’s been a bottom up approach to building systems. A lot of the people involved have been or still are heavily involved as developers on teams who see the real problems day-to-day with either the amount of time it takes to make change or how difficult it is to scale things and have been looking for an approach, which I’ve been looking for an approach that addresses some of these problems.
So I think for me the micro services style of building applications is intrinsically related to the service oriented architect because the service oriented way of building systems is an eminently sensible way of _______ _______ building systems.
I think the main differences for me are in the implementation details. So the decentralized governor’s models that these ______ —
Circuit Breaker
So when I got it rightly, when in Gmail I’m offline and say I will try back in 5 seconds and then it goes up to 10 seconds, then it goes up to 20 seconds, that’s an implementation of a circuit breaker.
What a circuit breaker in your software does is provides you a level of safety when you’re talking to external systems. So in my code I’ll implement a circuit breaker such that if the downstream system that I’m integrating with becomes slow or goes away and, of course, as we all know, there is essentially no difference between being slow and not being there at all.
These pieces of software, the circuit breakers, can trip and open and then alert people that the downstream systems aren’t available anymore. They also provide additional protection depending on the implementation. So things like avoiding the thread pull starvation and not blocking incoming requests and so forth.
Are you Ready for Microservices? Yes,
Increase agility
Enable extensibility
Facilitate distributed development
Support massive scalability
Build (almost) unbreakable systems
But, it requires
Highly complex architecture
Required unfamiliar design patterns
Requires new enabling infra
Requires DevOps processes and practices
Requires design and organizational discipline