Testing with Container
UNIT TEST
JUNIT -
JUnit is an open source test framework for Jave. If I were to try and describe it in the simplest terms I would say, it allows us to mark methods in Java as tests. Then we design those methods to exercise our application. This could be a unit, an API or some code that uses a UI library to drive a UI.
It then also provides a test runner, a test runner is a tool that is able to find the methods marked as tests in the source code/jar file and execute those blocks of code. Now tests aren’t much use to us without some validation/verification at the end, so JUnit comes with an assertion library. This assertion library allows us to compare objects we are expecting vs the actual so we can determine a pass or fail.
Unit test frameworks can contribute to almost every stage of software development, including software architecture and design, code implementation and debugging, performance optimization, and quality assurance.
JUnit is the foundation for more specialized unit testing tools, including Cactus, Jester, JUnitPerf,
Test that simply runs a program and checks its return code is a black box (functional) test, since nothing is known about how the program is written
Unit tests are usually white box (structural) tests, since the test framework is able to access the internal structure of the code being tested
Unit test frameworks are a key element of Test Driven Development (TDD), also known as “test-first programming.” TDD is one of the most significant and widely used practices in Extreme Programming (XP) and other Agile Development methodologies
TDD can be summarized as “test twice, code once
1. Write a test of the new code and see it fail.
2. Write the new code, doing “the simplest thing that could possibly work.”
3.See the test succeed, and refactor the code.
UNIT Test Family
PyUnit, - For python.
Mocha and Chai - for NodeJS
NUnit for .Net
CppUnit - for C++
JACOCO
Code coverage is a metric that can help you understand how much of your source is tested. It's a very useful metric that can help you assess the quality of your test suite,
Function coverage: how many of the functions defined have been called.
Statement coverage: how many of the statements in the program have been executed.
Branches coverage: how many of the branches of the control structures (if statements for instance) have been executed.
Condition coverage: how many of the boolean sub-expressions have been tested for a true and a false value.
Line coverage: how many of lines of source code have been tested.
It is a code coverage analysis tool what inspects our unit tests and our code base and is able to report some metrics. Based upon the amount of code tested by our Unit tests.
JaCoCo is a code coverage library developed by the EclEmma team. JaCoCo embeds an agent in JVM which scans the code paths traversed and creates a report.
This report can be imported into a wider DevOps code quality tool such as SonarQube. SonarQube is a platform that helps to manage code quality with numerous plugins and integrates nicely with DevOps processes
Use coverage reports to identify critical misses in testing. Focus on Unit Test first.
Good coverage does not equal good tests.
You might consider setting a failure threshold at 70% as a safety net for your CI culture.
Coverage - Python code coverage
Istanbul - Javascript
Find Bugs - Static Code Analysis
Static code analysis is a method of debugging by examining the source code before a program is run. You could think of static analysis tools that helps you maintain a healthy codebase without having to actually run that code
This type of analysis addresses weaknesses in source code that might lead to vulnerabilities. Static analysis is also used to comply with coding guidelines like MISRA a ISO 26262. Benefits
Speed - Takes time for manual code reviews
Depth - Covers every possible code execution path
Accuracy - Scans every line of code to identify potential problems. and is not error prone like manual code review.
Ex. String with null convert to upper or lower case or SQL query without prepared statement.
Facebook Infer - Java and C++
JSLint - nodejs
FindSecBugs, SpotBugs, Sonar Qube
Checkstyle
Coding conventions and standards are a way to ensure that code is formatted consistently across a project, team or organisation. It essentially comes down to making the code more readable and accessible to yourself and other developers
Checkstyle is a tool to help analyse Java code that adheres to a coding standard
By default checkstyle supports the Google Java Style Guide and Sun Code Conventions, but is highly configurable. https://google.github.io/styleguide/javaguide.html
Checks, Sources code file naming, Source files structure, Formatting, Naming, Programming practices and JavaDoc.
Custom configuration can be done using checksyle.xml
Example: No wildcard imports, no static import of classes
INTEGRATION TESTS
Mock every thing is over - We have moved from monolith to microservices - early what used to be just functional calls are now service calls. This is Service Mesh - which is dedicated infra layer for service to service communication are becoming very popular where we have control plane and data plane. Data Plane are intelligent proxy deployed as sidecar( like Envoy), these processes controls the flow of data traffic between microservices, where as Control Plane manages the policies and metrics. So, the point is that we have multiple small services and we need to find an effective way to test it. (LinkerD and Istio)
Problem comes with integration test, integration test are hard to write as there is a concept of orchestration. You need to spin up one or more services to test you code against them.
Use docker-compose to spin up all the required components and run test-integration to run the integration test suite against the docker containers has provisioned.
Here the problem is that, the containers are outside your control, and you believe every thing goes up well.
Validation is another issue - where you are not sure when to run, when you dont have control on that is running. For the same reason why orchestration is hard same goes with validation.
Falkiness - contolling all the changes from outside is hard.
No Parallelization -
What we can do better.
Docker has an API - When we run docker cli or docker run we are calling the dockerd daemon.
It gets the requests from docker-cli and does what you asked for and returns the results.
The idea is to spin up docker using docker api - to spin up containers inside our tests itself and run our tests against it.
Think about docker architecture like a onion where we have
Docker deamon running
On top we have rest api
Above it is the docker cli client
DockerD there is another demon docker containerD, Docker containerD is the release of containerD. ContainerD is a CNCF project.
All the api that are exposed by the chain of tools we used can be reused. If something you are looking for is not exposed, it doesnt mean that it canot be used, it means thta it is not exposed (as it is not a common usecase) and you need to implement yourself
Docker-in-Docker
You can spin up a container that can spin up another containers. The way to do this is to share the docker socket which is the unix socket that docker cli looks to communicate with DockerD.
If you share the host volume to inside the container, from inside the container you can communicate with dockerD. This is what lot of CI tools does, like Travis CI, CircleCI etc. So you can configure and share the docker volume inside the container, you run the container, test inside the container and test inside the container and destroy it later.
API - You can also expose API - the docker demon has a TCP endpoint so by default it exposes UNIX socket so that's why it is /var/run/docker.sock, because there ia file and you can communicate with the files to the socket. If you want to communicate with dockerD which is not running in you machine, you can expose the docker daemon via TCP and you can reach to the server from your Docker CLI. Just doing docker-cli and specifying the endpoint and point it to IP of the docker daemon. This is just an introduction and there is lot more that you can do as docker is super flexible because it has this API that you can use.
Dont stop just about running docker run or docker remove because there is a huge ecosystem that you can build just extending the API capabilities. When you are using this you have to remember that the connecting system can use dockerd on remote as root and which is not sure. Probably you should not let it publicly open, just skewed within the private network.
Docker SDK - docker has a set of API and client libraries. You can also use the SDK to run containers etc. So through code you can control everything.
So you can programatically providing integration tests. The idea is that you are not going to reply on docker-compose or an external Orchestrators to get your containers for the tests you are running but you are able to spin up the containers from the test itself, and they will tear them down once the test is done.
TEST CONTAINERS
Test containers is a community of libraries and there are libaries for lots of languages Java, Go, etc. Java is a famous one, there is also Go, Scala and rust versions.
You should think you them as a wrapper around the docker SDK
Moby-ryuk - One of the project that started the TestContainer idea is Moby Ryuk, which is a library that you can use and is used under the hood from test-containers to clean tyou containers. If you dont defair your containers there is Ryuk that runs in the background and it removed it. Test containers runs containers with label and Ruyuk remove the containers with labels which TestContainers created.
Something that you can do now that you have you programmatically measure your container and your test is to write utility functions. So, you can write a framework for your application that helps you to write integration tests. This is very critical and crucial because if you can be sping up a good library and you can provide a good library for your application, other people can use it and will be happy to write integration tests. Doesn't need to be complicated it needs to be just enough to make other people happy to write integration tests.
You can use KIND for test containers inside your tests so
https://medium.zenika.com/dockerize-your-integration-tests-8d26a7425baa
TEST CONTAINERS
https://softeng.oicr.on.ca/alex_lepsa/2018/06/04/Testcontainers-FTW/
Whether you follow the TDD approach and are writing your tests from the very beginning, the object is to ensure that the application works correctly and consistently end-to-end
Say you want to do a unit/intergration test with a DB. There are 2 approaches which are followed
You can set you a DB and while running you can connect to DB to test. This will result in maintaining a separate DB and multiple tests on the DB may result it in inconsistent state.
Use a in memory database like H2. While this is much simpler, it is then again a simulator and you are not testing with production image. And features like enums, encrytions are not supported uniformly.
Test container are there for rescue.
Wouldn’t it be great if we could have something that is as easy to setup and as portable as an in-memory DB, but that is an actual database not a simulation, and wouldn’t it also be nice if that one solution could support the most popular databases out of the box but also allow us to roll our own custom containers as needed.
Easy to setup
Manage with existing configuration (say application.yml )
Treat it as a full-fledged database.
Testcontainers integrates easily with Spring requiring nothing more than a couple additions to our POM and as little as two small edits in our application.yml, including pointing it to our SQL init script!
UI TESTING - SELENIUM
Selenium is one of the most commonly used frameworks for browser automation. Selenium provide an easy and scalable way to test web applications and reduce time-consuming tasks.
A standardized API that allows for automating web browsers
A server application that helps to coordinate and distribute tests across multiple operating systems and browser combinations
Test automation works through the following means:
Automation scripts are written by the testing and development teams. These scripts contain instructions for how to perform the test, including finding and manipulating objects on the web page, using information as input, verifying messages from the web application, and interacting with elements.
The Selenium WebDriver provides interfaces, classes, and methods to interact with browsers and elements in the web application.
Browser providers and third parties have developed browser drivers to work with the WebDriver API. This has made Selenium available to execute tests on all of the popular web browsers.
Selenium IDE
Record and Play
Cannot support large projects
Only be used with Firefox
Selenium RC
Supports multiple OS, browsers and languages
Works on any browsers which supports Javascript
How RC works
Any elements in webapage can be located with selenium and there by automated
It used JavaScript to locaate element
Needs a running RC (remote control) server on a machine
Selenium RC is complicated because it uses an intermediate RC Server to communicate with the browser.
The RC Server is installed initially before running the test scripts and acts as mediator between your Selenium commands and your browser
Drawbacks
Javascript injection is blocked by many browsers
Selenium RC is deprecated and WebDrive takes over
Selenium Webdrive
This is an API that provides a simple interface to interact with browsers in a programmatic way
No javascript injection
Supports multiple OS, browsers
No server is needed
Directly interacts with native code of browsers
Separate class for different browsers
Selenium Grid -Paralled/Sequential Execution
Designed to distribute test cases to different machines
Example scenario:if you have 3 machines and 200 test cases
Make one machine as hub and 2 machines as nodes and divide 100 test cases to both the nodes
Zalenium
A grid that scales selenium containers. All non supported capabilities are sent to cloud.
Dynamic on scale selenium grid
Expands and contracts during run time
Simple to setup
Cloud support
Docker + Selenium hub. Where you can see live video of the execution on the browser itself.
We have to hardcode the number of containers. They created a smarter hub. I will create the containers, I can create on the spot. If you request 5 threds I will create 5 containers for you. But it has big images more than 1GB. But any ways it is going to be one time activity.
Selenoid
It is lightweighted, fast and provides most of the featues as Zalenium.
CM - configuration manager module - makes it easy for installation and configuration. Consider it as a wrapper for docker commands.
Bring up a windows container in Linux
docker run --privileged -d --name edge -p 444:444 -p 5900:5900 registry.aerokube.com/windows/edge:18
List of Images - https://aerokube.com/selenoid/latest/?source=post_page---------------------------#_browser_image_information
Create Windows image - https://github.com/aerokube/windows-images