First and most important step is: write tests
Automation tests, the three-tire test automation pyramid:
This makes integration & UI tests cheaper and faster.
https://medium.freecodecamp.org/the-best-ways-to-test-your-serverless-applications-40b88d6ee31e
To make application testable, break up function into smaller ones. ("Imagine if testing automobiles was done that way. That would mean that every time you wanted to test a single screw or even a mirror in a car, you would have to assemble and then disassemble the whole car.")
Function business logic exposes few “ports” (or expects few arguments). For example, one for an incoming event, one for permanent storage, and one for notifications.
They have two adapters for the event that triggers a function, one for the real AWS Lambda trigger and another one for local testing.
They have several adapters for permanent storage and notifications. For example, DynamoDB table adapter and in-memory adapter.
Integration tests benefited a lot from Hexagonal Architecture. They were able to fully test integrations that they own. Third-party integrations are simulated with other adapters.
Yes! And it is easy with the help of a tools such as Serverless Chrome, Chromeless, and Puppeteer. (combination of serverless & headless browsers) Tools: Appraise https://github.com/AppraiseQA/appraise/blob/master/README.md/
To deploy the app, use any popular tool, such as Claudia.js, AWS SAM, and Serverless Framework.
You can still use your favorite CI tool (like Jenkins, TravisCI or SemaphoreCI), or if you want to stick with AWS, you can try AWS CodeBuild.
As serverless apps heavily depend on integrations, the risk shifts from your code to the integrations.
My observation: a big part of integration lies in cloudformation templates. The main point of cloudformation templates is its side effects - cloudformation service create resources with the template. This is best tested with real AWS cloudformation on the cloud. Mock API? Yes it can test if the expected template is provided etc. but the side effect of the template? No. Mock AWS services like localstack? On today (31 May 2019) it seems still very buggy. Also how fast can localstack chase the changes of AWS? There are already some odds with AWS (https://sites.google.com/site/bingsite/aws/gotya) what's the point to have to deal with the odds of a mocking environment? Maybe forget about it and just deploy to cloud. Focus on make the test streamlined so its as fast & automatic as possible, but mock has a limit. It's not the complete answer. Some mocks maybe ok to use (dynamodb, S3, etc.) but cloudformation looks like too complicated.
If integration (or something else) goes wrong, catch fast. Serverless monitoring tools of the good and popular options are IOpipe, Thundra, Dashbird, and Epsagon.
But, serverless apps often have a thick client, which means that back end monitoring is not enough. You need a similar tool for your front end. This market has a lot of nice tools too, such as Sentry and Rollbar (error & crash tracking).
Desole - serverless application exception & error tracking tools can be deployed to own account.
Testing the communication contract between the consumer and provider. https://docs.pact.io/
https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/logging-sdk-calls.html
However logging still call the actual AWS service.
https://medium.com/@amolkokje/mocking-aws-services-f80d1fd114c1
Using mocks lower costs. However mocks may have slight differences with actual infrastructure & therefore if cloud cost is low, why not just test on cloud? Answer from the author:
"In my experience, in the absence of these mocks, the costs are low in the beginning but rise rapidly as your development nears completion as you keep adding support for more features that use more services.
This is especially a very important consideration when you have regression testing setup. I would recommend setting up mocks early in the development phase of your project and develop solutions in such a way that they can be easily configured to test with the mocks."
This kind of services set up a local mock server with and endpoint. Then client needs to configure to use that endpoint (passing 'endpoint' when instantiating a service)
Fake S3: a lightweight server for testing S3, client needs to configure endpoint
Moto: can mock not just S3 but a whole lot of services
Dynalite: local dynamodb
Amazon’s DynamoDB Local
Kinesalite: local Kinesis
Localstack (article author's personal recommendation): pretty advanced mocking environment including error injection (frequently occurring in real cloud environments), actual REST services, etc.
Nock (not just AWS): HTTP server mocking and expectations library for Node.js. Nock can be used to test modules that perform HTTP requests in isolation.Nock works by overriding Node's http.request function. Also, it overrides http.ClientRequest too to cover for modules that use it directly. Nock can also record calls https://github.com/nock/nock#recording. Looks like someone actually uses nock with aws. Worth exploration. Maybe use nock & aws-sdk to record, and then somehow replay the recorded for repeated tests...
Contract testing message queue producer/consumer: https://www.dius.com.au/2017/09/22/contract-testing-serverless-and-asynchronous-applications/
Lambda: https://dius.com.au/2018/10/01/contract-testing-serverless-and-asynchronous-applications-part-2/
GraphQL (not mentioned AppSync) http://blog.pact.io/2018/07/24/contract-testing-a-graphql-api/
https://www.npmjs.com/package/aws-sdk-mock
To test without actually calling actual AWS services. It uses Sinon.js under the hood.
I feel that mocking has lots of tricky things to be careful of to make it work... but still worth trying. This package is actually a thin wrap surrounding sinon. Just check the source code. It looks like mainly concerning installing sinon mocks etc. to AWS instance.
What to be tested and how to test them
Overall system behavior (Acceptance testing):
Cloudformation / Infrastructure as code:
Lambda handlers: