Content

Command Query Responsibility Segregation (CQRS)

1. CQRS is an architectural pattern that separates reads from writes. 

2. A query reads from the data source, whereas a command writes or mutates the state. 

3. CQRS is a derivative architectural pattern from the design pattern Bertrand Meyer coined as Command and Query Separation (CQS).

4. 
   

6 Rs of migration

Twelve-Factor App methodology 

1. Principle I. Codebase

   1. “One codebase tracked in revision control, many deploys”

   2. Your code base should have a logical version control system that’s easy to understand.

2. Principle II. Dependencies

   1. “Explicitly declare and isolate dependencies”

   2. This principle maintains that you should never rely on the implicit existence of system-wide packages. Instead”

3. Principle III. Config

   1. “Store config in the environment”

   2. An application and its configuration should be completely independent. Further, storing configs constantly in code should be avoided entirely.

   3. Your configurations should have a separate file and shouldn’t be hosted within the code repository.

   4. Solution:

      1. Use AWS Secrets Manager

4. Principle IV. Backing services

   1. “Treat backing services as attached resources”

   2. In a 12-factor app, any services that don’t support the core app must be accessed as a service. These non-core essential services might include:  Databases,   External storage, Message queues, Etc.

   3. The code for a twelve-factor app makes no distinction between local and third-party services.

   4. Solution:

      1. AWS: Default model for Lambda 

5. Principle V. Build, release, run

   1. “Strictly separate build and run stages”

   2. A 12-factor app is strict about separating the three stages of building, releasing, and running.

   3. Start the build process by storing the app in source control, then build out its dependencies. 

   4. Separating the config information means you can combine it with the build for the release stage

   5. Then it’s ready for the run stage. It’s also important that each release have a unique ID.

   6. Solutions:

      1. CodePipeline

      2. CodeDeploy

      3. CodeCommit

      4. CodeBuild

6. Principle VI. Processes

   1. “Execute the app as one or more stateless processes”

   2. Store any data that is required to persist in a stateful backing service, such as databases. The idea is that the process is stateless and shares absolutely nothing.

   3. Solutions:

      1. Treat Lambda functions as stateless.

      2. 
         

7. Principle VII. Port binding Principle

   1. “Export services via port binding”

   2. 12-factor apps must always be independent from additional applications. Every function should be its own process—in full isolation.

   3. Example: Add a web server library or similar to the core app. This means the app can wait requests on a defined port, whether that’s HTTP or a different protocol.

   4. It does not rely on runtime injection of a web server into the run environment to create a web-facing service.

   5. Solution:

      1. Not applicable to AWS Lambda

      2. Applicable for container or Amazon EC2 based applications

      3. 
         

8. Principle VIII. Concurrency

   1. “Scale out via the process model”

   2. A true 12-factor app is designed for scaling. 

9. Principle IX. Disposability

   1. “Maximize robustness with fast startup and graceful shutdown”

   2. The concept of disposable processes means that an application can die at any time, but it won’t affect the user—the app can be replaced by other apps, or it can start right up again. 

   3. Building disposability into your app ensures that the app shuts down gracefully: it should clean up all utilized resources and shut down smoothly.

10. Principle X. Dev/prod parity

    1. “Keep development, staging, and production as similar as possible”

11. Principle XI. Logs

    1. “Treat logs as event streams”

    2. Unlike monolithic and traditional apps that store log information in a file, this principle maintains that you should stream logs to a chosen location—not simply dump them into a log file.

    3. Solution:

       1. Correlation IDs can play an important role when creating log files for microservice applications. They help you track issues in a complex distributed system.

       2. Example:

          1. 09-02-2015 15:03:24 ui-svc INFO [uuid-123] ……

          2. 09-02-2015 15:03:25 catalog-svc INFO [uuid-123] ……

          3. 09-02-2015 15:03:26 checkout-svc ERROR [uuid-123] ……

          4. 09-02-2015 15:03:27 payment-svc INFO [uuid-123] ……

          5. 09-02-2015 15:03:27 shipping-svc INFO [uuid-123] …… 

       3. To do this and to debug and track services, use Mapped Diagnostic Context (MDC) with logging. MDC is important for debugging and understanding microservices.  

12. Principle XII. Admin processes

    1. “Run admin/management tasks as one-off processes”

    2. The final 12-factor app principle proposes separating administrative tasks from the rest of your application. These tasks might include migrating a database or inspecting records.

    3. Though the admin processes are separate, you must continue to run them in the same environment and against the base code and config of the app itself. Shipping the admin tasks code alongside the application prevents drift.

Domain driven design

1. Placing the project's primary focus on the core domain and domain logic.

2. Basing complex designs on a model of the domain.

3. Initiating a creative collaboration between technical and domain experts to iteratively refine a conceptual model that addresses particular domain problems.

4. Bounded Context

   1. DDD deals with large models by dividing them into different bounded contexts and being explicit about their interrelationships.

   2. A bounded context encapsulates the details of a single domain, such as domain model, data model, application services, and defines the integration points with other bounded contexts/domains. 

   3. This matches perfectly with the definition of a microservice, which is autonomous, well-defined interfaces, implementing a business capability. 

   4. This makes context mapping (and DDD, in general) an excellent tool in the architect’s toolbox for identifying and designing microservices.

   5. A bounded context encapsulates a single domain.

   6. DDD deals with large models by dividing them into different Bounded Contexts and being explicit about their interrelationships. 

   7. Anti-corruption Layer

      1. An Anti-corruption Layer is used to communicate between bounded contexts.

      2. It translates from one context to the other, so that data in each context reflects the language and the way that context treats the data.

      3. When one system needs to talk to another system and if these 2 systems are not compatible, you can use an intermediate layer which is capable of translating first systems communication to the protocol which is used by the second system and vice versa, this intermediate system is called as an Anti Corruption Layer since that layer helps to avoid the data corruption which would have happened otherwise. 

      4. This is not a new concept and the traditionally, Enterprise Service Bus (ESB) has been used as this intermediate layer. 

   8. 
      

5. A domain-driven design

   1. Defines the integration points with other domains

   2. Aligns well with characteristics of microservices

6. Ubiquitous Language

   1. Practice of building up a common, rigorous language between developers and users.

   2. A model acts as a Ubiquitous Language to help communication between software developers and domain experts. 

7. Various factors draw boundaries between contexts, the dominant being human culture.

8. Bounded contexts have unrelated concepts (ex. support tickets in a customer support context).

9. They also share concepts (ex. products and customers).

Polyglot Persistence

1. When storing data, it is best to use multiple data storage technologies, chosen based upon the way data is being used by individual applications or components of a single application. 

2. Each service must own its own data. 

Polyglot Programming

1. It is the idea that applications should be written in a mix of languages to take advantage of the fact that different languages are suitable for tackling different problems.

2. 
   

All interactions with AWS services are via the AWS API.

Use SigV4 to sign all interactions with the AWS API with AccessKeyId and SecretAccessKey.

Create a user in IAM, download the access key/secret access key

Use the aws configure command to configure the CLI tools to interface with your amazon account using the IAM user access key/secret access key, and default region (Default output format can be left blank)

Configured credentials can be found in ~/.aws/credentials

Region and other configuration parameters can be found in ~/.aws/config

The default credential provider chain looks for credentials in this order:

1. Specified in the code

2. Environment variables

3. Default credential profile in the credentials file

4. Amazon ECS container credentials

5. Amazon EC2 instance role

Example Java:

1. Public static void main(String[] args) throws IOException {

2. AmazonS3 s3client = new AmazonS3Client(new ProfileCredentialsProvider());

3. try {

4. System.out.println(“Uploading a new object to S3 from a file\n”);

5. File file = new File(uploadFileName);

6. s3client.putobject(new PutObjectRequest(

7. bucketname, keyName, file));

8. } catch (AmazonServiceException ase) {

9. System.out.println(“Error Message:   “ + ase.getmessage());

10. } catch (AmazonClientException ace) {

11. System.out.println(“Error Message:   “ + ace.getmessage());

12. }

How do users/applications authenticate?

1. If running in an integrated development environment (IDE), add credentials to the AWS Toolkit setup.

2. If using the AWS Management Console, sign in as an IAM user.

3. Amazon Cognito vends scoped, temporary credentials to untrusted environments such as web and mobile apps.

With the AWS SDK, you can use AWS services in your application by using your preferred programming language. 

SDKs are available in a number of programming languages and technology platforms, such as Android, iOS, Go, Java, JavaScript, .NET, Node.js, PHP, Python, and Ruby.

Levels

1. High-level API 

   1. Provides methods you need to perform an operation, use the high-level API because of its simplicity .

   2. Has one class per conceptual resource

   3. Defines service resources and individual resources

   4. Example; when uploading a file to an Amazon S3 bucket, the high-level API uses the file size to determine whether to upload the file in a single operation or upload it in multiple parts.

   5. Some AWS SDKs, such as the AWS SDK for Python (Boto3), provide higher-level APIs called the B. 

   6. Example: 

      1. def listResource():

      2.     s3resource = boto3.resource('s3')

      3.     bucket = s3resource.Bucket('mybucket')

      4.         for object in bucket.objects.all():

      5.             print(object.key, object.last_modified)

   7. Example Java:

      1. // High Level API – TransferManager Class

      2. File file = new File(file_path);

      3. TransferManager xfer_mgr = TransferManagerBuilder.standard().build();

      4. try {

      5.   Upload xfer = xfer_mgr.upload(bucketname, keyName, file);

      6.   XferMgrProgress.showTransferProgress(xfer);

      7.   XferMgrProgress.waitForCompletion(xfer);

      8. } catch (AmazonServiceException e) {

      9.   System.err.println(e.getErrorMessage());

      10.   System.exit(1);

      11. }

2. Low-Level API

   1. Set of client classes, each exposing a direct mapping of an AWS service’s API. 

   2. These client objects have a method for each operation that the service supports, with corresponding objects representing the request parameters and the response data.

   3. Full control over the requests 

   4. Example: 

      1. def listClient():

      2.     s3client = boto3.client('s3')

      3.     response = s3client.list_objects_v2(Bucket='mybucket')

      4.         for content in response['Contents']:

      5.             print(content['Key'],content['LastModified’])

   5.  Example Java:

      1. // Low level API

      2. 
         

      3. File file = new File(file_path);

      4. AmazonS3 s3client = new AmazonS3Client(new ProfileCredentialsProvider());

      5. try {

      6.      System.out.println(“Uploading a new object to S3 from a file\n”);

      7.      s3client.putobject(new PutObjectRequest(bucketname, keyName, file));

      8. } catch (AmazonServiceException ase) {

      9.      System.out.println(“Error Message: “ + ase.getmessage());

      10. } catch (AmazonClientException ace) {

      11.      System.out.println(“Error Message: “ + ace.getmessage());

      12. }

   6. 
      

APIs can change. Therefore, if you rely on a particular version of the API in your code, AWS recommends that you lock the service to the version number that you are using. 

1. [profile development]

2. aws_access_key_id=foo

3. aws_secret_access_key=bar

4. api_versions =

5. ec2 = 2015-03-01

6. cloudfront = 2015-09-17

7. 
   

Switching profiles is also supported through your IDE when paired with AWS Toolkits. 

You can use AWS services by running the AWS Command Line Interface (AWS CLI) commands from your terminal program. 

By default, the access key ID and the secret access key are associated with the IAM account that you use to access AWS resources. 

You can also specify a default AWS Region for all commands. 

Invoke locally:

1. Linux shells 

2. Windows command line

3. macOS 

Invoke remotely:

1. EC2 over SSH/PuTTY

2. AWS Cloud9: Which is authenticated with the permissions of the logged-in AWS user automatically. 

3. AWS CloudShell

4. AWS Systems Manager Session

Example: Creating a lambda function

1. aws lambda create-function --function-name ProcessDynamoDBRecords \

2. --zip-file fileb://function.zip --handler index.handler --runtime nodejs12.x \

3. --role arn:aws:iam::123456789012:role/lambda-dynamodb-role

A named profile is a collection of settings and credentials that you can apply to an AWS CLI command. When you specify a profile to run a command, the settings and credentials are used to run that command. 

Using AWS CLI, you can configure multiple named profiles as a collection of settings and credentials used to interact with your AWS services. 

With named profiles, you can switch between accounts, users, roles, and regions. 

IDEs, such as VS Code, Eclipse, Visual Studio, and JetBrains, support named profiles. 

To add additional profiles to your config and credentials files, run $ aws configure --profile. 

Global settings affect all services. By contrast, environment variables affect only AWS SDKs and tools.

 You can also store settings specific to Amazon S3 in the config file.

Locations for credentials and config files: 

2. 1. Linux, macOS, or Unix: 

      1. ~/.aws/credentials 

      2. ~/.aws/config

   2. Windows:

      1. C:\Users\USERNAME\.aws\credentials

      2. C:\Users\USERNAME\.aws\config

Credentials configuration example:

2. 1. >> aws configure

   2. AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE

   3. AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

   4. Default region name [None]: us-west-2

   5. Default output format [None]: json

   6. 
      

   7. >> aws configure --profile user1

   8. AWS Access Key ID [None]: AKIAI44QH8DHBEXAMPLE

   9. AWS Secret Access Key [None]: PsdaswtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

   10. Default region name [None]: us-east-1

   11. Default output format [None]: json

Environment variables file example:

1. .aws/config

2. [default]

3. region=us-west-2

4. output=json

5. [profile dev]

6. region=us-east-1

7. output=json

8. retry_mode=standard

9. max_attempts = 4

10. s3 =

11.      max_concurrent_requests = 20

12.      max_queue_size = 10000

13.      multipart_threshold = 64MB

14. api_versions =

15.   ec2 = 2015-03-01

16.   cloudfront = 2015-09-017

17. 
    

Example Profile with a lambda function:

1. aws lambda create-function --function-name

2.   someFunctionName --profile aws-lab-env

3. 
   

Content

Operation calls on AWS services can be synchronous or asynchronous.

Examples:

1. Lambda can be invoked synchronously or asynchronously.

2. Amazon S3 invokes functions asynchronously.

3. The CreateTable operation for DynamoDB is an asynchronous operation

4. 
   

Synchronous/Blocking 

1. Client makes a request and waits for the command to complete.

Asynchronous/Non-Blocking 

1. Client makes a request but does not wait for the command to be complete.

2. The waiter utility provides an API for handling the polling task. 

3. You can configure waiters with a maximum number of attempts and the back-off strategy between each attempt.

4. Examples:

   1. aws dynamodb describe-table --table-name Notes --query "Table.TableStatus"

   2. aws dynamodb wait table-exists --table-name Notes

   3. aws cloudformation cancel-update-stack --stack-name myteststack

5. Example 2: (checking a table .NET)

   1. // creates a table with “request” information such as table name

   2. var response = client.CreateTable(request);

   3. do

   4. {   System.Threading.Thread.Sleep(5000); // Wait 5 seconds.

   5.     try

   6.     {

   7.         var res = client.DescribeTable(new DescribeTableRequest

   8.                     { TableName = tableName  });

   9.         Console.WriteLine("Table name: {0}, status: {1}",

   10.                     res.Table.TableName,

   11.                     res.Table.TableStatus);

   12.         status = res.Table.TableStatus;

   13.     }

   14.     catch (ResourceNotFoundException)

   15.     { /* handle the potential exception. */ }

   16. } while (status != "ACTIVE");

   17. 
       

6. Example: (checking a table (Waiter) Python)

   1. # Create table and wait until it is ready

   2. # Get the service resource.

   3. dynamodb = boto3.resource('dynamodb')

   4. # Create the DynamoDB table.

   5. table = dynamodb.create_table(

   6.     TableName=‘Notes',

   7.     ...)

   8. # Wait until the table exists.

   9. table.meta.client.get_waiter('table_exists').wait(TableName=‘Notes')

   10. # Print out some data about the table.

   11. print(table.item_count)

7. Example: (checking a table (Waiter) Java

   1. //Create waiter to wait on successful creation of table.

   2. Waiter waiter = client.waiters().tableExists();

   3.      try{

   4.           waiter.run(new WaiterParameters<>(new DescribeTableRequest(tableName)); 

   5.      }

   6.      catch(WaiterUnrecoverableException e){

   7.           //Explicit short circuit undesired state.}

   8.      catch(WaiterTimedOutException e){

   9.           //Failed to transition into desired state after polling}

   10.      catch(DynamoDBException e){

   11.           //Unexpected service exception     }

   12. 
       

tjyt

AWS:

1. 400 series: Handle error in application

2. 500 series: Retry operation

Java and .NET SDK

1. AmazonServiceException or subclass:  Request was correctly transmitted to the service. However, the service was not able to process it and returned an error response instead. 

2. AmazonClientException: This exception indicates that a problem occurred inside the Java client code, either while trying to send a request to AWS or while trying to parse a response from AWS 

3. IllegalArgumentException – This exception is thrown if you pass an illegal argument when performing an operation on a service.

4. 
   

AWS SDK for Python (Boto3)

1. botocore.exceptions.ClientError

Available for multiple IDEs (Rider, Eclipse, WebStorm, Visual Studio Code, PyCharm, IntelliJ, AWS Cloud9, Azure DevOps)

Make it easier to create, debug, and deploy applications using AWS

Considerations:

1. (JVM) caches DNS name lookups. Because AWS resources use DNS name entries that occasionally change, we recommend that you configure your JVM with a TTL value of no more than 60 seconds. 

2. Environment variables provide another way to specify configuration options and credentials. They can be useful for scripting or temporarily setting a named profile as the default. 

3. AWS SDK implements an exponential backoff algorithm for better flow control. The idea behind exponential backoff is to use progressively longer waits between retries for consecutive error responses. 

AWS Cloud9 is an IDE that supports developers who develop for the cloud. 

With AWS Cloud9 you can write, run, and debug code using only a browser.

You can run your development environment on a managed EC2 instance or an existing Linux server that supports Secure Shell (SSH).

Isolate your project’s resources by maintaining multiple development environments. 

Programmers can simultaneously edit the same code because it has features that support pair programming.

AWS Cloud9 comes prepackaged with tools for popular programming languages, including Java, JavaScript, Python, PHP, .NET, and more. 

Credentials Options:

1. In an EC2 environment, we recommend creating IAM roles that provide temporary credentials and follow AWS best practices. 

2. Attach an IAM instance profile to the AWS Cloud9 instance. 

3. When using an SSH environment, you can store your access credentials within the environment. However, this method is considered less secure.

4. 
   

Use AWS Security Token Service (AWS STS) to request limited permissions, temporary credentials for IAM users, or for users that you authenticate through identity federation. 

1. Temporary credentials are the basis for roles. With AWS STS, you can specify the temporary credentials expiry interval. However, any service requests you make with expired credentials will fail, so you must request a new set of temporary credentials.

2. Temporary security credentials are not stored with the user. Temporary security credentials are generated dynamically and provided to the user only when requested.  

3. Web identity federation – You can allow users to sign in using a well-known third-party identity provider. Examples are: log in with Amazon, Facebook, Google, or any OpenID Connect (OIDC) 2.0 compatible provider. 

Example: Obtaining role credentials

1. sts_client = boto3.client('sts')

2. 
   

3. # Call the assume_role 

4. assumed_role_object=sts_client.assume_role(

5.     RoleArn="arn:aws:iam::account-of-role-to-assume:role/name-of-role",

6.     RoleSessionName="AssumeRoleSession1"

7. )

8. 
   

9. # Get the temporary credentials 

10. credentials=assumed_role_object['Credentials']

11. 
    

12. # Use the temporary credentials

13. s3_resource=boto3.resource(

14.     's3',

15.     aws_access_key_id=credentials['AccessKeyId'],

16.     aws_secret_access_key=credentials['SecretAccessKey'],

17.     aws_session_token=credentials['SessionToken'],

18. )

19. 
    

20. # Use the Amazon S3 resource object

21. for bucket in s3_resource.buckets.all():

22.     print(bucket.name)

23. 
    

Content

Amazon S3 defines two sets of Amazon S3 commands for working with AWS CLI: s3 and s3api.

1. >> aws configure get region

2. us-east-2

3. >> aws s3 ls

4. 2021-04-20 10:41:08 notes-bucket

5. >> aws s3 mb s3://lab-bucket --region us-west-1

6. make_bucket: lab-bucket

7. >> aws s3api list-buckets --query 'Buckets[].Name'

8. [

9. “lab-bucket",

10. “notes-bucket”

11. ]

12. >> aws s3api get-bucket-location --bucket lab-bucket

13. {  "LocationConstraint": "us-west-1“  }

14. >> aws s3api get-bucket-location --bucket notes-bucket

15. {  "LocationConstraint": null  }

Define the version

1. >>aws s3api get-bucket-versioning --bucket notes-bucket --generate-cli-skeleton output

2. {    "Status": "Status",    "MFADelete": "MFADelete"}

3. >>aws s3api put-bucket-versioning --bucket notes-bucket --versioning-configuration Status=Enabled

4.  >>aws s3api get-bucket-versioning --bucket notes-bucket

5. {    "Status": "Enabled"}

Retrieve Metadata:

1. aws s3api head-object --bucket notes-bucket --key index.html

2. {

3.     "AcceptRanges": "bytes",

4.     "ContentType": "text/html",

5.     "LastModified": "Thu, 16 Apr 2021 18:19:14 GMT",

6.     "ContentLength": 77,

7.     "VersionId": "null",

8.     "ETag": "\"30a6ec7e1a9ad79c203d05a589c8b400\"",

9.     "Metadata": {}

10. }

11. 
    

SDK

1. Steps

   1. Configure Amazon S3 settings for the SDK (Configuration details, Credentials information ) .

      1. [default]

      2. region = us-east-2

      3. output = json

      4. [profile Beta]

      5. region = us-west-2

      6. output = json

      7. s3 =

      8.     max_concurrent_requests = 20

      9. max_queue_size = 10000

      10. multipart_threshold = 64MB

      11. multipart_chunksize = 16MB

      12. max_bandwidth = 50MB/s

      13. addressing_style = path

      14. 
          

   2. Define dependencies (Python & Java Example).

      1. boto3.s3

      2. boto3.s3.transfer

      3. -----------------

      4. software.amazon.awssdk.services.s3

      5. software.amazon.awssdk.services.s3.model

      6. software.amazon.awssdk.services.s3.S3Client

      7. software.amazon.awssdk.services.s3.S3AsyncClient

   3. Create an S3 client (service reference) to make service requests (Python & Java Example). A session is an object that stores the configuration state, and you can pull this information to create the client 

      1. # create an S3 “low level” client interface

      2. s3client = boto3.client(‘s3’)

      3. # create an S3 “high level” resource interface

      4. s3resource = boto3.resource (‘s3’)

      5. ----------You can also manage your own session and create low-level clients or resource clients from it: 

      6. # Create S3 resource using custom session

      7. session = boto3.session.Session(profile_name='staging')

      8. # Retrieve region from session object

      9. current_region = session.region_name

      10. # Create a high-level resource from custom session

      11. resource = session.resource('s3')

      12. bucket = resource.Bucket('notes-bucket')

      13. # Region-specific endpoints require the LocationConstraint parameter (Create a Bucket)

      14. bucket.create(

      15.     CreateBucketConfiguration={

      16.         'LocationConstraint': current_region

      17. }

      18. )

      19. ----------------JAVA

      20. S3Client s3 = S3Client.builder()

      21.   .region(region)

      22.          .credentialsProvider(ProfileCredentialsProvider.create("profile_name"))

      23.   .build();

      24. //create a bucket

      25. s3Client.createBucket(new CreateBucketRequest(bucketName));

   4. Perform operations.

   5. “Close” the S3 client when the operations are completed.

   6. 
      

2. HeadBucket API

   1. The HeadBucket operation is an HTTP HEAD request to the bucket. It returns the headers that an HTTP GET operation would return.

   2. A HeadBucket request is useful in determining whether a bucket exists and if you have permission to access it. 

   3. If the bucket exists and you have permission to access it, the action returns a code 200 OK. 

   4. If the bucket does not exist returns a generic 404 Not Found or 403 Forbidden code. 

   5. Example Java:

      1. public static boolean bucketExisting(S3Client s3, String bucketName) {

      2. try {

      3.    //Create HeadBucket request to determine if bucket exists and you have permissions

      4.    HeadBucketRequest request = HeadBucketRequest.builder()

      5.           .bucket(bucketName)

      6.           .build();

      7.    HeadBucketResponse result = s3.headBucket(request);

      8.    if (result.sdkHttpResponse().statusCode() == 200) {System.out.println("Bucket existing! "); }

      9. }

      10. catch (AwsServiceException awsEx) {

      11.    switch (awsEx.statusCode()) {

      12.      case 404:

      13.        System.out.println("No such bucket existing.");

      14.      case 400 :

      15.        System.out.println(“Attempted to access a bucket from a Region other than where it exists.");

      16.      case 403 :

      17.        System.out.println("Permission errors in accessing bucket...");

      18.    }

      19. }

      20.   return;

      21. }

   6. Python Example

      1. def verifyBucketName(s3Client, bucket):

      2.     try:

      3.         ## check if a bucket already exists in AWS

      4.         s3Client.head_bucket(Bucket=bucket)

      5.         # If the previous command is successful, the bucket is already in your account.

      6.         raise SystemExit('This bucket has already been created')

      7.     except botocore.exceptions.ClientError as e:

      8.         error_code = int(e.response['Error']['Code'])

      9.         if error_code == 404:

      10.           ## If you receive a 404 error code, a bucket with that name

      11.           ##  does not exist anywhere in AWS.

      12.           print('Existing Bucket Not Found, please proceed')

      13.         if error_code == 403:

      14.           ## If you receive a 403 error code, a bucket with that name exists 

      15.           ## in another AWS account.

      16.           raise SystemExit('This bucket has already owned by another AWS Account') 

3. Create a Bucket:

   1. Python

      1. s3_client = boto3.client('s3', region_name=region)

      2. location = {'LocationConstraint': region}

      3. s3_client.create_bucket(Bucket=bucket_name, CreateBucketConfiguration=location) 

      4. ---Waiter

      5. waiter = s3Client.get_waiter('bucket_exists’) 

      6. waiter.wait(Bucket=bucket)

   2. Java

      1. if (!s3Client.doesBucketExistV2(bucketName)) {

      2.  s3Client.createBucket(new CreateBucketRequest(bucketName));

      3. 
         

      4.  String bucketLocation = s3Client.getBucketLocation(new GetBucketLocationRequest(bucketName));

      5.  System.out.println("Bucket location: " + bucketLocation); }

      6. ---Waiter

      7. WaiterResponse<HeadBucketResponse> waiterResponse =s3Waiter.waitUntilBucketExists(bucketRequestWait);

      8. waiterResponse.matched().response().ifPresent(System.out::println);

4. Manage Objects

   1. Get

      1. Get object and metadata:  GetObject (can also be used to return a range of bytes)

      2. Get object metadata: HeadObject, GetObjectAcl, GetObjectTagging

      3. Python Example:

         1. def get_object(bucket, object_key):

         2.   try:

         3.       obj = s3.Object(bucketname, object_key)

         4.       body = obj.get()['Body'].read()

         5.       logger.info("Got object '%s' from bucket '%s'.", object_key, bucket.name)

         6.   except ClientError:

         7.       logger.exception(("Couldn't get object '%s' from bucket '%s'.",

         8.                         object_key, bucket.name))

         9.       raise

         10.   else:

         11.       return body

         12. 
             

      4. Java Example:

         1. ResponseHeaderOverrides headerOverrides = new ResponseHeaderOverrides()

         2.     .withCacheControl("No-cache")

         3.     .withContentDisposition("attachment; filename=example.txt");

         4. GetObjectRequest getObjectRequestHeaderOverride = new GetObjectRequest(bucketName, key)

         5.      .withResponseHeaders(headerOverrides);

         6. headerOverrideObject = s3Client.getObject(getObjectRequestHeaderOverride);

         7. displayTextInputStream(headerOverrideObject.getObjectContent());

         8. 
            

   2. AWS Select:

      1. Amazon S3 Select analyzes and processes data in an object in Amazon S3 buckets faster and cheaper than the GET method. 

      2. With GET, you retrieve the whole object in an Amazon S3 bucket. 

      3. With SELECT, you can retrieve a subset of data from an object by using simple SQL expressions. 

      4. Your applications do not have to use compute resources to scan and filter the data from an object. 

      5. Python Example:

         1. import boto3

         2. s3 = boto3.client('s3')

         3. r = s3.select_object_content(

         4.         Bucket='jbarr-us-west-2',

         5.         Key='sample-data/airportCodes.csv',

         6.         ExpressionType='SQL',

         7.         Expression="select * from s3object s where s.\"Country (Name)\" like '%United States%'",

         8.         InputSerialization = {'CSV': {"FileHeaderInfo": "Use"}},

         9.         OutputSerialization = {'CSV': {}},

         10. )

         11. for event in r['Payload']:

         12.     if 'Records' in event:

         13.         records = event['Records']['Payload'].decode('utf-8')

         14.         print(records)

         15.     elif 'Stats' in event:

         16.         statsDetails = event['Stats']['Details']

         17.         print("Stats details bytesScanned: ")

   3. Generate Presigned URL

      1. Python URL:

         1. import boto3

         2.     url = boto3.client('s3').generate_presigned_url(

         3.     ClientMethod='get_object', 

         4.     Params={'Bucket': ‘notes-bucket', 'Key': 'OBJECT_KEY'},

         5.     ExpiresIn=3600)

      2. Java URL:

         1. GeneratePresignedUrlRequest generatePresignedUrlRequest =

         2.         new GeneratePresignedUrlRequest(“notes-bucket”, objectKey)

         3.     .withMethod(HttpMethod.GET)

         4.     .withExpiration(expiration);

         5. URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);

         6. 
            

5. Bulk Operations

   1. High-level commands automatically handle multipart uploads and cleanup of incomplete uploads.

      1. Examples:

         1. aws s3 cp ./aFIle.txt s3://notes-bucket/docs/

         2. aws s3 sync s3://notes-bucket s3://other-bucket --exclude "*another/*"

6.  Iterate through the Bucket

   1. Example Java:

      1. List<Bucket> buckets = s3.listBuckets();

      2. System.out.println("Your {S3} buckets are:");

      3. for (Bucket b : buckets) {

      4.     System.out.println("* " + b.getName());}

   2. Example Python:

      1. # Retrieve the list of existing buckets

      2. response = s3.list_buckets()

      3. # Output the bucket names

      4. print('Existing buckets:')

      5. for bucket in response['Buckets']:

      6.     print(f'  {bucket["Name"]}')

7. Paging

   1. Some AWS operations return results that are paginated, meaning that the results are listed one page at a time. 

   2. Using a continuation token, you can process the next set of paginated results. 

   3. For example, listing the contents of your Amazon S3 bucket returns paginated results, up to 1000 objects at a time. 

   4. In the response, pay attention to this value:

      1. IsTruncated – A value of true indicates that the response is incomplete, and it was truncated to the number of objects specified by MaxKeys.

      2. ContinuationToken – If sent with the request, it is included in the response.

      3. NextContinuationToken – Sent when IsTruncated is true, which means that the bucket contains more keys that can be listed. In the request, use the value of the NextContinuationToken as the value for new continuation-token, to continue the next list requests to Amazon S3. 

   5. Example Java:

      1. ListObjectsV2Request listReq = ListObjectsV2Request.builder()

      2.             .bucket(bucketName).maxKeys(1).build();

      3. ListObjectsV2Iterable listRes = s3.listObjectsV2Paginator(listReq);

      4. // Process response pages

      5. listRes.stream()

      6.         .flatMap(r -> r.contents().stream())

      7.         .forEach(content -> System.out.println(" Key: " + content.key() ));

   6. Example Python:

      1. paginator = client.get_paginator('list_objects')

      2. page_iterator = paginator.paginate(Bucket='notes-bucket',

      3.                                    PaginationConfig={'MaxItems': 10})

      4. for page in page_iterator:

      5.     print(page['Contents'])

      6. 
         

Static Website

1. You can configure an S3 bucket as a static website. A static website contains static resources such as HTML or images, but it does not contain server-side processing or scripting.

2. Command:  aws s3 website s3://notes-bucket/ --index-document index.html --error-document error.html 

3. 
   

Interfaces

1. Object persistence interface

   1. The AWS SDK for .NET provides an object persistence model that you can use to map your client-side classes to Amazon DynamoDB tables. 

   2. Each object instance then maps to an item in the corresponding tables. 

   3. To save your client-side objects to the tables, the object persistence model provides the DynamoDBContext class an entry point to DynamoDB. This class provides you with a connection to DynamoDB. You can then access tables, perform various CRUD operations, and run queries.

2. Document interface

   1. Many AWS SDKs provide a document interface that you can use to perform data plane operations (create, read, update, delete) on tables and indexes. With a document interface, you do not need to specify data type descriptors. 

   2. The data types are implied by the semantics of the data itself. 

   3. These AWS SDKs also provide methods to convert JSON documents to and from native Amazon DynamoDB data types.

3. Low-level interface

   1. Every language-specific AWS SDK provides a low-level interface for Amazon DynamoDB, with methods that closely resemble low-level DynamoDB API requests.

DynamoDB Client Example

1. Java Normal Client Builder

   1. DynamoDbClient client = DynamoDbClient.builder()

   2.     .region(Region.US_WEST_2)

   3.     .credentialsProvider(ProfileCredentialsProvider.builder()

   4.         .profileName("myProfile")

   5.         .build())

   6.     .build(); 

2. Java Local Client Builder

   1. DynamoDbClient client = DynamoDbClient.builder()

   2.     .endpointOverride(URI.create("http://localhost:8000"))

   3. // The region is meaningless for local DynamoDb but required for client builder validation

   4.     .region(Region.US_EAST_1)

   5.     .credentialsProvider(StaticCredentialsProvider.create(

   6.     AwsBasicCredentials.create("dummy-key", "dummy-secret")))

   7.     .build();

3. Java Default Client

   1. DynamoDbClient client = DynamoDbClient.create();

4. Python Client

   1. import boto3

   2. 
      

   3. # Get the service resource.

   4. dynamodb = boto3.client('dynamodb')

   5. 
      

   6. # Create the DynamoDB table.

   7. table = dynamodb.create_table(

   8.     TableName='Notes',

   9.     KeySchema=[…],

   10.     AttributeDefinitions=[…],

   11.     ProvisionedThroughput={…}

   12. )

DynamoDB Create Table

1. Java

   1. //Set Key attributes and values to be used in request

   2. List<KeySchemaElement> keySchema = new ArrayList<KeySchemaElement>();

   3. keySchema.add(KeySchemaElement.builder().attributeName(”UserId”).keyType(KeyType.HASH).build());

   4. keySchema.add(KeySchemaElement.builder().attributeName(“NoteId”).keyType(KeyType.RANGE).build());

   5. //Set throughput on table to be used in request

   6. ProvisionedThroughput provisionedThroughput = ProvisionedThroughput.builder()

   7.       .writeCapacityUnits(5L)

   8.       .readCapacityUnits(5L)

   9.       .build();

   10. 
       

   11. //Build CreateTable request

   12. CreateTableRequest request = CreateTableRequest.builder()

   13.       .attributeDefinitions(attributeDefinitions)

   14.       .keySchema(keySchema)

   15.       .provisionedThroughput(provisionedThroughput)

   16.       .tableName(“Notes”)

   17.       .build();

   18. //Create table using request object

   19. CreateTableResponse response = ddb.createTable(request);

   20. 
       

2. Command

   1. >> aws dynamodb create-table --cli-input-json file://notestable.json --region us-west-2

   2. JSON

      1. {

      2.     "AttributeDefinitions": [

      3.             {"AttributeName": "UserId",

      4.             "AttributeType": "S"},

      5.             {"AttributeName": "NoteId",

      6.             "AttributeType": "N"},

      7.             {"AttributeName": "Is_Incomplete",

      8.                 "AttributeType": "S"}],

      9.     "TableName": "Notes",

      10.     "KeySchema": [

      11.             {"AttributeName": "UserId",

      12.             "KeyType": "HASH"},

      13.             {"AttributeName": "NoteId",

      14.             "KeyType": "RANGE"}],

      15.     "ProvisionedThroughput": {

      16.             "ReadCapacityUnits": 1,

      17.             "WriteCapacityUnits": 1},

      18.     "LocalSecondaryIndexes": [

      19.             {"IndexName": "Review",

      20.             "KeySchema": [

      21.                 {"AttributeName": "UserId",

      22.                 "KeyType": "HASH"},

      23.                 {"AttributeName": "Is_Incomplete",

      24.                 "KeyType": "RANGE"}],

      25.             "Projection": 

      26.             {"ProjectionType": "ALL"

      27.         }}]}

      28. 
          

Using a  waiter (JAVA)

1. //Define a waiter on DynamoDB client

2. DynamoDbWaiter dbWaiter = ddb.waiter();   

3. //Create table using request object

4. CreateTableResponse response = ddb.createTable(“Notes");

5. // Wait until the Amazon DynamoDB table is created

6. WaiterResponse<DescribeTableResponse> waiterResponse = dbWaiter.waitUntilTableExists(“Notes");

7. waiterResponse.matched().response().ifPresent(System.out::println);

Update Table

1. Java

   1. Table table = dynamoDB.getTable(“Notes");

   2. 
      

   3. ProvisionedThroughput  = new ProvisionedThroughput()

   4.     .withReadCapacityUnits(15L)

   5.     .withWriteCapacityUnits(15L);

   6. 
      

   7. table.updateTable(provisionedThroughput);

   8. 
      

Delete Table

1. Java

   1. Table table = dynamoDB.getTable(“Notes");

   2. 
      

   3. table.delete();

   4. 
      

List Table

1. Java

   1. TableCollection<ListTablesResult> tables = dynamoDB.listTables();

   2. Iterator<Table> iterator = tables.iterator();

   3. 
      

   4. while (iterator.hasNext()) { Table  = iterator.next();

   5.     System.out.println(table.getTableName());}

Create Item

1. Command PUT

   1. >> aws dynamodb put-item

   2.     --table-name Notes

   3.     --item \

   4. '{"Userid":{"S":"StudentD"}, "NoteId":{"N":"42"}, "Notes":{"S":"Test note"}}'

2. Command Batch Write

   1. Batch operations read and write items in parallel to minimize response latencies.

      1. BatchGetItem – Read up to 100 items from one or more tables up to 16 MB of data. 

      2. BatchWriteItem – Create or delete up to 25 items in one or more tables up to 16 MB of data.

   2. Example:

      1. aws dynamodb batch-write-item \

      2.     --request-items file://request-items.json

Read Items

1. Command GET: To read an item from a DynamoDB table, use the GetItem operation. 

   1. >> aws dynamodb get-item

   2. --table-name Notes

   3. --key '{"UserId": {"S": "StudentA"}, "NoteId": {"N": "11"}}'

   4. {

   5.     "Item": {

   6.         "Note": {

   7.          "S": "Hello World\n"

   8.         },

   9.         "NoteId": {

   10.          "N": "11"

   11.         },

   12.         "UserId": {

   13.          "S": "StudentA"

   14.         }

   15.     }

   16. }

   17. 
       

2. Query:  If you only need one item, use getItem. For a collection of items, use query and use filter expressions or Limit. 

   1. Must include a key condition expression
      =, <, >, <=, >=, AND, BETWEEN, and begins_with

   2. The Query operation in Amazon DynamoDB finds items based on primary key values.

   3. DynamoDB paginates the results from Query operations. 

   4. By default, DynamoDB returns a result only 1 MB in size or less. 

   5. Results are divided into pages. 

   6. Determine whether there are more results.

      1. Check for LastEvaluatedKey element.

      2. Absence of LastEvaluatedKey indicates that there are no additional items to retrieve. 

      3. Command: 

      4. aws dynamodb query

      5. --table-name Notes

      6. --key-condition-expression "UserId = :userid"

      7. --expression-attribute-values ‘{":userid":{"S":"StudentA”}’

      8. 
         

      9. {

      10.     "Count": 8,

      11.     "Items": [

      12.       {"UserId": {"S": "StudentA"}},

      13.       {"UserId": {"S": "StudentB"}},

      14.       {"UserId": {"S": "StudentC"}},

      15.       {"UserId": {"S": "StudentD"}},

      16.       {"UserId": {"S": "StudentE"}},

      17.       {"UserId": {"S": "StudentF"}},

      18.       {"UserId": {"S": "StudentG"}},

      19.       {"UserId": {"S": "StudentH"}}

      20.     ],

      21.     "LastEvaluatedKey": {

      22.       "UserId": {"S": "StudentH"},

      23.       "NoteId": {"N": "88"}

      24.     },

      25.     "ScannedCount": 8

      26.   }

   7. By default, a Query operation does not return any data on how much read capacity it consumes. 

      1. However, you can specify the ReturnConsumedCapacity parameter in a Query request to obtain this information. The following are the valid settings for ReturnConsumedCapacity.

         1. NONE—no consumed capacity data is returned. (This is the default).

         2. TOTAL—the response includes the aggregate number of read capacity units consumed.

         3. INDEXES—the response shows the aggregate number of read capacity units consumed, together with the consumed capacity for each table and index that was accessed.

3. Scan: 

   1. Returns a result set

   2. Maximum of 1 MB of data retrieved

   3. Because a Scan operation reads an entire page (by default, 1 MB), you can reduce the scan operation's impact by setting a smaller page size. 

   4. Filter expressions are applied after a scan finishes but before results are returned

   5. A Scan operation in Amazon DynamoDB reads every item in a table or a secondary index.

   6. In general, scans are less efficient than queries. 

   7. A Scan operation in Amazon DynamoDB reads every item in a table or a secondary index. 

   8. Therefore, a Scan consumes the same amount of read capacity, regardless of whether a filter expression is present. 

   9. The typical DynamoDB scan operation processes data sequentially. For large tables, this can cause the scan operation to take a long time to complete. 

   10. AWS DynamoDB provides a parallel scan operation that allows for multiple threads or workers to scan different sections or segments of a table simultaneously. This can improve scan performance.  

   11. Command Example:

       1. aws dynamodb scan

       2. --table-name Notes

       3. --key-condition-expression "UserId = :userid"

       4. --expression-attribute-values ‘{":userid":{"S":"StudentA"}}’

       5. 
          
4. 
   

Update

1. Command:

   1. UpdateItem updates only passed attributes.

   2. >> aws dynamodb update-item

   3.     --table-name Notes

   4.     --key '{"UserId": {"S": "StudentC"}, "NoteId": {"N": “12"}, “Favorite": {“S: “Yes"}}‘

   5. Conditional writes succeed only if item attributes meet one or more expected conditions. 

   6. >> aws dynamodb update-item

   7.     --table-name Notes

   8.     --key '{"UserId": {"S": "StudentD"}, "NoteId": {"N": “42"}}’

   9.     --update-expression “SET Notes = :newnote”

   10.     --condition-expression “Favorite NOT yes”

   11.     --expression-attribute-values file://expression-attribute-values.json

Delete items

1. Command:

   1. >> aws dynamodb delete-item

   2.     --table-name Notes

   3.     --key

   4. '{"UserId": {"S": "StudentB"}, "NoteId": {"N": “23"}, "Notes": {“S”: “AWS T&C"}}‘

2. 
   

Mapping a Table with a Class:

1. In this code example, the @DynamoDBTable annotation maps the NotesItems class to the Notes table. 

2. Java

   1. @DynamoDBTable(tableName="Notes")

   2.     public static class NotesItems {

   3.         //Set up Data Members that correspond to columns in the Notes table

   4.         private String UserId;

   5.         private Integer NoteId;

   6.         private String Notes;

   7. 
      

   8.         @DynamoDBHashKey(attributeName="UserId")

   9.         public String getUserId() { return this.UserId; }

   10.         public void setUserId(String UserId) { this.UserId = UserId; }

   11. 
       

   12.         @DynamoDBRangeKey(attributeName="NoteId")

   13.         public Integer getNoteId() { return this.NoteId; }

   14.         public void setNoteId(Integer NoteId) { this.NoteId = NoteId; }

   15. 
       

   16.         @DynamoDBAttribute(attributeName="Notes")

   17.         public String getNotes() { return this.Notes; }

   18.         public void setNotes(String Notes) { this.Notes = Notes; }

   19. 
       

   20.         @Override

   21.         public String toString() {

   22.             return "Notes [User=" + UserId + ", NoteId=" + NoteId + ", Notes=" + Notes + "]";

   23.         }

   24.     } 

   25. s

Complete Example

1. Domain Object

   1. @DynamoDBTable(tableName = “CloudAirTripSectors”)

   2. public class TripSector implements Serializable {

   3.     @DynamoDBHashKey

   4.     private Long date;

   5.     @DynamoDBRangeKey

   6.     @DynamoDBIndexHashKey(globalSecondaryIndexName = ORIGIN_CITY_INDEX)

   7.     private String originCity;

   8. 
      

   9.     public TripSector() { }

   10. }

2. DynamoDB mapper

   1. @Override

   2. Public List<TripSector> findTripsToCity(String city) {

   3.     Map<String, AttributeValue> eav =  new Hashmap<>();

   4.     eav.put(“:v1” , new AttributeValue().withS(city));

   5. 
      

   6.     DynamoDBQueryExpression<TripSector> query = new DynamoDBQueryExpression<TripSector>()

   7. .withIndexName(TripSector.DESTINATION_CITY_INDEX).withConsistentRead(false)

   8. .withKeyConditionExpression(“destinationCity =:v1”).withExpressionAttributeValues(eav);

   9.     return mapper.query(TripSector.class, query);

   10. }

3. Lambda handler

   1. public  class FindTripsToCityHandler 

   2.     extends LambdaHandlerWithXRayBase 

   3.     implements RequestHandler<TripSearchRequest, LambdaResult<List<TripSector>>>
      {

   4.   @Override

   5.     public LambdaResult<List<TripSector>> handleRequest(TripSearchRequest input, Context context) {

   6.         

   7.        LambdaResult<List<TripSector>> result = new LambdaResult<List<TripSector>>();

   8.         LambdaLogger logger = context.getLogger();

   9.         logger.log("Starting " + this.getClass().getName() + " Lambda with city " + input.getCity() + "\n");

   10. 
               try {

   11.             List<TripSector> listTripSector = DynamoDBTripDao.instance().findTripsToCity(input.getCity()); 

   12.             result.setData(listTripSector);

   13.             logger.log("Found " + listTripSector.size() + " total trips.\n");

   14.             

   15.             for(TripSector item : listTripSector){

   16.                 System.out.println(item.toString());

   17.             }
                   result.setSucceeded(true);

   18.             return result;

   19.         } catch (Exception e) {

   20.             result.setSucceeded(false);

   21.             result.setErrorCode(1);

   22.             result.setMessage(e.getMessage());

   23.             e.printStackTrace();

   24.         }
               return result;

   25.     }
       }

   26. 
       
   27. 
       

DynamoDB Accelerator (DAX)

1. Supported read operations

   1. GetItem, BatchGetItem, Query, Scan

2. Supported write operations

   1. BatchWriteItem, UpdateItem, DeleteItem, PutItem

3. DAX maintains an item cache to store the results from GetItem and BatchGetItem operations. 

4. For strongly consistent read request from an application, DAX Cluster pass all request to DynamoDB & does not cache for these requests.

5. 
   

Sharding write-heavy partition keys

1. One way to better distribute writes across a partition key space in Amazon DynamoDB is to expand the space. 

2. You can do this in several different ways. 

   1. You can add a random number to the partition key values to distribute the items among partitions. 

      1. Example

         1. For a partition key that represents today's date, you might choose a random number between 1 and 200 and concatenate it as a suffix to the date. 

         2. This yields partition key values like 2014-07-09.1, 2014-07-09.2, and so on, through 2014-07-09.200. 

         3. However, to read all the items for a given day, you would have to query the items for all the suffixes and then merge the results. 

         4. For example, you would first issue a Query request for the partition key value 2014-07-09.1. Then issue another Query for 2014-07-09.2, and so on, through 2014-07-09.200. Finally, your application would have to merge the results from all those Query requests. 

   2. Or you can use a number that is calculated based on something that you're querying on. 

      1. Now suppose that each item has an accessible OrderId attribute, and that you most often need to find items by order ID in addition to date. 

      2. Before your application writes the item to the table, it could calculate a hash suffix based on the order ID and append it to the partition key date. The calculation might generate a number between 1 and 200 that is fairly evenly distributed, similar to what the random strategy produces. 

      3. You can easily perform a GetItem operation for a particular item and date because you can calculate the partition key value for a specific OrderId value. 

      4. To read all the items for a given day, you still must Query each of the 2014-07-09.N keys (where N is 1–200), and your application then has to merge all the results. The benefit is that you avoid having a single "hot" partition key value taking all of the workload. 

Scaling read-heavy partition keys

1. Provisioned I/O capacity for the table is divided evenly among these physical partitions. 

2. Therefore, a partition key design that doesn't distribute I/O requests evenly can create "hot" partitions that result in throttling and use your provisioned I/O capacity inefficiently. 

3. Example:

   1. Some products might be popular among customers, so those items would be consistently accessed more frequently than the others.

   2. As a result, the distribution of read activity on ProductCatalog would be highly skewed toward those popular items. 

4. f

Best Practice

1. Each Query or Scan request with a smaller page size uses fewer read operations and creates a "pause" between each request. 

   1. For example, suppose that, each item is 4 KB, and you set the page size to 40 items. A Query request would then consume only 20 eventually consistent read operations or 40 strongly consistent read operations. 

AWS Serverless The AWS Serverless Application Model (SAM) is an open-source framework for building serverless applications. 

AWS SAM command line interface (AWS SAM CLI)

1. Verify that AWS SAM template files are written according to the specification.

2. Invoke Lambda functions locally.

3. Step through debug Lambda functions.

4. Package and deploy serverless applications to the AWS Cloud.

A serverless application is a combination of Lambda functions, event sources, and other resources that work together to perform tasks. 

It provides shorthand syntax to express functions, APIs, databases, and event source mappings. 

With just a few lines per resource, you can define the application you want and model it using YAML. During deployment, SAM transforms and expands the SAM syntax into AWS CloudFormation syntax, enabling you to build serverless applications faster. 

AWS SAM template specification. 

1. You use this specification to define your serverless application. 

2. It provides you with a simple and clean syntax to describe the functions, APIs, permissions, configurations, and events that make up a serverless application. 

AWS SAM command line interface (AWS SAM CLI). 

1. You use this tool to build serverless applications that are defined by AWS SAM templates. 

2. The CLI provides commands that enable you to verify that AWS SAM template files are written according to the specification, invoke Lambda functions locally, step through debug Lambda functions, package and deploy serverless applications to the AWS Cloud, and so on.

3. Tests serverless applications locally

4. Generates code blueprints to bootstrap development

5. Provides response object and function logs

6. Uses open source Docker-Lambda images

7. Emulates timeout, memory limits, and runtimes 

8. Verify that AWS SAM template files are written according to the specification.

9. Invoke Lambda functions locally.

10. Step through debug Lambda functions.

11. Package and deploy serverless applications to the AWS Cloud.

12. Install/upgrade

    1. pip install --upgrade aws-sam-cli

AWS SAM with AWS toolkits and debuggers to test and debug locally.

Consistently provision resources in multiple environments. Model (AWS SAM)

Resource types: 

1. Lambda functions 

2. API Gateway APIs 

3. DynamoDB tables  

Variables

1. Lambda environment variables

2. API stage variables

Open specification (Apache 2.0)

The optional Transform section specifies one or more macros that AWS CloudFormation uses to process your template.

1. AWSTemplateFormatVersion: '2010-09-09'
   Transform: 'AWS::Serverless-2016-10-31'

2. Resources: …

Sam resources mapping

1. Serverless API = Amazon API Gateway

2. Serverless Function = AWS Lambda

3. Serverless SimpleTable = Amazon DynamoDB

4. Serverless StateMachine = AWS Step Functions

Resources

1. AWS::Serverless::Api

   1. Creates a collection of Amazon API Gateway resources and methods that can be invoked through HTTPS endpoints. 

   2. An AWS::Serverless::Api resource need not be explicitly added to a AWS Serverless Application Definition template. A resource of this type is implicitly created from the union of Api events defined on AWS::Serverless::Function resources defined in the template that do not refer to an AWS::Serverless::Api resource. 

2. AWS::Serverless::Application

   1. Embeds a serverless application from the AWS Serverless Application Repository or from an Amazon S3 bucket as a nested application. 

   2. Nested applications are deployed as nested AWS::CloudFormation::Stack resources, which can contain multiple other resources including other AWS::Serverless::Application resources. 

3. AWS::Serverless::Function

   1. Creates an AWS Lambda function, an  (IAM) execution role, and event source mappings that trigger the function.

   2. The AWS::Serverless::Function resource also supports the Metadata resource attribute, so you can instruct AWS SAM to build custom runtimes that your application requires.

4. AWS::Serverless::HttpApi

   1. Creates an Amazon API Gateway HTTP API, which enables you to create RESTful APIs with lower latency and lower costs than REST APIs. 

5. AWS::Serverless::LayerVersion

   1. Creates a Lambda LayerVersion that contains library or runtime code needed by a Lambda Function. 

6. AWS::Serverless::SimpleTable

   1. Creates a DynamoDB table with a single attribute primary key. It is useful when data only needs to be accessed via a primary key.

   2. To use the more advanced functionality of DynamoDB, use an AWS::DynamoDB::Table resource instead.

7. AWS::Serverless::StateMachine

   1. Creates an AWS Step Functions state machine, which you can use to orchestrate AWS Lambda functions and other AWS resources to form complex and robust workflows. 

AWS SAM translates the resources in its template (YAML or JSON) into an equivalent AWS CloudFormation template. 

1. The CloudFormation template is then applied to create and update AWS resources.

2. Example:

   1. AWSTemplateFormatVersion: '2010-09-09'

   2. Transform: AWS::serverless-2016-10-31

   3. Globals: //Global Variables

   4.   Api:

   5.     MethodSettings:

   6.       - LoggingLevel: INFO

   7. Resources:  //Lambda Function

   8.   listFunction:

   9.     Type: AWS::Serverless::Function

   10.     Properties:

   11.       CodeUri: list-function/

   12.       Handler: app.lambda_handler

   13.       Runtime: python3.8

   14.       Policies: AmazonDynamoDBReadOnlyAccess

   15.       Events:

   16.         listNotes:

   17.           Type: Api

   18.           Properties:

   19.             Path: /notes

   20.             Method: get

Example -  AWS::Serverless::Function

1. AWSTemplateFormatVersion: '2010-09-09'
   Transform: 'AWS::Serverless-2016-10-31‘
   Resources:
   MySimpleFunction:
   Type: AWS::Serverless::Function
   Properties:
   Handler: index.handler
   Runtime: nodejs12.x
   CodeUri: s3://<source-bucket>/MyCode.zip
   Policies: AmazonDynamoDBFullAccess
   Environment:
   Variables:
   TABLE_NAME: !Ref Table
   Events:
   MyUploadEvent:
   Type: S3
   Properties:
   Id: !Ref MyBucket
   Events: Create
   MyBucket:
   Type: AWS::S3::Bucket

Example 2 - AWS::Serverless::API - Implicit Expression :

1. AWSTemplateFormatVersion: '2010-09-09'
   Transform: 'AWS::Serverless-2016-10-31'
   Resources:
     GetFunction:
       Type: AWS::Serverless::Function
         Properties:
           Handler: index.get
           Runtime: nodejs12.x
           CodeUri: s3://bucket/api_backend.zip
           Policies: AmazonDynamoDBReadOnlyAccess
           Environment:
             Variables:
               TABLE_NAME: !Ref Table
          Events:
             GetResource:
               Type: Api
               Properties:
                 Path: /resource/{resourceId}
                 Method: get

Example 3 - AWS::Serverless::API - Explicit Expression:

1. AWSTemplateFormatVersion: '2010-09-09'
   Transform: 'AWS::Serverless-2016-10-31' 

2. Resources: 

3. Api:

4.  Type: AWS::Serverless::Api

5.     Properties: 

6.     StageName: prod 

7.      DefinitionUri: swagger.yml

Example 4 - AWS::Serverless::SimpleTable - Simple Table:

1. AWSTemplateFormatVersion: '2010-09-09'
   Transform: 'AWS::Serverless-2016-10-31' 

2. Resources: 

3.   MyTable: 

4.     Type: AWS::Serverless::SimpleTable 

5.     Properties: 

6.       PrimaryKey: 

7.         Name: id 

8.         Type: String 

9.       ProvisionedThroughput: 

10.         ReadCapacityUnits: 5 

11.         WriteCapacityUnits: 5

Deploying Application

1. ZIP the function files.

2. Upload it to an Amazon S3 bucket.

3. Add a CodeUri property, specifying the location of the .zip file in the bucket for each function in appspec.yml.

4. Call the AWS CloudFormation CreateChangeSet operation with appspec.yml.

   1. aws cloudformation package --template-file app_spec.yml --output-template-file new_app_spec.yml --s3-bucket <your-bucket-name>

5. Call the AWS CloudFormation ExecuteChangeSet operation with the name of the change set you created in step 04.

   1. aws cloudformation deploy --template-file new_app_spec.yml --stack-name <your-stack-name> --capabilities CAPABILITY_IAM

Canary Deployment Example:

1. Resources:
     MyLambdaFunction:
       Type: AWS::Serverless::Function
       Properties:
         Handler: index.handler
         Runtime: nodejs12.x
         CodeURI: s3://bucket/codezip

2.       AutoPublishAlias: Live
         DeploymentPreference:
          Type: Canary10Percent10Minutes

3. 
           Alarms:
             # A list of alarms that you want to monitor
             - !Ref AliasErrormetricGreaterThanZeroAlarm
             - !Ref latestVersionErrormetricGreaterThanZeroAlarm

4.         Hooks:
             # Validation Lambda functions that are run
             # before & after traffic shifting
             PreTraffic:  ! Ref PreTrafficLambdaFunction
             PostTraffic: ! Ref PostTrafficLambdaFunction

5. 
   

Lambda Deployment Types:

1. Canary: Traffic is shifted in two increments. 

   1. The options specify the percentage of traffic that's shifted to your updated Lambda function version in the first increment, and the interval, in minutes, before the remaining traffic is shifted in the second increment.

   2. Linear: Traffic is shifted in equal increments with an equal number of minutes between each increment. 

   3. All-at-once: All traffic is shifted from the original Lambda function to the updated Lambda function version at once.

2. Types:

   1. Canary10Percent30Minutes

   2. Canary10Percent5Minutes

   3. Canary10Percent10Minutes

   4. Canary10Percent15Minutes

   5. Linear10PercentEvery10Minutes

   6. Linear10PercentEvery1Minute

   7. Linear10PercentEvery2Minutes

   8. Linear10PercentEvery3Minutes

   9. AllAtOnce

SAM deployment and Build process

1. Step 1: Prepare the SAM application

   1. Place the function code at the root level of the working directory along with the YAML file

   2. sam-app/

   3.    ├── README.md

   4.    ├── events/

   5.    │   └── event.json

   6.    ├── hello_world/

   7.    │   ├── __init__.py

   8.    │   ├── app.py  #Lambda handler logic.

   9.    │   └── requirements.txt  #Dependencies

   10.    ├── template.yaml   #AWS SAM template 

   11.    └── tests/

   12.        └── unit/

   13.            ├── __init__.py

   14.            └── test_handler.py

2. Step 2: Build your application

   1. First, change into the project directory, where the template.yaml file for the sample application is located.

   2.  The sam build command builds any dependencies that your application has, and copies your application source code to folders under .aws-sam/build

      1. Command: sam build

   3. Output;

      1.  .aws-sam/

      2.    └── build/

      3.        ├── HelloWorldFunction/

      4.        └── template.yaml

3. Step 3: Deploy your application to the AWS Cloud

   1. Command: sam deploy --guided

4. Step 4: (Optional) Test your application locally

   1. Command: sam local start-api

   2. curl http://127.0.0.1:3000/hello

Local Testing

1. sam local invoke

   1. Run an AWS Lambda function locally in a Docker container.

2. sam local start-api

   1. Replicate an Amazon API Gateway endpoint locally.

3. sam local generate-event

   1. Generate sample payloads from different event sources.

   2. Supported services include Amazon S3, API Gateway, and Amazon SNS

Workflow

1. 1.Init – To initialize a new AWS SAM project, use the sam init command. You can choose a built-in application template or a custom template.

   1. >>sam init

   2. >>sam init --runtime python3.8 --app-template hello-world

   3. >>sam init --location /path/to/template.zip

   4. >>sam init --location https://example.com/path/to/template.zip

   5. Which template source would you like to use?

   6.         1 - AWS Quick Start Templates

   7.         2 - Custom Template Location

   8. Choice:_

2. Package (opt) -  Packages an AWS SAM application. This command creates a .zip file of your code and dependencies, and uploads the file to Amazon Simple Storage Service (Amazon S3). 

   1. sam package [OPTIONS] [ARGS]...

3. Build – To build your AWS SAM application, run the sam build command. You can build inside a container.

   1. >>sam build

   2. Building codeuri: C:\my_app\hello_world runtime: python3.9 metadata: {} architecture: x86_64 functions: ['HelloWorldFunction’] ...

4. Test – To run your AWS SAM application locally for testing purposes, run the sam local invoke command.

   1. >>sam local invoke "HelloWorldFunction" -e event.json

   2. >>sam build --use-container

5. Deploy – To deploy through an interactive prompt, if you are using a configuration file, run the sam deploy command.

   1. >>sam deploy 

   2. >>sam deploy --template-file deploy.yml

   3. >>$ sam build && sam package --s3-bucket <bucket_name>

6. 
   

Handler

1. When a Lambda function is invoked, the handler code runs. 

2. The handler is a specific code method (Java, C#) or function (Node.js, Python) that you’ve created and included in your package. 

3. You specify the handler when creating a Lambda function. 

4. Arguments

   1. Event object

      1. When your Lambda function is invoked.

      2. The event differs in structure and contents, depending on which event source created it. 

      3. For example:

         1. API Gateway creates will contain details related to the HTTPS request that was made by the API client. Details include path, query string, request body. 

         2. Amazon S3 when a new object is created includes details about the bucket and the new object.

      4. Example:

         1. {  "Records": [ { "eventVersion": "2.1",

         2. "eventSource": "aws:s3",

         3. "awsRegion": "us-east-2",

         4. "eventTime": "2019-09-03T19:37:27.192Z",

         5. "eventName": "ObjectCreated:Put",

         6. "userIdentity": {  "principalId": "AWS:AIDAINPONIXQXHT3IKHL2"  },

         7. "requestParameters": {"sourceIPAddress": "205.255.255.255" },

         8. "responseElements": {"x-amz-request-id": "D82...", "x-amz-id-2": "vl...="},

         9. "s3": {  "s3SchemaVersion": "1.0", "configurationId": "828. . .", "bucket": { "name": “notes-bucket",

         10. "ownerIdentity": { "principalId": "A3I... }, "arn": "arn:aws:s3:::lambda-artifacts-dea..." }, "object": { "key": "b2...", "size": 1305107, "eTag": "b21…", "sequencer": "0C0F6F405D6ED209E1“ }

         11.       }}}]}

   2. Context object (optional)

      1. Allows your function code to interact with the Lambda runtime environment. 

      2. The contents and structure of the context object vary, based on the language runtime your Lambda function is using. 

      3. However, at a minimum, it will contain the following elements:

         1. AWS RequestId – Used to track specific invocations of a Lambda function (important for error reporting or when contacting AWS Support).

         2. Timeout – The amount of time in milliseconds that remain before your function timeout occurs. (Lambda functions can run a maximum of 900 seconds as of this publishing, but you can configure a shorter timeout.)

         3. Logging – Each language runtime provides the ability to stream log statements to CloudWatch Logs. The context object contains information about which CloudWatch Logs stream your log statements will be sent to. 

      4. Example:

         1. {

         2.   "callbackWaitsForEmptyEventLoop": true,

         3.   "functionVersion": "$LATEST",

         4.   "functionName": "printContext",

         5.   "memoryLimitInMB": "128",

         6.   "logGroupName": "/aws/lambda/printContext",

         7.   "logStreamName": "2019/12/02/[$LATEST]b3352ee98b4048eb8266cf5d4379693d",

         8.   "invokedFunctionArn": "arn:aws:lambda:ap-southeast-2:180814441145:function:printContext",

         9.   "awsRequestId": "501444a8-6f91-4d6d-a088-179e0df01a31"

         10. }

      5. Example (Methods Java)

         1. getRemainingTimeInMillis() 

         2. getFunctionName() 

         3. getFunctionVersion(). 

         4. getInvokedFunctionArn() 

         5. getMemoryLimitInMB()

         6. getAwsRequestId() 

         7. getLogGroupName() 

         8. getLogStreamName() 

         9. getIdentity() 

         10. getClientContext() 

         11. getLogger() 

      6. Context Java:

         1. exports.handler = async function(event, context) {

         2.    console.log('Remaining time: ', context.getRemainingTimeInMillis())

         3.    console.log('Function name: ', context.functionName)

         4.    return context.logStreamName

         5. }

5. Handler Switching

   1. Our handler functions looks like filename.function-name 

   2. For example, if we export a function named "handler" in a file named index.js, our handler name would be index.handler. This means we can ship different files within our function and export different methods.

   3. Another useful feature is that we can dynamically tell AWS which of these filename / export name permutations to choose from, on the fly, without the need to redeploy code. The command to dynamically reconfigure a Lambda handler is:

      1. $ aws lambda update-function-configuration \

      2. --function-name ${FUNCTION_NAME} \

      3. --handler ${FILE_NAME}.${METHOD_NAME}

   4. When your application is reconfigured in this manner, AWS will begin creating new instances of your application with the appropriate environment variables set. 

   5. AWS will also ensure all the currently running requests finish. 

   6. New requests will then begin going to the new instances. The old instances will then all be destroyed once they have no more requests to handle. None of the old instances will be reused, so any state within the processes will be lost.

   7. 
      

Function Handler Examples

1. Python (1)

   1. def handler_name(event, context):     

   2.      ...

   3. return some_value

   4. 
      

   5. def my_handler(event, context):      

   6.      message = 'Hello {} {}!'.format(event['first_name'],

   7.                                     event['last_name'])

   8.     return {

   9.         'message' : message

   10.         }

2. Python (2)

   1. # Initializations outside of handler to make a more unit-testable function

   2. dynamoDBResource = boto3.resource('dynamodb')

   3. pollyClient = boto3.client('polly') 

   4. s3Client = boto3.client('s3')

   5. 
      

   6. def lambda_handler(event, context):

   7.     # Extract the user parameters from the event and environment

   8.     userId = event["userId"]

   9.     noteId = event["noteId"]

   10.     voiceId = event['voiceId']

   11.     mp3Bucket = os.environ['MP3_BUCKET_NAME']

   12.     ddbTable = os.environ['Notes_Table']

   13.     # Get the note text from the database

   14.     text = getNote(dynamoDBResource, ddbTable, userId, noteId)

   15.     # Save an MP3 file locally with the output from polly

   16.     filePath = createMP3File(pollyClient, text, voiceId, noteId)

   17.     # Host the file on S3 that is accessed by a presigned url

   18.     signedURL = hostFileOnS3(s3Client, filePath, mp3Bucket, userId, noteId)

   19.     return signedURL

   20. 
       

3. Java

   1. In the example Java code, the first handler parameter is the input to the handler (myHandler), which can be any of the following:

      1. Event data (published by an event source, such as Amazon S3)

      2. Custom input you provide, such as an Integer object (as in this example) or any custom data object.

      3. MyOutput output handlerName(MyEvent event, Context) 

      4.  { ...  }

      5. 
         

      6. package example;

      7. 
         

      8. Import com.amazonaws.services.lambda.runtime.Context;

      9. Import com.amazonaws.services.lambda.runtime.RequestHandler;

      10. 
          

      11. Public class Hello implements RequestHandler<Integer, String>{

      12.     Public String myHandler(init myCount, Context context) {

      13.         return String.valueOf(myCount);

      14.     }

      15. }

      16. 
          
   2. 
      

Environment Variables

1. Use environment variables to pass operational parameters to your function.

Lambda API

1. CreateFunction

   1. Create a Lambda function through a deployment package

   2. Package type can be a ZIP or a container image

2. UpdateFunctionCode

   1. Update the code of the Lambda function

   2. Code must be unpublished

3. UpdateFunctionConfiguration

   1. Modify the version-specific settings of the Lambda function

4. Invoke

   1. Invokes a Lambda function synchronously or asynchronously

Creating a Lambda function Command

1. >>  aws lambda create-function --function-name dictate-function --handler app.lambda_handler --runtime python3.8 –role arn:aws:iam::563926481938:role/lambdaPollyRole --environment Variables={TABLE_NAME=$notesTable} --zip-file fileb://dictate-function.zip

2. {   "LastUpdateStatus": "Successful",

3.     "FunctionName": "dictateABC",

4.     "LastModified": "2021-08-18T20:46:28.341+0000",

5.     "RevisionId": "a76cada1-ee56-4b0b-924e-d2c38fbdf9ab",

6.     "MemorySize": 128,

7.     "Environment": {  "Variables": { "TABLE_NAME": "Notes"   } },

8.     "State": "Active",

9.     "Version": "$LATEST",

10.     "Role": "arn:aws:iam::563926481938:role/lambdaPollyRole",

11.     "Timeout": 3,

12.     "Handler": "app.lambda_handler",

13.     "Runtime": "python3.8",

14.     "TracingConfig": {  "Mode": "PassThrough“  },

15.     "CodeSha256": "Oow63xlj1nCVbbfY2rJGTjAAclBrufctZxzjfXt0MOU=",

16.     "Description": "",

17.     "CodeSize": 1273,

18.     "FunctionArn": "arn:aws:lambda:us-west-2:563926481938:function:dictateABC",

19.     "PackageType": "Zip"

20. } 

Update Config Command

1. >> aws lambda update-function-configuration --function-name dictate-function 

2.    --environment Variables="{MP3_BUCKET_NAME=$apiBucket, Notes_Table=$notesTable}"

3. {

4.     "FunctionName": "dictate-function",

5.      . . .

6.     "Environment": {

7.         "Variables": {

8.             "MP3_BUCKET_NAME": "labstack-ykantos-qbtxaus339lj-pollynotesapibucket",

9.             "Notes_Table": "pollynotes_table"

10.         }

11.     },

12.      . . .
    }

Versions

1. Lambda functions are versioned. 

2. When you publish a new version, the version number is incremented, and the earlier version is stored. 

3. You can publish one or more versions of your Lambda function. Each version has a unique ARN.

Aliases

1. You can use aliases to invoke specific versions of the function. 

2. A Lambda alias is like a pointer to a specific function version. Aliases can be updated to point to other versions.

3. Examples:

   1. arn:aws:lambda:aws-region:acct-id:function:dictate-function:Prod

   2. arn:aws:lambda:aws-region:acct-id:function:dictate-function:Test

Layers

1. A layer is a .zip file archive that contains code that is beyond the business logic, such as libraries, custom runtimes, and other dependencies.  

2. With layers, you can share such code across different versions of the same function or across multiple functions in a workload. 

3. The functions continue to have the same access to the dependencies but use less storage. 

4. Layers help keep your function deployment package small. That means that during a cold start, your deployment package can download faster.

5. A deployed layer is an immutable version. Each time a new layer is published, the version number increments. 

6. When you include a layer in a function, you specify the layer version that you want to use.

7. You can include up to five layers per function, which count towards the standard Lambda deployment size limits.

8. Limits:

   1. 250 MB 

   2. 5 layers