AWS S3

  • Logging (server across logging) into the same bucket causes exponential log growth.

  • S3 prefixes

    • Allows organizing objects, Easily search objects

    • Useful with IAM and bucket policies to restrict/grant access at object level

  • Access S3 privately (rather than across the Internet) - Configure a hardware VPN to a VPC and configure an S3 endpoint

  • S3 Select

    • simple structured query language (SQL) statements to filter the contents of Amazon S3 objects and retrieve just the subset of data that you need

    • lower the latency of retrieving data from S3 and reduce costs

  • Encryptions:

    • Amazon S3-Managed Keys (SSE-S3) : the data is encrypted by Amazon S3 using keys that are managed through S3 (AES-256)

      • Request header: x-amz-server-side-encryption(AES256)

    • Server-Side Encryption with Customer-Provided Keys (SSE-C) :

      • You manage the encryption keys and Amazon S3 manages the encryption

      • use HTTPS is mandatory

    • Server-Side Encryption with Customer Master Keys (CMKs)

      • Stored in AWS Key Management Service (SSE-KMS) :

      • auditing and permissions , control

    • Server side encryption with AWS KMS (SSE-KMS)

      • x-amz-server-side-encryption(aws:kms) and

      • x-amz-server-side-encryption-aws-kms-key-id(ARN for key in KMS)

      • throttles S3 performance because AWS KMS API calls has quota limit

  • S3 Static web hosting

    • Enable static website option

    • Disable “Block Public Access”

    • Configure Bucket policy to enable public read access

    • CloudFront Distributions to reduce latency of websites to users

  • S3 Replication

    • To meet regulations, reduces latency

    • Versioning should be enabled on both source and destination bucket

    • Replication can be in same or different region

    • Only new objects are replicated, (explicitly copy the old objects)

    • Ex: To replicate data between dev and test environments

    • Cross-region replication (CRR)

      • The source and destination buckets must have versioning enabled.

      • The source and destination buckets must be in different AWS Regions.

      • Amazon S3 must have permissions to replicate objects from that source bucket to the destination bucket on your behalf.

  • Pre-signed URL

    • Prevent unauthorized access of your S3 objects (photos , videos) hosted on S3 static website

    • Time limited permission (few hours to 7days)

    • Created using AWS SDK API

  • Lifecycle Policies

    • Move files automatically between storage classes and save cost

    • Transition: move files from one class to another

    • Expiration: Delete objects

S3 Glacier

  • S3 Glacier data retrieval options

    • Expedited: 1–5 minutes , higher-cost

    • Standard retrievals: 3–5 hours. This is default option.

    • Bulk retrievals: 5–12 hours, lowest-cost, for large or petabytes of data retrieval