AWS S3

3.1. When a file is created, execute a lambda

3.2. Upload an Object Using the AWS SDK for Java

3.2. Browser based Amazon S3 upload using HTTP POST

1. Introduction

Amazon Simple Storage Service is storage for the Internet. Amazon S3 has a simple web services interface that you can use to store and retrieve any amount of data, at any time, from anywhere on the web.

2. Reference

Amazon Simple Storage Service Documentation

https://docs.aws.amazon.com/s3/index.html

Tutorial: Using AWS Lambda with Amazon S3

https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html

Tool for fast deletion and emptying of S3 buckets (versioning supported)

https://dev.to/aws-builders/tool-for-fast-deletion-and-emptying-of-s3-buckets-versioning-supported-6dn

3. How-to

3.1. When a file is created, execute a lambda

(a) In the S3 bucket, tab Properties, box Events

Add notification

Name: XxxForCreate

Events: All object create events

Prefix: incoming/

Suffix:

Send to: Lambda Function

Lambda: xxx-consolidation-dev

(b) For testing it:

Add a file to the "incoming/" folder
View logs in the CloudWatch console for verifying that the lambda has executed

3.2. Upload an Object Using the AWS SDK for Java

https://docs.aws.amazon.com/AmazonS3/latest/dev/UploadObjSingleOpJava.html

3.2. Browser based Amazon S3 upload using HTTP POST

Amazon S3 supports HTTP POST requests so that users can upload content directly to Amazon S3, based on the reference:

Authenticating Requests: Browser-Based Uploads Using POST (AWS Signature Version 4)

https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-authentication-HTTPPOST.html

Example: Browser-Based Upload using HTTP POST (Using AWS Signature Version 4)

https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-post-example.html

Create a signed AWS API request

https://docs.aws.amazon.com/IAM/latest/UserGuide/create-signed-request.html

S3 API POST Object

https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html

To authenticate an HTTP POST request you do the following:

The form must include the following fields to provide signature and relevant information that Amazon S3 can use to re-calculate the signature upon receiving the request:

policy: The Base64-encoded security policy that describes what is permitted in the request. For signature calculation this policy is the string you sign. Amazon S3 must get this policy so it can re-calculate the signature.
x-amz-algorithm: The signing algorithm used. For AWS Signature Version 4, the value is AWS4-HMAC-SHA256.
x-amz-credential: In addition to your access key ID, this provides scope information you used in calculating the signing key for signature calculation. It is a string of the following form: <your-access-key-id>/<date>/<aws-region>/<aws-service>/aws4_request For example: AKIAIOSFODNN7EXAMPLE/20130728/us-east-1/s3/aws4_request For Amazon S3, the aws-service string is s3. For a list of Amazon S3 aws-region strings, see Regions and Endpoints in the AWS General Reference.
x-amz-date: It is the date value in ISO8601 format. For example, 20130728T000000Z. It is the same date you used in creating the signing key. This must also be the same value you provide in the policy (x-amz-date) that you signed.
x-amz-signature: (AWS Signature Version 4) The HMAC-SHA256 hash of the security policy. For more information on options for the signature, see reference "Create a signed AWS API request".

Hex() function [lowercase base 16 encoding] Java implementation available at class com.amazonaws.util.Base16Lower, method String encodeAsString(byte... bytes)

The POST policy must include the following elements:

x-amz-algorithm: The signing algorithm that you used to calculation the signature. For AWS Signature Version 4, the value is AWS4-HMAC-SHA256.
x-amz-credential: In addition to your access key ID, this provides scope information you used in calculating the signing key for signature calculation. It is a string of the following form: <your-access-key-id>/<date>/<aws-region>/<aws-service>/aws4_request For example, AKIAIOSFODNN7EXAMPLE/20130728/us-east-1/s3/aws4_request
x-amz-date: The date value specified in the ISO8601 formatted string. For example, "20130728T000000Z". The date must be the same that you used in creating the signing key for signature calculation.

For signature calculation the POST policy is the string to sign.

DTO example with the information needed by the frontend:

Reference: fwdefense

package edu.cou.myapp.dto.s3;

import java.io.Serializable;

import lombok.AllArgsConstructor;

import lombok.Data;

import lombok.EqualsAndHashCode;

import lombok.NoArgsConstructor;

/**

* Info needed by the user browser for uploading a file directly to S3 bucket.

*

* <pre>

* Reference:

*

* Authenticating Requests: Browser-Based Uploads Using POST (AWS Signature Version 4)

* https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-authentication-HTTPPOST.html

* </pre>

*/

@EqualsAndHashCode(callSuper = false)

@NoArgsConstructor

@AllArgsConstructor

@Data // NOSONAR

public class S3BrowserUploadDto implements Serializable {

private static final long serialVersionUID = 8070132977159145872L;

/** The S3 bucket name for uploading the file to. */

private String buckedName;

/** The S3 bucket object key to be used for uploading the file */

private String objectKey;

/**

* [Form required] The Base64-encoded security policy that describes what is permitted in the

* request.

*/

private String policy;

/** [Form required] The signing algorithm used. */

private String amzAlgorithm;

/**

* [Form required] In addition to you access key ID, this provides scope information you used in

* calculating the signing key for signature calculation.

*/

private String amzCredential;

/** [Form required] The date value in ISO8601 format, eg: 20230908T000000Z. */

private String amzDate;

/** [Form required] (AWS Signature Version 4) The HMAC-SHA256 hash of the security policy. */

private String amzSignature;

}

Methods used for filling the previous DTO for the frontend

public S3BrowserUploadDto getS3BrowserBasedUploadInfo(String objectKey)

throws InvalidKeyException, NoSuchAlgorithmException {

// DOC: 1=<your-access-key-id>, 2=<date>, 3=<aws-region>, 4=<aws-service>

val xAmzCredentialFormat = "%s/%s/%s/%s/aws4_request";

// DOC. Eg: "20230908"

var dateYYYYMMDD = LocalDate.now().format(DateTimeFormatter.ofPattern("yyyyMMdd"));

// DOC. Eg: "20230908T000000Z"

var dateISO8601 = dateYYYYMMDD + "T000000Z";

var s3tf = this.appProperties.getS3tf();

String bucketName = s3tf.getBucketName();

// DOC. Sample: AKIAIOSFODNN7EXAMPLE/20130728/us-east-1/s3/aws4_request

String xAmzCredential = String.format(xAmzCredentialFormat, s3tf.getAk(), dateYYYYMMDD,

S3Helper.REGION_ID_EU_IRELAND, S3Helper.SERVICE_NAME);

var xAmzAlgorithm = "AWS4-HMAC-SHA256";

String policyBase64 = getPolicyBase64Encoded(bucketName, objectKey, xAmzCredential,

xAmzAlgorithm, dateISO8601);

String xAmzDate = dateISO8601;

/** [Form required] (AWS Signature Version 4) The HMAC-SHA256 hash of the security policy. */

var signingKey = this.awsSigningKey(s3tf.getSak(), dateYYYYMMDD, S3Helper.REGION_ID_EU_IRELAND,

S3Helper.SERVICE_NAME);

String xAmzSignature = this.awsSignatureVersion4HexEncoded(policyBase64, signingKey);

return new S3BrowserUploadDto(bucketName, objectKey, policyBase64, xAmzAlgorithm,

xAmzCredential, xAmzDate, xAmzSignature);

}

Private method getPolicyBase64Encoded:

/* A base 64 string can be decoded w/:

* byte[] decodedBytes = Base64.getDecoder().decode(encodedString)

* String decodedString = new String(decodedBytes)

*/

private String getPolicyBase64Encoded(String bucketName, String objectKey, String xAmzCredential,

String xAmzAlgorithm, String yyyyMMdd) {

//

var expirationInMinutes = 16;

var policyTemplate = """

{

"expiration": "%s",

"conditions": [

{"bucket": "%s"},

{"key": "%s"},

{"x-amz-credential": "%s"},

{"x-amz-algorithm": "%s"},

{"x-amz-date": "%s"}

]

}

""";

var policy = String.format(policyTemplate, nowInISO8601(expirationInMinutes), bucketName,

objectKey, xAmzCredential, xAmzAlgorithm, yyyyMMdd);

return Base64.getEncoder().encodeToString(policy.getBytes());

}

Private method nowInISO8601:

private String nowInISO8601(int extraMinutes) {

// Get the current date and time in UTC

var now = Instant.now();

// Add 'extraMinutes' minutes to the current time

Instant future = now.plus(extraMinutes, ChronoUnit.MINUTES);

// Convert the instant to a ZonedDateTime object in UTC

var zonedDateTime = ZonedDateTime.ofInstant(future, ZoneId.of("UTC"));

// Create a DateTimeFormatter object with the desired format

var formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss'Z'");

// Format the date and time and return the result

return zonedDateTime.format(formatter);

}

Private method awsSigningKey:

/**

* Obtain the AWS signing key.

*

* Reference >

* https://docs.aws.amazon.com/es_es/general/latest/gr/signature-v4-examples.html#signature-v4-examples-java

*

* @param yyyymmdd eg: "20181009"

* @param regionName eg: "eu-west-1"

* @param serviceName eg: "s3"

* @return Signing key

* @throws NoSuchAlgorithmException -

* @throws InvalidKeyException -

* @throws BadRequestAppException if yyyymmdd is not in format yyyyMMdd

*/

private byte[] awsSigningKey(final String secretAccessKey, final String yyyymmdd,

final String awsRegion, final String awsService) throws InvalidKeyException,

NoSuchAlgorithmException {

// Safety check

Date yyyymmddDate;

try {

yyyymmddDate = sdf.parse(yyyymmdd);

} catch (ParseException e) {

throw new BadRequestAppException(

"The AWS signing key date must be a date in format 'yyyyMMdd' but was '" + yyyymmdd

+ "'");

}

if (!sdf.format(yyyymmddDate).equals(yyyymmdd)) {

throw new BadRequestAppException("The AWS signing key date has format 'yyyyMMdd' but was '"

+ yyyymmdd + "'");

}

// [https://docs.aws.amazon.com/images/AmazonS3/latest/API/images/signing-overview.png]

byte[] dateKey = hmacSha256(yyyymmdd, ("AWS4" + secretAccessKey).getBytes(

StandardCharsets.UTF_8));

byte[] dateRegionKey = hmacSha256(awsRegion, dateKey);

byte[] dateRegionServiceKey = hmacSha256(awsService, dateRegionKey);

// DOC. Return byte[] signing key

return hmacSha256("aws4_request", dateRegionServiceKey);

}

Private method hmacSha256:

/**

* Function hmacSHA256.

*

* @param data .

* @param key .

* @return .

* @throws InvalidKeyException -

* @throws NoSuchAlgorithmException -

*/

private static byte[] hmacSha256(final String data, final byte[] key) throws InvalidKeyException,

NoSuchAlgorithmException {

var algorithm = HmacAlgorithms.HMAC_SHA_256.toString();

Mac mac;

mac = Mac.getInstance(algorithm);

mac.init(new SecretKeySpec(key, algorithm));

return mac.doFinal(data.getBytes(StandardCharsets.UTF_8));

}

Private method getPolicyBase64Encoded:

/* It calculates an AWS signature version 4, and returns it using hex encoding.

*/

private String awsSignatureVersion4HexEncoded(String policyBase64, byte[] signingKey)

throws InvalidKeyException, NoSuchAlgorithmException {

byte[] signatureKey = hmacSha256(policyBase64, signingKey);

// DOC. Return the signature (hex encoded)

return com.amazonaws.util.Base16Lower.encodeAsString(signatureKey);

}

Sample "index.html" form for uploading a file to the S3 bucket

Note: It needs to be loaded via http:// protocol (doesn't work w/ file://), eg: python3 -m http.server

<!doctype html>

<head>

</head

<body>

Object key:

X-Amz-Credential:

X-Amz-Algorithm:

X-Amz-Date:

<br />

File:

</form>

</body>

</html>

3.3. Pre-signed URL Amazon S3 download

/**

* Obtain the presigned url.

*

* @param bucketName the name of s3 bucket name.

* @param objectKey the object key in 'bucketName'

* @param urlValidityInMs Validaity, in ms, of the presigned url to be created

* @return An URL that expires in <b>urlValidityInMs</b>

* @throws AppException -

*/

public String getS3PresignedUrl(final @NotBlank String bucketName,

final @NotBlank String objectKey, final long urlValidityInMs) {

/* DOC. Requests that are pre-signed by SigV4 algorithm are valid for at most 7 days */

URL url;

String clientRegion = REGION_ID_EU_IRELAND;

BasicAWSCredentials awsCreds = null;

try {

// Pasamos las credenciales AWS S3

awsCreds = new BasicAWSCredentials(this.appProperties.getS3tf().getAk(), this.appProperties

.getS3tf().getSak());

final AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withRegion(clientRegion)

.withCredentials(new AWSStaticCredentialsProvider(awsCreds)).build();

// Set the presigned URL to expire after URL_VALIDITY_MS

var expiration = new java.util.Date();

long expTimeMillis = expiration.getTime();

expTimeMillis += urlValidityInMs;

expiration.setTime(expTimeMillis);

// Generate the presigned URL.

log.info("Generating pre-signed URL.");

var generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName, objectKey,

HttpMethod.GET).withExpiration(expiration);

url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);

// (bucketName, key, expiration)

log.info("Pre-Signed URL: " + url.toString());

} catch (AmazonServiceException e) {

throw new ServiceUnavailableAppException(e.getMessage(), e);

// The call was transmitted successfully, but Amazon S3 couldn't process

// it, so it returned an error response.

} catch (SdkClientException e) {

// Amazon S3 couldn't be contacted for a response, or the client

// couldn't parse the response from Amazon S3.

throw new BadRequestAppException(e.getMessage(), e);

}

return url.toString();

}