Serverless architecture
★S3にIP制限をかける
ポリシー作成ツール http://awspolicygen.s3.amazonaws.com/policygen.html
例：
{
"Id": "Policy15...",
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt15...",
"Action": "s3:*",
"Effect": "Deny",
"Principal": "*",
"Resource": "arn:aws:s3:::<bucket_name>/*",
"Condition": {
"NotIpAddress": {
"aws:SourceIp": [
"...",
"..."
]
}
}
}
]
}
★API Gateway本文マッピングテンプレート
{
"sourceIp" : "$context.identity.sourceIp", リクエストIP
"method" : "$context.httpMethod,
"body" : "$input.json('$')", POST/PUTの内容
"name" : "$input.params('name')" パス・クエリ・ヘッダ
}
※Lambdaにevent.*で取得できる
★S3 Select
※圧縮データを使用すべき
s3 = boto3.client('s3')
resp = s3.select_object_content(
Bucket='s3select-demo',
# Key='sample_data.csv',
Key='sample_data.csv.gz',
ExpressionType='SQL',
Expression="SELECT * FROM s3object s where s.\"Name\" = 'Andy'",
# InputSerialization = {'CSV': {"FileHeaderInfo": "Use"}, 'CompressionType': 'NONE'},
InputSerialization = {'CSV': {"FileHeaderInfo": "Use"}, 'CompressionType': 'GZIP'},
OutputSerialization = {'CSV': {}},
)
for event in resp['Payload']:
if 'Records' in event:
records = event['Records']['Payload'].decode('utf-8')
print(records)
elif 'Stats' in event:
statsDetails = event['Stats']['Details']
print(statsDetails['BytesScanned'])
print(statsDetails['BytesProcessed'])
print(statsDetails['BytesReturned'])
★AWS Cognito UserPool
ユーザーの登録とConfirmation
事前準備
1.アクセスキー取得（AWSマネジメントコンソール上）：IAM -> ユーザー -> 認証情報
2.aws configureコマンドでAccess Key、Secret Access Key、Regionを登録
aws cognito-idp sign-up --client-id <作成したClientId> --username <ユーザーID> --password <パスワード> --user-attributes Name=email,Value=<Emailアドレス>
aws cognito-idp admin-confirm-sign-up --user-pool-id <作成したPoolId> --username <ユーザーID>
★Athenaにファイル名とパスを表示したい
SELECT ..., "$path" FROM "<database>"."<table>" ...
MSCK REPAIR TABLE {table} -- Load partition
CTAS（CREATE TABLE AS）
★DynamoDB 条件付き更新
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table("table1")
try:
update_status_with_condition(task_id, "FAILED", "RUNNING")
...
update_status_with_condition(task_id, "RUNNING", "DONE")
except Exception as error:
if error.response['Error']['Code'] == "ConditionalCheckFailedException":
print("Status is not 'FAILED'")
else:
print("Unknown exception")
sys.exit(1)
def update_status_with_condition(task_id, status_from, status_to):
key = {"Id": task_id}
update = 'SET Status = :status_to'
condition = "Status = :condition"
attr = {":status_to": status_to, ":condition": status_from}
table.update_item(
Key=key,
UpdateExpression=update,
ConditionExpression=condition,
ExpressionAttributeValues=attr)
★DQLでDynamoDBをSQL操作
https://dev.classmethod.jp/cloud/sql-against-dynamodb-with-dql/
pip install -U dql
dql --version
dql -r ap-northeast-1
SCAN * FROM sample-table;
SCAN count(*) FROM sample-table;
DELETE FROM sample-table;
SELECT count(*) FROM sample-table;
LOAD test.csv INTO sample-table;
AWS Glue components
