Building your own GoToCloud platform
Contents
IAM user with Administrator Access authority logs into an AWS account from the AWS console.
Change to the region you plan to use.
Navigate to the "CloudFormation" console page. Click ”Create stack"and select " With new resources (standard)".
Select the following items and click on "Next" in "Create stack".
Prepare templates -> Choose an existings template
Template source -> Upload a template file
Choose file -> kek-gtc-master.yml
Enter the following items and click on "Next" in "Specify stack details".
Stack name -> kek-gtc-master-[AWS account name] (e.g. kek-gtc-master-kek901)
The stack name is used as part of the shared S3 bucket name, so the stack name must be changed if an S3 bucket with the same name already exists.
Parameters remain default, no need to change
Enter the following items and click on "Next" in "Configure stack options"
Enter the tag value with reference to the following.
Tags
key value
gtc:iam-user AWS account name (e.g. kek901)
gtc:method admin
gtc:project admin
gtc:account AWS account ID (e.g. 123456789012)
User AWS account name (e.g. kek901)
Service admin
Team admin
No other optional items need to be changed.
After Confirm entry details in "Review and create", check the box in Capabilities and click on "submit" at the bottom.
Check status of kek-gtc-master is [CREATE_COMPLETE] in "Stacks" list.
Next, create a stack of VpcPeeringAcceptorRole
Change the value of Parameters -> PeeringAccount ->Default in vpc-peering-acceptor-role.yml to the Account ID where the masterEFS exists.
Select the following items and click on "Next" in "Create stack".
Prepare templates -> Choose an existings template
Template source -> Upload a template file
Choose file -> vpc-peering-acceptor-role.yml
Enter the following items and click on "Next" in "Specify stack details".
Stack name -> vpc-peering-acceptor-role
Parameters - PeeringAccount -> the Account ID where the masterEFS exists.
Supplementary information
CloudFormation performs the followings in this step.
Create VPC for master EFS
Create multiple subnets
Create route table
Create shared S3 bucket
Create master EFS
Create mount targets on each subnet
Create security group
Permit TCP communication using port 2049 from 10.0.0.0/8
Create master EFS location for AWS DataSync
Create IAM Roles for the followings.
For writing to EFS
For AWS DataSync
Create IAM Role for VPC peering.
IAM user with Administrator Access authority logs into an AWS account from the AWS console.
Change to the region you plan to use.
Navigate to the "CloudFormation" console page and display Stack list in "CloudFomation", check if stack named kek-gtc-shareN(N is an integer greater than or equal to 1) exist.
Click ”Create stack"and select " With new resources (standard)".
Select the following items and click on "Next" in "Create stack".
Prepare templates -> Choose an existing template
Template source -> Upload a template file
Choose file -> kek-gtc-share.yml
Enter the following items and click on "Next" in "Specify stack details".
Stack name -> kek-gtc-shareN
(N is an integer greater than or equal to 1. If a stack with the same name already exists, it is +1 to the largest one.)
AvailabilityZone1 -> ap-northeast-1a
Specify the availability zone name corresponding to the availability zone ID used by KEK GoToCloud.
Because the mapping between availability zone name and availability zone ID is different for each AWS account, check the "Your AZ ID" on AWS RAM console. For more information, see https://docs.aws.amazon.com/ram/latest/userguide/working-with-az-ids.html
DataSyncRole -> ARN of DataSyncEFSFullAccessRole (Usually default)
MasterEFSLocation -> Location ARN of Master EFS for Datasync location
This is the export value of kek-gtc-master-efs-loc in the exports list of CloudFormation(Usually default)
Subnet1CidrBlock ->10.N.0.0/20 (N is the same as used in the stack name)
VpcCidrBlock ->10.N.0.0/16 (N is the same as used in the stack name)
Enter the following items and click on "Next" in "Configure stack options"
Enter the tag value with reference to the following.
Tags
key value
gtc:iam-user AWS account name (e.g. kek901)
gtc:method admin
gtc:project admin
gtc:account AWS account ID (e.g. 123456789012)
User AWS account name (e.g. kek901)
Service admin
Team admin
No other optional items need to be changed.
After Confirm entry details in "Review and create", check the box in Capabilities and click on "submit" at the bottom.
Check status of kek-gtc-shareN is [CREATE_COMPLETE] in "Stacks" list.
Supplementary information
CloudFormation performs the followings in this step.
Create VPC for shared EFS
Create single subnet
Create route table
Create shared EFS
Create mount target for shared EFS on the single subnet.
Create security group
Permit TCP communication using port 2049 from 10.0.0.0/8
Create shared EFS location for AWS DataSync
Create IAM Roles for AWS DataSync
Create task for DataSync to synchronize with the master EFS.
To be performed on AWS account that owns shared EFS
IAM user with Administrator Access authority logs into an AWS account that owns shared EFS from the AWS console.
Navigate to the "CloudFormation" console page and select "VPCPeeringAcceptorRole" in stacks list (This stack exists only in the region where the master EFS resides)
Click on the top right "Update".
Select the following items and click on "Next" in "Update stack".
Prepare templates -> use existing template
Add the AWS account ID you want to connect VPC peering to shred VPC. Enter the following items and click on "Next" in "Specify stack details".
PeeringAccount -> Add the AWS account ID you want to connect VPC peering to shared VPC at the end.
"Configure stack options" are not changed and click on "Next".
After Confirm entry details in "Review VPCPeeringAcceptorRole", check the box in "Capabilities" and click on "submit" at the bottom.
Check status of VPCPeeringAcceptorRole is [CREATE_COMPLETE] in "Stacks" list.
Create kek_gtc_user_<AWS account ID>.yml. Change the following items and send kek_gtc_user_<AWS account ID>.yml file to user.
VPC CIDR
Subnet CIDR
Availability zone name
PeerOwnerAccountVpc
To be performed on user's AWS accounts
IAM user with Administrator Access authority logs into an user's AWS account for data analysis
Change to the region you plan to use.
Navigate to the "CloudFormation" console page. Click ”Create stack"and select " With new resources (standard)".
Select the following items and click on "Next" in "Create stack".
Prepare templates -> Choose an existing template
Template source -> Upload a template file
Choose file -> kek_gtc_user_<AWS account ID>.yml
Enter the following items and click on "Next" in "Specify stack details".
Stack name ー＞ kek-gtc-user
The following parameters do not need to be changed if they nave been set in the YML file in advance.
AvailabilityZone1
Specify the availability zone name corresponding to the availability zone ID used by KEK GoToCloud.
Because the mapping between availability zone name and availability zone ID is different for each AWS account, check the "Your AZ ID" on AWS RAM console. For more information, see https://docs.aws.amazon.com/ram/latest/userguide/working-with-az-ids.html
IsFirstStack
Select "true" if this is the first time kek-gtc-user stack is created in the this AWS account, including other regions.
Select "false" after the second time, including other regions.
Password
Login password for IAM user: kek-gtc-user01 created by CloudFormation
PeerOwnerAccountId
AWS account ID that owns shared EFS
PeerOwnerAccountVpc
ID of VPC where the shared EFS is located
PeerOwnerAccountVpcCidrBlock
CIDR of VPC where the shared EFS is located
Subnet1CidrBlock
CIDR of subnet created by CloudFormation
VpcCidrBlock
CIDR of VPC created by CloudFormation
"Configure stack options" are not changed and click on "Next".
After Confirm entry details in "Review VPCPeeringAcceptorRole", check the box in "Capabilities" and click on "submit" at the bottom.
Check status of kek-gtc-user is [CREATE_COMPLETE] in "Stacks" list.
To be performed on AWS account that owns shared EFS
IAM user with Administrator Access authority logs into an AWS account that owns shared EFS from the AWS console.
Change to the region you plan to use.
Navigate to the "VPC" console page. Select "Peering connections" from the menu on the left side.
Check the peering connection created by CloudFormation. If the Name item is left blank in Peering connection list, fill in its requester CIDR.
Select "Route tables" from the menu on the left side and select route table of VPC where the shared EFS is located (Usually "kek-gtc-share1-rtb")
Select "Routes" tab and click "Edit routes". Click "Add route" and enter the following items
Destination -> CIDR of VPC
Target -> Select Peering Connection and Peering Connection ID
Supplementary information
CloudFormation performs the followings for user's account in this step.
Create VPC for data analysis
Create single subnet
Create route table
Create Internet gateway
Create IAM group "pcluster-user"
Create IAM user "kek-gtc-user01"
Create peering connection between VPC for data analysis and VPC where the shared EFS is located.
You should install GoToCloud scripts to master EFS.
IAM user with Administrator Access authority logs into an AWS account that owns master EFS from the AWS console.
Navigate to Cloud9 console page and create environment like "Creating AWS Cloud9" in "GoToCloud Platform for cryo-EM SPA > Getting Start".
Navigate to EC2 console page and select "Instances" from left menu.
Select Cloud9 instance created step No.2 and click "Actions" - "Security" - "Modify IAM role"
Choose "EFSWriteAccessRole" in dropdown list and click "Update IAM role".
Navigate to Cloud9 console page and open Cloud9 IDE created step No.2.
Open the new terminal on Cloud9 and run following commands for mounting masterEFS.
$ sudo yum -y install amazon-efs-utils
$ sudo mkdir /efs
$ sudo mount -t efs -o tls,iam,mounttargetip=[masterEFS mount target IP adress] [masterEFS file system ID] /efs
Install GoToCloud scripts to /efs/em directory.
$ cd /efs
$ mkdir em
$ cd em
$ git clone https://github.com/KEK-SBRC-CryoEM/gotocloud.git
$ cp -r gotocloud/gtc_sh/ ./
Download the followings scripts from https://github.com/KEK-SBRC-CryoEM/gotocloud and upload to shared S3 bucket created in step1.
gtc_efs_setting.json
gtc_setup_gotocloud_environment.sh
post_install.sh
Modify /efs/em/gtc_sh/gtc_efs_setting.json file to match your environment.
Add informations for masterEFS and sharedEFSs. The information needed is available from AWS console.
(1) Navigate to EFS console page and select "kek-gtc-master-efs" or kek-gtc-shareN-efs"
The following information can be obtained from the Network tab.
Information for the master EFS includes only the information of AvailabilityZone specified in step 3.
FileSystemName
MountTargetId
AvailabilityZoneId
AvailabilityZoneName
FileSystemId
SubnetId
IpAddress
(2) "VpcId" is of "kek-gtc-master-[AWS account name]-vpc" or "kek-gtc-shredN-vpc".
Navigate to VPC console page and VPC IDs are obtained from "Your VPCs" menu in the left.
(3) "OwnerId" is the AWS account ID where masterEFS and sharedEFSs exist.
Change access setting for shared S3 bucket.
(1) Navigate to S3 console page and select "kek-gtc-master-[AWS account name]-s3-bucket".
(2) Select "Permissions" tab and click "Edit" in "Block public access (bucket settings)".
(3) Uncheck "Block all public access" and click "Save changes".
(4) Scroll down to the bottom in "Permissions" tab.
(5) Click "Edit" in "Object Ownership".
(6) Select "ACLs enabled" and "Object writer".
(7) Check the box before "I acknowledge that ACLs will be restored" and click "Save changes".
(8) Select "Objects" tab.
(9) Select all the following files and click "Actions" - "Make public using ACL".
gtc_efs_setting.json (modified in step 4.2.2)
gtc_setup_gotocloud_environment.sh
post_install.sh
(10) Click "Make public".
Click here to perform the installation
This procedure REMOVEs all GoToCloud environments.
To be performed on user's AWS accounts
IAM user with Administrator Access authority logs into an user's AWS account for data analysis.
Change to the region you use.
Navigate to the "CloudFormation" console page and select ”kek-gtc-user" from stacks list.
Click on "Delete" button at the top.
To be performed on AWS account that owns shared EFS
IAM user with Administrator Access authority logs into an AWS account that owns shared EFS from the AWS console.
Change to the region you use.
Navigate to the "CloudFormation" console page.
Select the following stacks in order form stacks list and click "Delete" button at the top.
kek-gtc-user
kek-gtc-shareN (N is an integer)
kek-gtc-master
VPCPeeringAcceptorRole