Some difficulties using AWS within Cloud teaching

Final update. Possibly. After enduring a variety of time-consuming and sometimes quite painful experiences (many headdesk moments), and following the removal of some quite peculiar constraints a time after having invested time into workarounds [*sigh*], AWS Academy is now usable in the same way that AWS Educate once was before this saga began. But if you want to look properly at account security credentials and other IAM matters, and billing, on your own credit card be it.

As to Microsoft and Google ... for the former, one lesson in 2022 seems off-putting: https://new.pythonforengineers.com/blog/my-poor-experience-with-azure-or-why-im-sticking-with-aws/; for the latter, some unconstrained library updates in GAE along with an unexpected and seemingly uncontrollable file caching issue does make one wonder about the rest of the services.

Latest update - 3/11/21: "Educator tools have moved to AWS Academy. You can continue to use existing resources through December 31, 2021". A message seen only by logging in to AWS Academy. No other notification at all - email must be hard to use! Is my institution in AWS Academy? It is not. So to get there will require finding appropriate parties to file things .... but AWS Academy pages seem geared to AWS accredited things.

The combination of this, with actions below, seems to provide for a consistent message - AWS don't want to support this kind of thing. Message received - let us see what Google and/or Microsoft can offer instead.

--

[Certain details have been omitted, as this rant story is long enough already. This page might be updated depending on how/if the saga continues or if corrections become necessary]

As an academic who has been making use of AWS as one commercial cloud provider (of just two at this point) for over a decade, this has been made progressively more difficult in recent years due to successive changes in AWS offerings for education that are increasingly being communicated at short notice, and not necessarily to those who need to know, and also then a lack of clarity over what is being done has become problematic.

A certain amount of that change might have been unnecessary, for all parties involved, if AWS had ways to cap spend.

Having seen others call for such a thing, for many years, it does start to look as though AWS is keen on accidental overspend – “pay as you … oh”.

Receiving an alert that you have gone over a budget a day later is probably not that helpful – it’s likely to cause quite some anxiety, particularly as a recipient (read: student here, but basically any new user) slowly realised how much time, and possibly money, has gone in the interim. Responsible academics work hard to embed a “switch off” mentality but a small, yet still significant, number of students will have accidents. And those create panic. And that panic is fielded first by academics who, having spent time calming said student, then has to point them to AWS and tell them to cross their fingers and be patient.

For a number of years, such AWS use was relatively straightforward – fill in a form, and fairly quickly receive a list of codes for AWS credits. One mail merge later, and each student would be able to get going. Service limits already prevented extensive use of various instances. Relatively few accidents occurred.

If, at that point, there were an option to cap spend to the value of those credits – better still, to have incremental caps so that the blast radius of accidents was constrained - panic would have diminished quite a way. Yes, some could still do daft things, but if at such a limit they could do no more, and had to release the next tranche within their credits, a lesson might have been learned. If they kept having accidents that wiped them out, perhaps some additional credits available to the academic that could be provided along with very stern warnings that there would be no more, would address: next time, your problem.

Could have happened ... What did?

AWS ditched those credits.

And from there on things started to get more difficult.

Students could sign up to AWS Educate, and still get something, but it seems that many did, and so that then became limited to those for whom educators had registered - this change was announced on 14 December 2020 and would be applied from 4 January 2021. Coincidentally, that's about the same timespan as was then required to get an LMS link (after two attempts) – and cue a whole new problem with LMS links at the start of 2021 as a proportion went and registered for AWS Educate by … searching for it using Google; some then got into some complex and time-consuming loops of voiding accounts, one of which required many iterations.

Alternatives to obtaining an account through this route – a so-called Starter Account – are few; “Classrooms” and a so-called central credit. The former may entail managing a bunch of other things, though that is yet to be fully understood. The latter seems to be something an educator can have as well as a period of time working through IAM, but if their students go above the credit amounts given the implication seems to be that the educator could suffer a denial of finance attack on their credit card: setting hard work for students might cost you, and could affect your credit rating?

Of course, spending caps with increments might also offer a solution for that.

The Starter Account then, at least at the start of 2021….

Good points: spend limited to the credit amount, and no credit card involved. Looks promising.

Bad points: sub account, so you don’t get to see what services are actually costing – no access to billing, so billing alerts a bit irrelevant – and Free Tier resources dissipate the credits. Can’t create sub accounts through IAM, so have to use a proper account to address policies. Oh, and credentials last 3 hours so if you go long you’ll suddenly find nothing works until you go back to the site and click through several buttons to be able to copy and paste new credentials.

Several pages of other restrictions are also available to catch the unwary [1] – who are limited to us-east-1 [waves to China, and in particular the students who returned there during the pandemic and know - at least - how latency feels].

So, you can use AWS for (some) resources, but diving into Security or Billing, as some might think are quite important things to be aware of in cloud, means having your own account using your own credit card … and then remembering not to use this account for anything billable. So, lots more messages now to check which account is being used for what – on top of those about switching off.

Cue the latest change, with no prior information or fanfare.

At the end of April 2021, an alert was seen inside Starter Accounts – not, it would be important to note, in the interface for AWS Educate nor in any email to educators as previous changes have been, that two things were happening.

First, that when those credentials time out instances will also be stopped – not only will you have to go find new credentials, you’ll have to restart anything that you may have been working on. Don’t go expecting to leave something running for a few hours that you can come back to later. Get clicking!

Second, that available instance types would change.

Both these things would happen on 11^th May, so from the message being seen there was less than 2 weeks to adapt. Oh, and with some 150 students with a deadline on 18^th May – during a pandemic – this would do little to alleviate stresses on them. Or me.

But wait, which instance types?

Hunting around the web and inside AWS Educate revealed nothing (except that not all documents are up to date - more on that shortly), so off we go into the support system [2], where severity of all things Educate is set Low by default.

And the first response was that the following is the comprehensive set of instance types that will be available –

EC2: t2.micro, t2.medium and t2.small;

RDS: db.t1.micro, db.t2.small, db.m1.small, db.t2.medium.

And that's all.

EMR can use t2s? Er, no – that’s another service not supported there which is going to be tough on those that had been using it (say goodbye to “Big Data” or in fact anything reasonably computationally demanding).

Could they wait a couple of weeks, given that there’s been no real notice, and no documentation?

Again no.

Indeed, what I should do is set up an entirely new way for the students to be working – either using Classrooms or bearing financial and credit rating risk. Not much time available though, remember.

Here is part of the reasoning for the instance types change:

“The large instance types will no longer be supported in the Educate Starter Account (only) due to the high cost associated with those instances. These instances cause that students run out of their credits before they are up for renewals annually. This change is to ensure that student accounts remain active and are not deactivated due to over-spending.”

[Spending caps … increments …..]

Reading that another way though: we give students credits, and some use them up. So we'll inhibit them all.

So, avoiding credit risk, that leaves Classrooms – for which there are 3 options.

Neither of the first two options explicitly mentions EMR - though both mention EC2. Notable here is the announcement (15^th August 2020) that a number of other classrooms had been “retired”: Big data, Machine learning and artificial intelligence, Serverless computing [and a couple of others]. So might it be a mistake to automatically assume that because EMR uses EC2 instances it is included?

Those first two templates differ by just ELB. Neither mentions Lambda, although the description of the first (not in “Services enabled”) does mention IAM – the second doesn’t. Not necessarily clear. But it does get worse if you click “Learn More” – the resulting page still shows the retired Classrooms and is quite specific that EMR exists only under “Big Data” and Lambda only under “Serverless”.

AWS Cloud Basics template. Services enabled: EC2, S3, RDS

Big Data template. Services enabled: EC2, Athena, DynamoDB, EMR, Glue, RDS, S3

Building Scalable Websites template. Services enabled:EC2, S3, RDS, ELB

Cloud9 template. Services enabled: Cloud9, S3

Machine Learning and AI template. Services enabled: Machine Learning, Rekognition, Lex, Polly, Comprehend, Translate, Transcribe, SageMaker

Serverless Computing template. Services enabled: Lambda, API Gateway, S3

Ah, but the third way!

Indeed – option 3 is called “Starter Account Template” and “enables creating a classroom with all AWS Educate Starter Account services enabled”. This template, uniquely, links to a PDF stating the set of services, which is exactly the same document linked to by … erm…. Starter Accounts that are having instance types restricted massively downwards.

Once again, into the support system (Severity: Low?) to query the recommendation to use Classrooms. Actually, by now, interactions in the support system were already at a moderate number.

By now, it's already 7^th May, but some hints into what might be going on:

“I understand that the PDFs for services covered under Classrooms and Starter account are currently reflecting the same services. This is because we only have the one PDF listing the services available on both Classrooms and Individual Starter accounts. We are working on updating the PDF to reflect the upcoming changes, and a new PDF will be available when the new service imitations launch. Our service team has confirmed that these changes will not be rolled out to Classroom environments, so Classrooms accounts will be able to use different services from AWS Starter accounts that we issue to individual students.”

So, 4 days from the change but no documentation yet – I mean, apart from “Learn More” which in May 2021 hasn’t caught up with August 2020. And if we go back to a previous response, that “(only)” was trying to do a heck of a lot of work.

Potentially, though, some promise …. so, let us track matters a bit more actively....

By 12^th May 2021

Currently hoping that the change has failed, or will not be pushed without documentation. Or, alternatively, that EMR was not chosen by the vast majority and so things aren't about to blow up really quite horribly. Classrooms PDF shows it was last updated in October 2020 and is still the same as what you get via https://aws.amazon.com/education/awseducate/aws-educate-faqs/ so who knows.

And an application for Classrooms is still Under Review after 5 days.

13th May 2021

A student makes an unfortunate new discovery - less than a week to deadline - that accidentally asking for 20 EC2 instances blocks all access to the account. Nothing as simple as a refusal with a warning for these folk. Student forced towards a proper account while waiting to find out if all setup efforts to date have also been lost. See updated below.

14th May 2021

1. Informed that the document had been quietly updated: View the services supported with an AWS Educate Starter account. - metadata qualifies its creation as 12/05/2021, 21:34:44. Classrooms "Learn More" still not made it beyond August 2020.

2. Classrooms application still Under Review - 7 days.

3. Response to student's account having been locked:
   
   "Please be informed that Educate Account is provided by our third party vendor Vocareum which comes with a definite pre-loaded credits and if students tend to launch more than 20 instances, the account is locked as it may exhaust all the available credits and it is designed to behave that way. At this point in time, we do not have this mentioned on public documents however, I shall take this as a feedback and collaborate with my internal teams to update this on the documentation available for you. However, in the meanwhile, you may inform your students about the 20 instances cap that they have on their accounts."
   
   Yes, that's right, nothing as simple as a friendly warning for folk who are learning how to make use of such things - ask for too much and you're thrown out. Bodily. Face scraped by concrete as you land. But you'll not know that such a rule existed in advance so ..."surprise!!!".
   
   Now awaiting a response regarding how the student is supposed to get the account working again, whether what they were doing is retained, and whether there are any other undocumented traps waiting to be sprung along similar lines, e.g. "hey, we don't like that filename, so we've ransomwared all your data in S3. Pay $50 in bitcoin if you want to see it again".

14th May 2021 (pm)

First confirmation that m5s cannot be launched. Have to adjust assessment just a few days before the deadline - informed students also to avoid paying to use EMR, as that would be like some perverse financial reward. Not good. Not good at all.

18th May 2021 (noon)

Nothing received about Classrooms application.

Attention was drawn to two additional instance types made available - beyond those told about before - but still neither exists on this list: https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-supported-instance-types.html (one of those is in the m5 family, yet somehow very precisely selected to ensure that it is not one that is listed there)

20th May 2021 (noon)

Classroom 'approved'. Not clear how long it would have taken if I had not be agitating. But apparently an email goes out to students to Invite before it has been activated by the educator .. so the educator now gets more emails asking why it can't be used. Definitely not the best start. Oh, and beware the undocumented limit to 100 students - the application upload process will accept a longer list, so this is going to be FCFS for the many students who now have an extension of a week with a post invite email (more time taken) trying to put off all but the most needy/keen.

"Please check the current list of supported services by clicking [link]. AWS Educate Classroom Accounts may have slightly different permissions." - which can be discovered ... how?

Surely, AWS, this can be better? Students could be your future customers - is this the experience you want for them?

And build some spend control!

--

[1] This is distinct from the unlucky who, if using default instance types for EMR may have use-az3 in their mappings and at some point find their cluster attempting to launch there using m5.xlarge, which is not supported there. m4.xlarge is, but why have a default not supported across all AZs? Defaulting to m4 would, of course, annoy you in e.g. eu-west-3.

[2] Another trap for unwary students since the options are “Have AWS account” and “Don’t” and a Starter Account relates the latter better than the former.