AWS EC2 Maintenance Announcements

About the need to automate the AWS EC2 Maintenance Announcement’s tasks

As a common practice in cloud environments the cloud provider will have to go through infrastructure maintenances to keep the underlying data center infrastructure current and up to date.
Those maintenances typically are live-updates with less to no impact but some may have impacts to services used by clients – such like EC2 intances in an AWS context. 

AWS typically sends an email to the account holder email address as well as it is announced in the AWS Console. Both will require either access to the email account that receives the note or returning check through the AWS Console to become aware of an upcoming maintenance schedule. Both option do not scale, do not provide an ability to automate, require hands-on to not miss an announcement which might impact a main or core business function … any many more. Specifically in larger organizations where hundreds if not thousands of EC2 instances are deployed and utilized across a number of teams and organizations to support business needs and functions an automated process to handle a maintenance announcement becomes apparent and not just a help but a very need.

Overall it is as simple as a full instance stop and start is required to ensure instance announced in the maintenance notification is being moved off of the underlying hardware where it is currently deployed and executed on. Therefore either AWS will do a stop and start of those EC2 instances which got announced if no action is taken or someone with enough access rights to the AWS Console, the AWS EC2 API would have to execute a stop, start to initiate an EC2 relocation to a different underlying host in the AWS infrastructure.

A more actionable and automated process towards an ability to automatically let EC2 instances be stopped and started could be done by various options.

Automation based on Resource Tags

Resource Tags will be tremendously helpful if used and utilized properly as for example a tag could be set to identify a given environment an instance would belong to – such like Development vs Production vs Testing

A few thoughts around Resource Tags are collected in this post: https://into-the-cloud.mechmann.com/2020/06/24/resource-tags-in-cloud-deployments/

Monitoring driven

Monitoring applications such like Zabbix which collect instance level metrics through an agent can be used to read/pull the instance metadata.

A so called Template with an Item would be created. Alongside the Agent a so called UserParameter would be created which returns the EC2 metadata as a JSON formatted output

curl http://169.254.169.254/latest/meta-data/events/maintenance/scheduled

The UserParameter would be assigned to the Item where a schedule would be set (i.e. it would be executed once a day). A so called Trigger can be created to initiate an automated execution through the AWS EC2 API this then pre-checked for execution by validation of a given and correctly set resource tag.

Since this Post is about the AWS EC2 Maintenance Announcement the Monitoring driven approach stays high level described.

Asset Management driven

In case a repository of all deployed instances is collected throughout a day to maintain a library of workloads and how/where those are deployed to the AWS EC2 Describe API can provided similar information as described above in the Monitoring driven approach.

The daily data collection could be done via an AWS Lambda function. However the overall approach would be very similar to the above mentioned solution.

Event driven via AWS Cloudwatch

Last but not least an event driven option which would be fully cloud native one will be described in this paragraph.

Event Rules in Cloudwatch can be used to initiate an action to a target. In case of an instance retirement announcement an event rules can help to get proper and required actions automated.

The eventTypeCode is called: AWS_EC2_PERSISTENT_INSTANCE_RETIREMENT_SCHEDULED

Within the Targets an automation through SSM can be selected where then the event as described above would initiate an EC2 instance restart automatically.

Fazit

Many options can be utilized to allow an automated processing whenever AWS maintenances would be announced. For sure there is not one path only which otherwise helps to keep creativity moving and business needs can be met most appropriately.