Troubleshoot Boot and Networking Issues with New EC2 Serial Console

Fixing manufacturing points is likely one of the key duties of system and community directors. Actually, I’ve at all times discovered it to be one of the vital attention-grabbing components of infrastructure engineering. Diving as deep as wanted into the issue at hand, not solely do you (finally) have the satisfaction of fixing the difficulty, you additionally study a number of issues alongside the way in which, which you in all probability wouldn’t have been uncovered to below regular circumstances.

Working techniques actually current such alternatives. Over time, they’ve grown ever extra advanced, forcing directors to grasp a zillion configuration information and settings. Though infrastructure as code and automation have enormously improved provisioning and managing servers, there’s at all times room for errors and breakdowns that stop a system from beginning accurately. The record is limitless: lacking drivers, misconfigured file techniques, invalid community configuration, incorrect permissions, and so forth. To make issues worse, many points can successfully lock directors out of a system, stopping them from logging in, diagnosing the issue and making use of the suitable repair. The one choice is to have an out-of-band connection to your servers, and though clients may view the console output of an EC2 occasion, they couldn’t work together with it – till now.

Right this moment, I’m extraordinarily joyful to announce the EC2 Serial Console, a easy and safe approach to troubleshoot boot and community connectivity points by establishing a serial connection to your Amazon Elastic Compute Cloud (EC2) situations.

Introducing the EC2 Serial Console
EC2 Serial Console entry is offered for EC2 situations based mostly on the AWS Nitro System. It helps all main Linux distributions, FreeBSD, NetBSD, Microsoft Home windows, and VMWare.

With none want for a working community configuration, you may hook up with an occasion utilizing both a browser-based shell within the AWS Administration Console, or an SSH connection to a managed console server. No want for an sshd server to be working in your occasion: the one requirement is that the root account has been assigned a password, as that is the one you’ll use to log in. Then, you may enter instructions as when you have a keyboard and monitor straight hooked up to one of many occasion’s serial ports.

As well as, you may set off working system particular procedures:

• On Linux, you may set off a Magic SysRq command to generate a crash dump, kill processes, and so forth.
• On Home windows, you may interrupt the boot course of, and boot in protected mode utilizing Emergency Administration Service (EMS) and Particular Admin Console (SAC).

Gaining access to an occasion’s console is a privileged operation that ought to be tightly managed, which is why EC2 Serial Console entry shouldn’t be permitted by default on the account stage. When you allow entry in your account, it applies to all situations on this account. Directors can even apply controls on the group stage due to Service Management Insurance policies, and at occasion stage due to AWS Identification and Entry Administration (IAM) permissions. As you’ll anticipate, all communication with the EC2 Serial Console is encrypted, and we generate a singular key for every session.

Let’s do a fast demo with Linux. The method is comparable with different working techniques.

Connecting to the EC2 Serial Console with the AWS Administration Console
First, I launch an Amazon Linux 2 occasion. Logging in to it, I resolve to mangle the community configuration for its Ethernet community interface (/and so forth/sysconfig/network-scripts/ifcfg-eth0), setting a totally bogus static IP handle. PLEASE don’t do this on a manufacturing occasion!

Then, I reboot the occasion. Just a few seconds later, though the occasion is up and working within the EC2 console and port 22 is open in its Safety Group, I’m unable to hook up with it with SSH.

$ ssh -i ~/.ssh/mykey.pem ec2-user@ec2-Three-238-Eight-46.compute-1.amazonaws.com
ssh: hook up with host ec2-Three-238-Eight-46.compute-1.amazonaws.com port 22: Operation timed out

EC2 Serial Console to the rescue!

First, I want to permit console entry in my account. All it takes is ticking a field within the EC2 settings.

Then, proper clicking on the occasion’s title within the EC2 console, I choose Monitor and troubleshoot; then EC2 Serial Console.

This opens a brand new window confirming the occasion id and the serial port quantity to hook up with. I merely click on on Join.

This opens a brand new tab in my browser. Hitting Enter, I see the acquainted login immediate.

Amazon Linux 2
Kernel four.14.225-168.357.amzn2.x86_64 on an x86_64
ip-172-31-67-148 login:

Logging in as root, I’m relieved to get a pleasant shell immediate.

Enabling Magic SysRq for this session (sysctl -w kernel.sysrq=1), I first record out there instructions (CTRL-Zero + h), after which ask for a reminiscence report (CTRL-Zero + m). You’ll be able to click on on the picture beneath to get a bigger view.

Fairly cool! This might undoubtedly come in useful to troubleshoot advanced points. No want for this right here: I rapidly restore a sound configuration for the community interface, and I restart the community stack.

Attempting to hook up with the occasion once more, I can see that the issue is solved.

$ ssh -i ~/.ssh/mykey.pem ec2-user@ec2-Three-238-Eight-46.compute-1.amazonaws.com