How to expand the disk size in EC2

This is a note about what I did to resize the disk in EC2

Photo by Luana De Marco on Unsplash

Background

I use Redash for non-engineer team members to interact with our database using SQL. I host the Redash on EC2 and suddenly, last week, it raised a 502 error without any changes and none could use it at all. Then I checked the logs on the AWS console and found the following logs.

OSError: [Errno 28] No space left on device

So I needed to expand the disk size and thought it would be straightforward. However, it wasn’t that straightforward. Then I decided to document what I did in this article.

What I did

I quickly googled how to do it and found the following official documents. Basically, I followed the steps on them.

Request modifications to your EBS volumes

Extend the file system after resizing an EBS volume

Increase the size of the EBS volume

• Log in to the AWS Management Console.
• Go to the EC2 dashboard and select “Volumes” from the menu on the left.
• Select the target EBS volume and click “Actions” > “Modify Volume”.
• Specify the new size and click “Modify” to confirm.

Stop the EC2 instance

• Select “Instances” on the EC2 dashboard.
• Locate and select the instance ID that runs the Redash server
• Click “Instance state” > “Stop instance” to stop the instance.

Detach the EBS volume

• With the stopped instance selected, open the “Storage” tab at the bottom.
• Right-click the target EBS volume and select “Detach Volume.

Create a new EC2 instance

• Click “Instances” > “Launch Instances”.
• Create a temporary instance with the minimum required settings.

I want to mention two things which I could have done better. Firstly, I needed to do SSH with this instance later and forgot to allow me to do it. In this step, I needed to set up an SSH configuration. I think the easiest way is to set your key pair and set a security group for SSH.

This is the terraform file for the security group.

resource "aws_security_group" "allow_ec2_connect" {
  name        = "allow_ec2_connect"
  description = "Allow EC2 Instance Connect inbound traffic"
  vpc_id      = data.aws_vpc.default.id

  ingress {
    description = "EC2 Instance Connect"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "allow_ec2_connect"
  }
}

Secondly, I needed to attach the volume which I detached from the Redash instance with this instance and I couldn’t attach the volume to an instance in a different availability zone (AZ). In this step, I needed to select the same AZ as the Redash one.

Attach the detached EBS volume to the new instance

• Select the target volume in “Volumes” and click “Actions” > “Attach Volume”.
• Select the newly created instance and attach it.

Extend the volume

• Connect to the new instance via SSH.
• Extend the partition and file system with the following commands

sudo growpart /dev/xvdf 1
sudo resize2fs /dev/xvdf1

Device names may vary depending on your environment. You need to change xvdf the device name you selected in the previous step. I put the actual logs below.

[ec2-user@xxx ~]$ sudo lsblk
NAME      MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
xvda      202:0    0   8G  0 disk
├─xvda1   202:1    0   8G  0 part /
├─xvda127 259:0    0   1M  0 part
└─xvda128 259:1    0  10M  0 part /boot/efi
xvdp      202:240  0  75G  0 disk
└─xvdp1   202:241  0  50G  0 part

[ec2-user@xxx ~]$ sudo growpart /dev/xvdp 1
CHANGED: partition=1 start=2048 old: size=104855519 end=104857567 new: size=157284319 end=157286367

[ec2-user@xxx ~]$ sudo resize2fs /dev/xvdp1
resize2fs 1.46.5 (30-Dec-2021)
Please run 'e2fsck -f /dev/xvdp1' first.

[ec2-user@xxx ~]$ sudo e2fsck -f /dev/xvdp1
e2fsck 1.46.5 (30-Dec-2021)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
cloudimg-rootfs: 703944/6400000 files (0.1% non-contiguous), 13102843/13106939 blocks

[ec2-user@xxx ~]$ sudo resize2fs /dev/xvdp1
resize2fs 1.46.5 (30-Dec-2021)
Resizing the filesystem on /dev/xvdp1 to 19660539 (4k) blocks.
The filesystem on /dev/xvdp1 is now 19660539 (4k) blocks long.

Return the EBS volume to the original instance

• Stop the new instance.
• Detach the EBS volume from the new instance.
• Reattach the volume to the original instance.

Start the original instance

• Select the Redash instance and click “Instance state” -> “Start instance”.

Then the Redash started to work again.

That’s it!