AWS SQS body becomes totally different with just one subscription settting

This is a note what I found last week about the AWS SQS

Photo by Melanie Klepper on Unsplash

Background

Last week, I needed to create terraform files for existing AWS resources which were made manually. The resources are SQS subscribing a SNS topic. I created the following terraform file and let my backend server subscipbe the SQS.

resource "aws_sqs_queue" "test_queue" {
  name                       = "test_queue"
  visibility_timeout_seconds = 30
  delay_seconds              = 0
  fifo_queue                 = true
  max_message_size           = 262144
  message_retention_seconds  = 345600
  receive_wait_time_seconds  = 0
  sqs_managed_sse_enabled    = true
}

data "aws_iam_policy_document" "test_policy" {
  statement {
    sid    = "Allow SNS publish to SQS"
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["sns.amazonaws.com"]
    }

    actions   = ["SQS:SendMessage"]
    resources = [aws_sqs_queue.test_queue.arn]

    condition {
      test     = "ArnEquals"
      variable = "aws:SourceArn"
      values   = ["arn:aws:sns:my_sns_arn"]
    }
  }
}

resource "aws_sns_topic_subscription" "test_subscription" {
  topic_arn            = "arn:aws:sns:my_sns_arn"
  protocol             = "sqs"
  endpoint             = aws_sqs_queue.test_queue.arn
}

However, the backend came to raise errors saying the body from the SQS was missing what it expected to include. Then started to I investigate.

What I did

I checked the body causing errors and found the following body.

{
  "Messages": [
    {
      "MessageId": "***",
      "ReceiptHandle": "***",
      "MD5OfBody": "***",
      "Body": "{\n  \"Type\" : \"Notification\",\n  \"MessageId\" : \"***\",\n  \"TopicArn\" : \"arn:aws:sns:my_sns_arn\",\n  \"Message\" : \"custom body",\n  \"Timestamp\" : \"2024-10-06T23:48:11.305Z\",\n  \"UnsubscribeURL\" : \"***\",\n  \"MessageAttributes\" : {}\n}"
    }
  ]
}

I also checked the body which the backend expected.

{
  "Messages": [
    {
      "MessageId": "***",
      "ReceiptHandle": "***",
      "MD5OfBody": "***",
      "Body": "custom body"
    }
  ]
}

They had different structures from each other. This was why the error happened. The expected one had only custom body in it. In the other hand, there were many metadata in it. I googled the reason and found the raw message delivery setting.

Amazon SNS raw message delivery

The amount of data seem to be reduced by activating the setting. Actually, in my case, we didn’t really need the setting because we didn’t send much information in the body. However, the backend raised errors so I ticked the settings and the error had gone. All I had to do was just add one line to the terraform file.

resource "aws_sqs_queue" "test_queue" {
  name                       = "test_queue"
  visibility_timeout_seconds = 30
  delay_seconds              = 0
  fifo_queue                 = true
  max_message_size           = 262144
  message_retention_seconds  = 345600
  receive_wait_time_seconds  = 0
  sqs_managed_sse_enabled    = true
}

data "aws_iam_policy_document" "test_policy" {
  statement {
    sid    = "Allow SNS publish to SQS"
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["sns.amazonaws.com"]
    }

    actions   = ["SQS:SendMessage"]
    resources = [aws_sqs_queue.test_queue.arn]

    condition {
      test     = "ArnEquals"
      variable = "aws:SourceArn"
      values   = ["arn:aws:sns:my_sns_arn"]
    }
  }
}

resource "aws_sns_topic_subscription" "test_subscription" {
  topic_arn            = "arn:aws:sns:my_sns_arn"
  protocol             = "sqs"
  endpoint             = aws_sqs_queue.test_queue.arn
  raw_message_delivery = true # <- Add only this line
}

I’m always surprized that only one setting makes a big difference in infra development.

That’s it!