AWS Lambda Message Consumer

It is an AWS Lambda function that consumes messages from an SQS queue, processes email templates stored in S3, and sends personalized emails using Amazon SES.

Description

This Lambda function is designed to consume email sending messages from an SQS queue, search for corresponding templates in Amazon S3, personalize them with dynamic variables, and send emails using Amazon SES. It is the final processor of the email sending system, where personalization and actual email sending takes place.

Architecture

SQS Queue → Lambda Consumer → S3 (Templates) → SES (Email Sending)

Features

• Message consumption from SQS queue
• Automatic template search in S3
• Template personalization with dynamic variables
• Email sending using Amazon SES
• Error handling and detailed logging
• Processing of multiple messages per invocation

Project Structure

├── message_consumer.py        # Main Lambda function
├── find_template_s3.py       # Template path search in S3
├── search_personalize_template.py # Template download and personalization from S3
├── send_personalized_email.py # Email sending with SES
├── config.py                 # AWS S3 and SES configuration
└── README.md                # Project documentation

Configuration

Required Environment Variables

Variable	Description	Example
S3_BUCKET_NAME	Name of the S3 bucket where templates are stored	bucket-templates
SES_REGION	Amazon SES region for email sending	us-east-1

Required IAM Permissions

The Lambda function needs the following permissions:

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sqs:ReceiveMessage",
                "sqs:DeleteMessage"
            ],
            "Resource": "arn:aws:sqs:region:account-id:queue-name"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::BUCKET_NAME",
                "arn:aws:s3:::BUCKET_NAME/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ses:SendEmail"
            ],
            "Resource": "*"
        }
    ]
}

Input Format

SQS Message Structure

The function consumes SQS messages with the following structure:

{
  "Records": [
    {
      "messageId": "message-id-123",
      "body": "{\"user_email\": \"user@example.com\", \"template_id\": \"welcome_user_external\", \"subject\": \"Welcome\", \"sender\": \"no-reply@osbornlab.com\", \"variables\": {\"user_name\": \"Sofía\"}}"
    }
  ]
}

Message Payload Structure

Each message in the body (JSON) must contain:

{
  "user_email": "user@example.com",
  "template_id": "folder_name",
  "subject": "Email subject",
  "sender": "no-reply@osbornlab.com",
  "variables": {                // optional
    "user_name": "Sofía",
    "otp_code": "123456"
  }
}

Important rules:

• Required fields: user_email, template_id, subject, sender
• The template_id must correspond to a folder in S3
• The template must have an index.html file in the corresponding folder
• variables is optional; if sent, the keys must match the placeholders in the HTML

Dynamic Variables Handling

Templates in S3 can contain placeholders in the form {{variable_name}}. The consumer replaces each placeholder with the value of the corresponding key within variables.

• Example: if the HTML contains Hello {{user_name}}, and in variables you send { "user_name": "Sofía" }, the result will be Hello Sofía.
• If a variable does not exist in variables, the placeholder remains unchanged in the HTML.

Template Structure in S3

Required Organization

For template processing to work correctly, templates must follow this structure in S3:

s3://bucket-templates/
├── welcome_user_external/
│   ├── index.html
│   └── images/
│       ├── header.png
│       └── logo.png
└── newsletter/
    ├── index.html
    └── styles/
        └── newsletter.css

Important rules:

• Each template must be in its own folder with the name of the template_id
• The main file must be named index.html
• Resources (images) must be in subfolders; to display in the email, they must be referenced with publicly accessible URLs (not relative S3 paths)
• Variables are replaced using the syntax {{variableName}} in the HTML

Template Processing

Personalization Flow

1. Search: The existence of template_id/index.html in S3 is validated (find_template_s3.py).
2. Download and personalization: The index.html is obtained and variables are replaced (search_personalize_template.py).
3. Sending: The personalized email is sent via SES (send_personalized_email.py).
4. Orchestration: message_consumer.py parses the SQS event and coordinates the flow.

Currently Supported Variables

Arbitrary variables are supported; any placeholder {{key}} present in the HTML will be replaced if variables["key"] exists.

Testing with Postman

Postman Configuration

1. Method: POST

2. URL: The API Gateway endpoint (for direct testing)

3. Headers:

   Content-Type: application/json

4. Body:

   • Select raw or JSON
   • Use the provided example structure

5. Send the request

Example Usage with cURL

# Simulate SQS message for testing
curl -X POST https://your-api-gateway-url \
  -H "Content-Type: application/json" \
  -d '{
    "Records": [
      {
        "messageId": "test-123",
        "body": "{\"user_email\": \"test@example.com\", \"template_id\": \"welcome_user_external\", \"subject\": \"Welcome\", \"sender\": \"no-reply@osbornlab.com\", \"variables\": {\"user_name\": \"Test User\"}}"
      }
    ]
  }'

Response Format

Successful Processing

The function does not return an HTTP response (it is an SQS consumer), but logs the following:

processing message with id: message-id-123
Template found at Key: welcome_user_external/index.html
Email sent! Message ID: 0000014a-f4d4-4073-89eb-123456789012

Error Handling

ERROR: Could not retrieve or process the email template.
ERROR: message message-id-123 is not a valid json.
Error to process message message-id-123: [error description]
WARNING: Template 'template_id/index.html' was not found in s3 (Error 404).
Error getting the template from S3: [error description]

Limitations and Considerations

• Maximum template size: Limited by S3 response size (5GB)
• Timeout: Configure appropriate timeout based on template size
• Memory: Adjust memory based on template complexity
• Rate Limits: SES has sending limits per second

Processing Flow

1. Consumption: Lambda receives messages from SQS
2. Parsing: The message JSON is parsed
3. Search: The template is searched in S3 using template_id
4. Download: The HTML content of the template is downloaded
5. Personalization: Dynamic variables are replaced
6. Sending: The email is sent using SES
7. Logging: The processing result is logged

Monitoring and Logs

CloudWatch Logs

Function logs are available in CloudWatch under:

/aws/lambda/Osborn-MessageConsumer

SES Configuration

1. Verify domain: Configure domain for email sending
2. Configure sender: Use the verified domain:`@osbornlab.com