processing-s3-uploads-with-step-functions

The full skill.

Original instructions from the publisher’s SKILL.md

# Step Functions Workflow: Route S3 Uploads to Lambda or Fargate

## Overview

This skill deploys an event-driven workflow using AWS CLI. When a file is uploaded to
an S3 bucket, EventBridge triggers a Step Functions state machine. The state machine
checks the file size and routes processing to either a Lambda function (files ≤ 6 MB)
or a Fargate task (files > 6 MB).

The architecture includes:

- An S3 bucket with EventBridge notifications enabled
- An EventBridge rule that triggers Step Functions on S3 object creation
- A Step Functions state machine with a Choice state for routing
- A Lambda function for processing small files
- An ECS Fargate task for processing large files
- A VPC with two subnets, internet gateway, and security group
- An ECR repository for the Fargate container image
- Scoped IAM roles for Lambda, Step Functions, and ECS tasks

Use this skill when:

- You need to process S3 uploads with different compute based on file size
- You want a serverless workflow that can handle both small and large files
- You need Step Functions orchestration with Lambda and Fargate

Do not use this skill when:

- All files are small enough for Lambda (use S3 → Lambda directly)
- You need real-time streaming (use Kinesis)
- You don't need file-size-based routing

## Prerequisites

1. **AWS CLI v2** — Installed and configured. Verify with `aws sts get-caller-identity`.
2. **Python 3.12** — For the Lambda function runtime.
3. **Docker** — For building and pushing the Fargate container image.

## Parameters

- bucket_name (required): Name for the S3 bucket (globally unique, lowercase, 3-63 characters)
- region (required): AWS region for all resources
- ecr_repo_name (required): Name for the ECR repository
- state_machine_name (required): Name for the Step Functions state machine
- kms_key_arn (optional): ARN of a KMS key for CloudWatch Logs encryption. If not provided, create one with `aws kms create-key --description "Key for CloudWatch Logs encryption" --region {region}`

Constraints for parameter acquisition:

- You MUST ask for all required parameters upfront in a single prompt
- You MUST support multiple input methods (direct input, file path, URL)
- You MUST confirm successful acquisition of all parameters before proceeding
- You MUST validate that bucket_name follows S3 naming rules

## Procedures

### Step 0: Verify Dependencies

Constraints:

- You MUST verify the following tools are available: aws-cli, python3 (3.12+), docker
- You MUST inform the user about any missing tools with a clear message
- You MUST ask if the user wants to proceed despite missing tools
- You MUST respect the customer's decision to abort at any point
- You MUST explain to the customer what step is being executed, why, and which tool is being called

### Step 1: Retrieve AWS Account ID

Constraints:

- You MUST retrieve the account ID with: `aws sts get-caller-identity --query 'Account' --output text`
- You MUST store the result as {account_id} for use in all subsequent steps
- You MUST abort if credentials are not configured

### Step 2: Get the Default VPC and Networking

Constraints:

- You MUST retrieve the default VPC ID with:
  `aws ec2 describe-vpcs --filters Name=isDefault,Values=true --query 'Vpcs[0].VpcId' --output text --region {region}`
- If no default VPC exists, inform the user they must create one with `aws ec2 create-default-vpc --region {region}` or provide a VPC ID manually
- You MUST retrieve two subnet IDs from the default VPC:
  `aws ec2 describe-subnets --filters Name=vpc-id,Values={vpc_id} --query 'Subnets[0:2].SubnetId' --output text --region {region}`
- You MUST create a security group in the default VPC:
  `aws ec2 create-security-group --group-name fargate-sg --description "Security group for Fargate tasks" --vpc-id {vpc_id} --region {region}`
- You MUST configure security group egress rules to allow only HTTPS and DNS outbound. First revoke the default allow-all egress rule:
  `aws ec2 revoke-security-group-egress --group-id {sg_id} --ip-permissions IpProtocol=-1,IpRanges='[{CidrIp=0.0.0.0/0}]' --region {region}`
  Then add scoped rules:
  `aws ec2 authorize-security-group-egress --group-id {sg_id} --protocol tcp --port 443 --cidr 0.0.0.0/0 --region {region}` and
  `aws ec2 authorize-security-group-egress --group-id {sg_id} --protocol udp --port 53 --cidr 0.0.0.0/0 --region {region}`
- You MUST recommend VPC endpoints for S3 and CloudWatch Logs for production workloads to avoid internet-routed traffic and eliminate the need for broad egress rules
- You MUST capture {vpc_id}, {subnet1_id}, {subnet2_id}, and {sg_id} for use in later steps

### Step 3: Create the ECR Repository

Constraints:

- You MUST create the repository with:
  `aws ecr create-repository --repository-name {ecr_repo_name} --region {region}`
- You MUST capture the repositoryUri from the response

### Step 4: Build and Push the Container Image

Constraints:

- You MUST verify Docker is installed by running `docker --version`. If Docker is not installed, instruct the user to install it from https://docs.docker.com/get-docker/ and abort until it is available
- You MUST authenticate Docker with ECR:
  `aws ecr get-login-password --region {region} | docker login --username AWS --password-stdin {account_id}.dkr.ecr.{region}.amazonaws.com`
- The Dockerfile and processor code are in `scripts/Dockerfile` and `scripts/fargate_processor.py`
- You MUST build and push the image from the scripts directory:

  ```
  cd scripts
  docker build --platform linux/amd64 -t {ecr_repo_name} .
  docker tag {ecr_repo_name}:latest {account_id}.dkr.ecr.{region}.amazonaws.com/{ecr_repo_name}:latest
  docker push {account_id}.dkr.ecr.{region}.amazonaws.com/{ecr_repo_name}:latest
  cd ..
  ```

### Step 5: Create IAM Roles

Follow the detailed instructions in `references/iam-roles.md` to create all IAM roles (Lambda, ECS task execution, ECS task, Step Functions, and EventBridge roles).

- You MUST wait at least 10 seconds for IAM role propagation

### Step 6: Create the Lambda Function

Constraints:

- The function code is in `scripts/lambda_function.py`
- You MUST be in the skill root directory before packaging and creating the function
- You MUST package it with: `python3 -c "import zipfile,io; z=io.BytesIO(); f=zipfile.ZipFile(z,'w'); f.writestr('lambda_function.py', open('scripts/lambda_function.py').read()); f.close(); open('/tmp/lambda_function.zip','wb').write(z.getvalue())"`
- You MUST create the function with:

  ```
  aws lambda create-function \
      --function-name sfn-file-processor \
      --runtime python3.12 \
      --handler lambda_function.lambda_handler \
      --role arn:aws:iam::{account_id}:role/sfn-lambda-role \
      --zip-file fileb:///tmp/lambda_function.zip \
      --timeout 60 \
      --architectures x86_64 \
      --region {region}
  ```

- You MUST verify the function was created with:
  `aws lambda get-function --function-name sfn-file-processor --region {region}`

### Step 7: Create the CloudWatch Log Group

Constraints:

- You MUST create the log group for Fargate:
  `aws logs create-log-group --log-group-name /StepFunctionFargateTask --region {region}`
- You MUST encrypt the log group with a KMS key:
  `aws logs associate-kms-key --log-group-name /StepFunctionFargateTask --kms-key-arn {kms_key_arn} --region {region}`

### Step 8: Create the ECS Cluster and Task Definition

Follow the detailed instructions in `references/ecs-task-definition.md` to create the ECS cluster and register the Fargate task definition.

- You MUST capture the task definition ARN from the response

### Step 9: Create the S3 Bucket with EventBridge Notifications

Constraints:

- You MUST create the bucket with:
  `aws s3api create-bucket --bucket {bucket_name} --region {region} --create-bucket-configuration LocationConstraint={region}`
- You MUST NOT include `--create-bucket-configuration` if region is us-east-1
- You MUST enable EventBridge notifications on the bucket:
  `aws s3api put-bucket-notification-configuration --bucket {bucket_name} --notification-configuration '{"EventBridgeConfiguration": {}}' --region {region}`
- You MUST enable default encryption on the bucket:
  `aws s3api put-bucket-encryption --bucket {bucket_name} --server-side-encryption-configuration '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"aws:kms"}}]}' --region {region}`

### Step 10: Create the Step Functions State Machine

Constraints:

- The state machine definition is in `scripts/statemachine.asl.json`
- You MUST create a working copy and replace all placeholders:

  ```
  sed -e 's|${LambdaFunction}|arn:aws:lambda:{region}:{account_id}:function:sfn-file-processor|g' \
      -e 's|${Cluster}|arn:aws:ecs:{region}:{account_id}:cluster/sfn-cluster|g' \
      -e 's|${TaskDefinition}|{task_definition_arn}|g' \
      -e 's|${Subnet1}|{subnet1_id}|g' \
      -e 's|${Subnet2}|{subnet2_id}|g' \
      -e 's|${SecurityGroup}|{sg_id}|g' \
      scripts/statemachine.asl.json > /tmp/statemachine.asl.json
  ```

- You MUST create the state machine with:

  ```
  aws stepfunctions create-state-machine \
      --name {state_machine_name} \
      --definition file:///tmp/statemachine.asl.json \
      --role-arn arn:aws:iam::{account_id}:role/sfn-state-machine-role \
      --type STANDARD \
      --region {region}
  ```

- You MUST capture the stateMachineArn from the response

### Step 11: Create the EventBridge Rule

Constraints:

- You MUST create the EventBridge rule to trigger on S3 object creation:

  ```
  aws events put-rule \
      --name s3-to-stepfunctions \
      --event-pattern '{
        "source": ["aws.s3"],
        "detail-type": ["Object Created"],
        "detail": {
          "bucket": {
            "name": ["{bucket_name}"]
          }
        }
      }' \
      --region {region}
  ```

- You MUST add the state machine as a target:

  ```
  aws events put-targets \
      --rule s3-to-stepfunctions \
      --targets '[{
        "Id": "StepFunctionsTarget",
        "Arn": "{state_machine_arn}",
        "RoleArn": "arn:aws:iam::{account_id}:role/sfn-eventbridge-role"
      }]' \
      --region {region}
  ```

### Step 12: Configure Monitoring

Constraints:

- You MUST create a Dead Letter Queue for failed EventBridge invocations:
  `aws sqs create-queue --queue-name s3-to-stepfunctions-dlq --region {region}`
- You MUST update the EventBridge target to attach the DLQ:

  ```
  aws events put-targets \
      --rule s3-to-stepfunctions \
      --targets '[{
        "Id": "StepFunctionsTarget",
        "Arn": "{state_machine_arn}",
        "RoleArn": "arn:aws:iam::{account_id}:role/sfn-eventbridge-role",
        "DeadLetterConfig": {
          "Arn": "arn:aws:sqs:{region}:{account_id}:s3-to-stepfunctions-dlq"
        }
      }]' \
      --region {region}
  ```

- You MUST create a CloudWatch alarm for Step Functions execution failures:
  `aws cloudwatch put-metric-alarm --alarm-name sfn-execution-failures --metric-name ExecutionsFailed --namespace AWS/States --statistic Sum --period 300 --threshold 1 --comparison-operator GreaterThanOrEqualToThreshold --evaluation-periods 1 --dimensions Name=StateMachineArn,Value={state_machine_arn} --region {region}`

### Step 13: Validate

Constraints:

- You MUST test with a small file (< 6 MB) to verify Lambda processing:

  ```
  echo 'test data' > /tmp/small-file.txt
  aws s3 cp /tmp/small-file.txt s3://{bucket_name}/small-file.txt --region {region}
  ```

- You MUST wait 15 seconds then check the Step Functions execution:
  `aws stepfunctions list-executions --state-machine-arn {state_machine_arn} --region {region}`
- You MUST verify the execution succeeded and routed to Lambda
- You MUST provide a summary of all created resources including: VPC ID, subnet IDs, security group ID, ECR repo URI, ECS cluster ARN, task definition ARN, Lambda function ARN, state machine ARN, bucket name, and EventBridge rule name

## Troubleshooting

### EventBridge rule not triggering

- Verify EventBridge notifications are enabled on the bucket: `aws s3api get-bucket-notification-configuration --bucket {bucket_name}`
- Verify the rule exists: `aws events describe-rule --name s3-to-stepfunctions --region {region}`
- Check that the target has the correct state machine ARN and role

### Step Functions execution fails at Fargate task

- Verify the container image exists in ECR: `aws ecr describe-images --repository-name {ecr_repo_name} --region {region}`
- Check that the subnets have internet access (route table with IGW)
- Verify the security group allows outbound traffic
- Check CloudWatch Logs at `/StepFunctionFargateTask`

### Lambda invocation fails

- Check CloudWatch Logs: `aws logs tail /aws/lambda/sfn-file-processor --region {region}`
- Verify the Step Functions role has `lambda:InvokeFunction` permission

### IAM PassRole errors

- The Step Functions role must have `iam:PassRole` for both the ECS execution role and task role ARNs

### Fargate task stuck in PROVISIONING

- Verify the subnets have auto-assign public IP enabled
- Verify the internet gateway is attached and route table has 0.0.0.0/0 route

## Security Considerations

- Fargate tasks with public IPs are exposed to the internet. Revoke the default allow-all egress rule and configure scoped egress: `aws ec2 revoke-security-group-egress --group-id {sg_id} --ip-permissions IpProtocol=-1,IpRanges='[{CidrIp=0.0.0.0/0}]'` then add `aws ec2 authorize-security-group-egress --group-id {sg_id} --protocol tcp --port 443 --cidr 0.0.0.0/0` and `aws ec2 authorize-security-group-egress --group-id {sg_id} --protocol udp --port 53 --cidr 0.0.0.0/0`. For production, consider using VPC endpoints for S3 and CloudWatch Logs instead of internet-routed traffic.
- Scan container images for vulnerabilities before pushing to ECR. Enable ECR image scanning with: `aws ecr put-image-scanning-configuration --repository-name {ecr_repo_name} --image-scanning-configuration scanOnPush=true --region {region}`
- Use IAM roles for credentials — never hardcode access keys in container code.
- Enable encryption at rest for the S3 bucket: `aws s3api put-bucket-encryption --bucket {bucket_name} --server-side-encryption-configuration '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"aws:kms"}}]}'`
- Enable CloudWatch Logs encryption for Fargate container logs: `aws logs associate-kms-key --log-group-name /StepFunctionFargateTask --kms-key-arn <KMS_KEY_ARN>`
- Configure a Dead Letter Queue on the EventBridge rule for failed invocations
- Set up CloudWatch alarms on Step Functions execution failures for operational visibility

## Version information

- **AWS CLI**: 2.x
- **Python runtime**: 3.12
- **Last validated**: 2026-04-27

## Additional Resources

- [Step Functions developer guide](https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html)
- [EventBridge S3 events](https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventBridge.html)
- [Fargate task definitions](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definitions.html)
- [ECR pushing images](https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html)
- [Step Functions Fargate integration](https://docs.aws.amazon.com/step-functions/latest/dg/connect-ecs.html)