Understanding metadata endpoints and their role in AWS applications

Amazon Elastic Container Service (ECS) is a highly scalable container orchestration service that allows developers to run and manage Docker containers in the cloud. To enhance the usability and manageability of tasks running on ECS, AWS provides metadata endpoints. These endpoints expose valuable information about running tasks and containers. Let's explore what these endpoints are, how they work, and the data they provide.

Introduction to AWS Credential Management

When using AWS SDKs, understanding the default credential provider chain is crucial. This chain is a series of built-in credential providers invoked sequentially until valid credentials are found. For instance, in the PHP SDK, the credential chain operates as follows:

1. Environment Variables: The SDK searches for AWS access keys set as environment variables.
2. Web Identity Token: The SDK searches for IAM role and web identity token file settings.
3. Shared Config Files: Configuration is read from the shared AWS config and credentials files, using the "default" profile unless AWS_PROFILE is set.
4. IAM Identity Center: The SDK looks for IAM Identity Center configuration settings.
5. Credential Process: The SDK checks for the credential_process setting in the shared credentials file.
6. Shared Credentials File: AWS credentials or IAM role information are located in the shared credentials file.
7. ECS Credentials: The SDK uses AWS_CONTAINER_CREDENTIALS_RELATIVE_URI or AWS_CONTAINER_CREDENTIALS_FULL_URI for temporary credentials in ECS tasks.
8. EC2 Instance Profile: The SDK queries the EC2 Instance Metadata service (IMDSv1 or IMDSv2) for temporary credentials.

Note on EC2 Metadata Service (IMDS)

In EC2, the instance metadata service (IMDS) operates on different IPs depending on the version:

• IMDSv2: Requires a token to access metadata.

  TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` \
  && curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access

• IMDSv1: Directly queries the metadata endpoint without a token.

  curl http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access

The ECS Metadata Endpoint: An Overview

The ECS metadata endpoint allows containers running in ECS to retrieve details about their own execution environment. This information is critical for debugging, logging, monitoring, and dynamic configuration of containers.

Environment Variables Injected by ECS

When a container runs on ECS, the following environment variables are automatically injected by the ECS agent:

• AWS_EXECUTION_ENV: Indicates the execution environment. For tasks running on Fargate, this is set to AWS_ECS_FARGATE.
• ECS_CONTAINER_METADATA_URI: Points to the metadata endpoint for version 3 of the metadata API.
• ECS_CONTAINER_METADATA_URI_V4: Points to the metadata endpoint for version 4 of the metadata API.
• ECS_AGENT_URI: Points to the agent API endpoint for interaction with the ECS container agent.
• AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: Specifies the relative URI for accessing task role credentials.

For example, a running container might have the following environment variables:

AWS_EXECUTION_ENV=AWS_ECS_FARGATE
ECS_CONTAINER_METADATA_URI=http://169.254.170.2/v3/abcdefghi-0353131915
ECS_CONTAINER_METADATA_URI_V4=http://169.254.170.2/v4/abcdefghi-0353131915
ECS_AGENT_URI=http://169.254.170.2/api/abcdefghi-0353131915
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/v2/credentials/some-guid

If AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is set, the full URL for retrieving credentials must be built using the default ECS IP 169.254.170.2. For example:

http://169.254.170.2/v2/credentials/abcdefghi

To retrieve the credentials, you can use:

curl http://169.254.170.2/v2/credentials/abcdefghi

Response:

{
  "RoleArn": "arn:aws:iam::xxxx:role/elastic-prod-portal-task-role",
  "AccessKeyId": "ASIATEMPACCESSKEY",
  "SecretAccessKey": "HERETHESECRETACCESSKEY",
  "Expiration": "2025-01-13T14:03:58Z"
}

Metadata API Versions

ECS_CONTAINER_METADATA_URI (Version 3)

Accessing the URL in ECS_CONTAINER_METADATA_URI provides container-level information, such as:

curl http://169.254.170.2/v3/970ab5aa57c845e1955e744297938c15-0353131915

Response:

{
  "DockerId": "970ab5aa57c845e1955e744297938c15",
  "Name": "portal",
  "DockerName": "portal",
  "Image": "xxxx.dkr.ecr.eu-west-1.amazonaws.com/elastic-prod-portal:a000fd4e0549994378bc7afac9a51e8d288cefc7",
  "ImageID": "sha256:5671c25d67b3e743f98d122643e27f2da276b8a058831236e57eabf171b4cc56",
  "Labels": {
    "com.amazonaws.ecs.cluster": "arn:aws:ecs:eu-west-1:xxxx:cluster/elastic-prod-cluster",
    "com.amazonaws.ecs.container-name": "portal",
    "com.amazonaws.ecs.task-arn": "arn:aws:ecs:eu-west-1:xxxx:task/elastic-prod-cluster/970ab5aa57c845e1955e744297938c15",
    "com.amazonaws.ecs.task-definition-family": "elastic-prod-portal",
    "com.amazonaws.ecs.task-definition-version": "196"
  },
  "DesiredStatus": "RUNNING",
  "KnownStatus": "RUNNING",
  "Limits": {
    "CPU": 2
  },
  "CreatedAt": "2025-01-10T17:47:19.053845376Z",
  "StartedAt": "2025-01-10T17:47:19.053845376Z",
  "Type": "NORMAL",
  "Networks": [
    {
      "NetworkMode": "awsvpc",
      "IPv4Addresses": ["10.1.23.61"]
    }
  ]
}

ECS_CONTAINER_METADATA_URI_V4 (Version 4)

Version 4 extends the metadata by including additional details, such as logging configurations and network details. For instance:

curl http://169.254.170.2/v4/970ab5aa57c845e1955e744297938c15-0353131915

Response:

{
  "DockerId": "970ab5aa57c845e1955e744297938c15",
  "Name": "portal",
  "DockerName": "portal",
  "Image": "xxxx.dkr.ecr.eu-west-1.amazonaws.com/elastic-prod-portal:a000fd4e0549994378bc7afac9a51e8d288cefc7",
  "ImageID": "sha256:5671c25d67b3e743f98d122643e27f2da276b8a058831236e57eabf171b4cc56",
  "Labels": {
    "com.amazonaws.ecs.cluster": "arn:aws:ecs:eu-west-1:xxxx:cluster/elastic-prod-cluster",
    "com.amazonaws.ecs.container-name": "portal",
    "com.amazonaws.ecs.task-arn": "arn:aws:ecs:eu-west-1:xxxx:task/elastic-prod-cluster/970ab5aa57c845e1955e744297938c15",
    "com.amazonaws.ecs.task-definition-family": "elastic-prod-portal",
    "com.amazonaws.ecs.task-definition-version": "196"
  },
  "DesiredStatus": "RUNNING",
  "KnownStatus": "RUNNING",
  "Limits": {
    "CPU": 2
  },
  "CreatedAt": "2025-01-10T17:47:19.053845376Z",
  "StartedAt": "2025-01-10T17:47:19.053845376Z",
  "Type": "NORMAL",
  "LogDriver": "awslogs",
  "LogOptions": {
    "awslogs-group": "elastic-prod-portal-ND3sFzaK",
    "awslogs-region": "eu-west-1",
    "awslogs-stream": "portal_portal/portal/970ab5aa57c845e1955e744297938c15"
  }
}

Inconsistencies between EC2 and Fargate metadata endpoints

Important note there is a special endpoint that ends with /taskWithTags. This endpoint does not work on Fargate tasks it currently always gives back a 404 not found.

It does work on EC2 based ECS tasks.

Should you run into issues with your metadata endpoints? Feel free to reach out and ask us for help.