AWS CloudWatch
This Guide Integrates:
| ✅ Logs | ✖️ Metrics | ✖️ Traces |
Note: Cloud-only feature. The self-hosted open source version of HyperDX does not support this integration.
Setup HyperDX CloudWatch Integration
There are two options (automated vs manual) to create a AWS Lambda function that collects CloudWatch logs and sends them to HyperDX in bulk over HTTP.
Automated CloudFormation deployment
Start the automated deployment
To deploy this project, click the button that matches the region you wish to deploy your Stack to:
| REGION | DEPLOYMENT |
|---|---|
us-east-1 |  (opens in a new tab) |
us-east-2 | (opens in a new tab) |
us-west-1 | (opens in a new tab) |
us-west-2 | (opens in a new tab) |
eu-central-1 | (opens in a new tab) |
eu-north-1 | (opens in a new tab) |
eu-west-1 | (opens in a new tab) |
eu-west-2 | (opens in a new tab) |
eu-west-3 | (opens in a new tab) |
sa-east-1 | (opens in a new tab) |
ca-central-1 | (opens in a new tab) |
ap-northeast-1 | (opens in a new tab) |
ap-northeast-2 | (opens in a new tab) |
ap-northeast-3 | (opens in a new tab) |
ap-south-1 | (opens in a new tab) |
ap-southeast-1 | (opens in a new tab) |
ap-southeast-2 | (opens in a new tab) |
Specify stack details
Specify the stack details as per the table below, check the checkboxes and select Create stack.
| Parameter | Description |
|---|---|
| hyperdxIngestionKey (Required) | The Ingestion API Key (opens in a new tab) of the account you want to ship to. |
| logGroup* | CloudWatch Log Group name from where you want to send logs. |
| hyperdxServiceName | The service name used to identify your service in the HyperDX app, it can be any name you want. |
hyperdxFormat (Default: text) | json or text. If json, the Lambda function will attempt to parse the message field as JSON and populate the event data with the parsed fields. |
| hyperdxEnrich | Enrich CloudWatch events with custom properties, formatted as key1=value1;key2=value2. |
requestTimeout (Default: 15) | Timeout in seconds for each http request for sending logs into HyperDX. |
Manual configuration
Create a new Lambda function
- Open the AWS Lambda Console, and click Create function.
- Choose Author from scratch.
- In Name, add the log type to the name of the function.
- In Runtime, choose Python 3.9.
- Click Create Function.
After a few moments, you'll see configuration options for your Lambda function. You'll need this page later on, so keep it open.
Zip the source files
Clone the CloudWatch Logs Shipper - Lambda project from GitHub to your computer,
and zip the Python files in the src/ folder as follows:
git clone https://github.com/hyperdxio/hyperdx-aws-serverless.git \
&& cd hyperdx-aws-serverless/python3/cloudwatch/ \
&& make buildUpload the zip file and set environment variables
- In the Code source section, select Upload from > .zip file.
- Click Upload, and choose the zip file you created earlier
(
dist/hyperdx-cloudwatch-log-shipper.zip). - Click Save.
- Navigate to Configuration > Environment variables.
- Click Edit.
- Click Add environment variable.
- Fill in the Key and Value fields for each variable as per the table below:
| Parameter | Description |
|---|---|
| TOKEN (Required) | The Ingestion API Key (opens in a new tab) of the account you want to ship to. |
| TYPE | The service name used to identify your service in the HyperDX app, it can be any name you want. |
FORMAT (Default: text) | json or text. If json, the Lambda function will attempt to parse the message field as JSON and populate the event data with the parsed fields. |
| ENRICH | Enrich CloudWatch events with custom properties, formatted as key1=value1;key2=value2. |
REQUEST_TIMEOUT (Default: 15) | Timeout in seconds for each http request for sending logs into HyperDX. |
Set the CloudWatch Logs event trigger
- Find the Add triggers list (left side of the Designer panel) and choose CloudWatch Logs from this list.
- In the Log group field, select the applicable log group.
- Type a Filter name (required) and Filter pattern (optional).
- Click Add, and then click Save at the top of the page.
Automatically Stream All Log Groups (Optional)
Once the HyperDX Lambda function is installed, you can enable the AWS integration (opens in a new tab) to automatically stream all CloudWatch log groups within a given region. The integration will periodically scan for new log groups and update the forwarder Lambda to listen to the new log group.