Integrating DataDog with AWS for Real-Time Monitoring and Logging

Introduction

DataDog is a powerful monitoring and analytics platform that helps businesses track the performance of their infrastructure. When integrated with AWS EC2, it provides real-time monitoring, allowing you to gain insights into your instances’ performance, troubleshoot issues, and manage logs effectively. In this tutorial, we’ll walk through the steps to integrate AWS EC2 with DataDog and view your logs on the DataDog dashboard.

Prerequisites
Before proceeding, ensure that you have the following:
An active AWS account.
Access to an EC2 instance.
A DataDog account (sign up if you don’t have one yet.

  Step 1: Set Up DataDog API Key
1.Log in to your DataDog account.
2.Navigate to the Integrations tab.
3.Under the API Keys section, click on New Key.
4.Copy the generated API Key. You’ll need this for the integration.

    Step 2: Install the DataDog Agent on AWS EC2
You can install the DataDog agent on your EC2 instance by following these steps:

1. SSH into your EC2 instance

   ssh -i your-key.pem ec2-user@your-ec2-public-ip

2. Update your system.

   sudo yum update -y

3. Install the DataDog agent
   For Amazon Linux 2 (use the appropriate version for your OS):

   DD_AGENT_MAJOR_VERSION=7 DD_API_KEY=your_datadog_api_key sudo sh -c "DD_AGENT_VERSION=7.31.1 \
   echo \"deb https://apt.datadoghq.com stable 7\" > /etc/apt/sources.list.d/datadog.list && \
   curl -L https://www.datadoghq.com/keys/datadog.asc | apt-key add -"
   sudo apt-get install datadog-agent

4. Start the DataDog agent.

   sudo systemctl start datadog-agent

   Step 3: Verify DataDog Agent Is Running
   To verify if the agent is working properly:

   sudo datadog-agent status

   Step 4: Configure Logs to Be Sent to DataDog
   Once the agent is installed, you can configure it to send logs from your EC2 instance to DataDog.

   1. Enable log collection in the DataDog agent configuration file
   Open the configuration file:

 sudo vim /etc/datadog-agent/datadog.yaml

2.Enable log collection
Find the logs_enabled line and set it to true.

logs_enabled: true

3. Restart the DataDog agent

sudo systemctl restart datadog-agent

4.Configure the log source
Now, configure your EC2 instance’s log sources to be collected by the agent:

sudo vim /etc/datadog-agent/conf.d/<log_source>.yaml

  Step 5: Monitor Logs on DataDog Dashboard
Once the DataDog agent is collecting logs from your EC2 instance, log in to your DataDog       dashboard, go to the Logs section, and you should see the logs coming from your EC2 instance. You   can filter, search, and visualize logs to monitor your application performance in real-time.