Category: CI/CD & Monitoring

What is Prometheus?

Prometheus is an open-source monitoring and alerting toolkit originally developed at SoundCloud. It is designed for reliability and scalability, ideal for collecting time-series metrics from cloud-native environments like Kubernetes or AWS. Prometheus stores data in a custom time-series database and uses a flexible query language called PromQL.

Prometheus works by scraping metrics from targets (like EC2, Node Exporter, or containers) at specified intervals. These metrics are then stored locally and can be queried or used to trigger alerts.

Prometheus supports service discovery, meaning it can automatically detect and begin monitoring new services based on changes in your infrastructure.

What is Grafana?

Grafana is an open-source analytics and visualization platform that integrates seamlessly with Prometheus. It helps users create interactive and customizable dashboards to visualize metrics data. Grafana supports alerts, templating, role-based access, and can be used to monitor everything from infrastructure and logs to user applications.

Grafana doesn’t collect data itself — it reads it from sources like Prometheus, InfluxDB, MySQL, Loki, and more.

How Prometheus and Grafana Work Together

• Prometheus collects and stores metrics from services and infrastructure.

• Grafana reads these metrics via Prometheus’s HTTP API and visualizes them.

• This combination provides real-time monitoring, alerting, and visual dashboards for DevOps and SRE teams.

  Install Prometheus and Grafana on AWS EC2

• You can set up Prometheus and Grafana on your EC2 instance in just a few steps.
  

  Install Prometheus

  # Amazon Linux 2
  sudo yum update -y
  
  # Ubuntu
  sudo apt-get update

• 2. Create a Prometheus user and folders

  sudo useradd --no-create-home --shell /bin/false prometheus
  
  sudo mkdir /etc/prometheus /var/lib/prometheus

• 3. Download and install Prometheus

  wget https://github.com/prometheus/prometheus/releases/download/v2.34.0/prometheus-2.34.0.linux-amd64.tar.gz
  tar -xvzf prometheus-2.34.0.linux-amd64.tar.gz
  cd prometheus-2.34.0.linux-amd64
  
  sudo cp prometheus promtool /usr/local/bin/
  sudo cp -r consoles console_libraries /etc/prometheus
  sudo cp prometheus.yml /etc/prometheus
  sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

  4. Create systemd service file

  sudo vim /etc/systemd/system/prometheus.service

• Paste the following:

  [Unit]
  Description=Prometheus
  After=network.target
  
  [Service]
  User=prometheus
  ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/
  Restart=always
  
  [Install]
  WantedBy=multi-user.target

  5. Start Prometheus

  sudo systemctl daemon-reexec
  
  sudo systemctl enable prometheus
  
  sudo systemctl start prometheus

• Prometheus will run on port 9090
  Check by visiting:
  

  http://<your-ec2-public-ip>:9090

• Install Grafana
  ubuntu

  sudo apt-get install -y software-properties-common
  sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
  sudo apt-get update
  sudo apt-get install grafana

  Amazon Linux 2

  sudo yum install -y https://dl.grafana.com/oss/release/grafana-8.3.0-1.x86_64.rpm

  Start Grafana

  sudo systemctl enable grafana-server
  
  sudo systemctl start grafana-server

  Grafana will run on port 3000
  Access it via:
  

  http://<your-ec2-public-ip>:3000

  Login to Grafana

  • Username: admin

  • Password: admin (you will be prompted to change it)

  Add Prometheus as a Data Source in Grafana

  1. Click the gear icon in the left menu → Data Sources.

  2. Select Prometheus.

  3. Set the URL to:
     http://<your-ec2-public-ip>:9090

  4. Click Save & Test to verify the connection.
     

     Using Node Exporter for System Metrics

     To monitor EC2-level system metrics like CPU, disk, and memory:

     1. Download and install Node Exporter:
        
        wget https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.linux-amd64.tar.gz
        tar -xvzf node_exporter-1.5.0.linux-amd64.tar.gz
        cd node_exporter-1.5.0.linux-amd64
        ./node_exporter
        
        

     2. Update  prometheus.yml

     scrape_configs:
     
     - job_name: 'node'
     
     static_configs:
     
     - targets: ['localhost:9100']

  3. Restart Prometheus to load changes:

               sudo systemctl restart prometheus

Conclusion

Prometheus and Grafana together form a powerful, open-source monitoring and visualization stack. Setting them up on AWS EC2 gives you full control of your metrics and dashboards. Whether you’re monitoring applications, infrastructure, or containers, this setup gives you the tools to ensure performance and reliability.

If you’re building applications on AWS, you often need to connect your EC2 instance (which hosts your app or website) to an Amazon RDS database for data storage. AWS makes this connection easy with an automatic connection feature that configures the necessary security settings for you.

In this tutorial, I’ll walk you through how to automatically connect an EC2 instance to an RDS database using the AWS Management Console.

Why Use the Automatic Connection Feature?

When connecting EC2 to RDS manually, you have to configure security groups to allow communication between them. This can be tricky and error-prone.

The automatic connection feature takes care of:

• Creating security groups for your EC2 instance and RDS database

• Setting up inbound/outbound rules that allow traffic only between the two

• Ensuring least privilege security

This lets you focus on building your application without worrying about network configuration details.

Prerequisites

• AWS account with permission to create EC2 and RDS resources

• Basic familiarity with AWS Management Console

• EC2 instance and RDS database in the same AWS Region and same VPC

Option 1: Automatically Connect EC2 to RDS Using the EC2 Console

Step 1: Launch an EC2 Instance

1. Open the EC2 Console in AWS.

2. Click Launch Instances.

3. Select your preferred AMI (e.g., Ubuntu Server).

4. Choose instance type (e.g., t2.micro for free tier).

5. Configure instance details — make sure to launch the instance in your desired VPC.

6. Add storage and tags as needed.

7. In Configure security group:

   • You can create a new security group or select an existing one.

   • Don’t worry about database access yet; this will be handled automatically.

8. Launch your instance.

Step 2: Create or Identify an RDS Database

1. Open the RDS Console.

2. Click Databases > Create database.

3. Choose MySQL or any database engine you prefer.

4. Choose Standard create and fill in the necessary fields.

5. Important: Choose the same VPC as your EC2 instance.

6. Complete other settings and create the database.

Step 3: Automatically Connect the EC2 Instance to RDS

1. Go back to the EC2 Console.

2. Select your EC2 instance.

3. In the Actions dropdown, choose Connect to database.

4. Select your RDS database from the list.

5. AWS will automatically:

   • Create and assign two security groups:

     • One for the EC2 instance (outbound rule to RDS security group).

     • One for the RDS database (inbound rule from EC2 security group).

   • Configure the security group rules to allow traffic on the database port (usually 3306 for MySQL).

6. You will see a confirmation once this setup completes.

Option 2: Automatically Connect EC2 to RDS Using the RDS Console

Alternatively, you can do the automatic connection starting from the RDS Console.

1. Open the RDS Console.

2. Select your database.

3. Click the Actions dropdown.

4. Choose Connect to instance.

5. Select your EC2 instance.

6. AWS will automatically create the security groups and configure access rules just like in Option 1.

---

How It Works Under the Hood

• Two security groups are created:

  • EC2 security group with outbound access to the RDS security group.

  • RDS security group with inbound access from the EC2 security group.

• This separation allows better security management and least privilege access.

• EC2 can now securely communicate with RDS on the database port (e.g., 3306).

---

What’s Next?

• Connect to your EC2 instance and test the database connection using your database client or application.

• Use the RDS database endpoint and credentials in your app’s configuration.

• If you’re running a WordPress site, for example, you would enter the RDS endpoint and DB credentials during setup.

---

Summary

Automatically connecting your EC2 instance to your RDS database via the AWS Management Console:

• Saves time and reduces mistakes configuring security groups

• Ensures secure, least-privileged communication between resources

• Makes your AWS architecture cleaner and easier to manage

Introduction

DataDog is a powerful monitoring and analytics platform that helps businesses track the performance of their infrastructure. When integrated with AWS EC2, it provides real-time monitoring, allowing you to gain insights into your instances’ performance, troubleshoot issues, and manage logs effectively. In this tutorial, we’ll walk through the steps to integrate AWS EC2 with DataDog and view your logs on the DataDog dashboard.

Prerequisites
Before proceeding, ensure that you have the following:
An active AWS account.
Access to an EC2 instance.
A DataDog account (sign up if you don’t have one yet.

  Step 1: Set Up DataDog API Key
1.Log in to your DataDog account.
2.Navigate to the Integrations tab.
3.Under the API Keys section, click on New Key.
4.Copy the generated API Key. You’ll need this for the integration.

    Step 2: Install the DataDog Agent on AWS EC2
You can install the DataDog agent on your EC2 instance by following these steps:

1. SSH into your EC2 instance

   ssh -i your-key.pem ec2-user@your-ec2-public-ip

2. Update your system.

   sudo yum update -y

3. Install the DataDog agent
   For Amazon Linux 2 (use the appropriate version for your OS):

   DD_AGENT_MAJOR_VERSION=7 DD_API_KEY=your_datadog_api_key sudo sh -c "DD_AGENT_VERSION=7.31.1 \
   echo \"deb https://apt.datadoghq.com stable 7\" > /etc/apt/sources.list.d/datadog.list && \
   curl -L https://www.datadoghq.com/keys/datadog.asc | apt-key add -"
   sudo apt-get install datadog-agent

4. Start the DataDog agent.

   sudo systemctl start datadog-agent

   Step 3: Verify DataDog Agent Is Running
   To verify if the agent is working properly:

   sudo datadog-agent status

   Step 4: Configure Logs to Be Sent to DataDog
   Once the agent is installed, you can configure it to send logs from your EC2 instance to DataDog.

   1. Enable log collection in the DataDog agent configuration file
   Open the configuration file:

 sudo vim /etc/datadog-agent/datadog.yaml

2.Enable log collection
Find the logs_enabled line and set it to true.

logs_enabled: true

3. Restart the DataDog agent

sudo systemctl restart datadog-agent

4.Configure the log source
Now, configure your EC2 instance’s log sources to be collected by the agent:

sudo vim /etc/datadog-agent/conf.d/<log_source>.yaml

  Step 5: Monitor Logs on DataDog Dashboard
Once the DataDog agent is collecting logs from your EC2 instance, log in to your DataDog       dashboard, go to the Logs section, and you should see the logs coming from your EC2 instance. You   can filter, search, and visualize logs to monitor your application performance in real-time.