Skip to main content

Set Up Monitoring and Observability

Objective: Deploy Prometheus, Grafana, and configure dashboards.

Description: This subtask covers creating Docker services for the monitoring stack and configuring service metrics collection.

Dependencies: Create Kubernetes Deployment Manifests

Details:

  • Add Prometheus, Grafana, and exporters to docker-compose.yml.
  • Configure dashboards and metrics collection for all core services.
  • Store configuration in the docker/ and infrastructure/ directories.

Status: Done

Test Strategy: Access the Grafana dashboard and verify metrics are being collected for all services.

Monitoring Stack Architecture​

flowchart TD
  subgraph Monitoring
    P[Prometheus]
    G[Grafana]
    E1[Postgres Exporter]
    E2[Redis Exporter]
    E3[Node Exporter]
    E4[Milvus Exporter]
  end
  P --> G
  E1 --> P
  E2 --> P
  E3 --> P
  E4 --> P
  P -->|Scrapes| PG[PostgreSQL]
  P -->|Scrapes| RS[Redis]
  P -->|Scrapes| MV[Milvus]

Explanatory Notes​

  • Prometheus: Collects and stores time-series metrics from exporters and services.
  • Grafana: Visualizes metrics and provides dashboards for real-time monitoring.
  • Exporters: Bridge between services and Prometheus, exposing metrics in a standard format.
  • Dashboards: Enable rapid diagnosis of issues and performance bottlenecks.
  • Alerting: Set up alerts for critical metrics to enable proactive response.
  • Best Practices:
    • Regularly review and update dashboards.
    • Monitor resource usage and service health.
    • Document alerting rules and escalation procedures.
  • Troubleshooting:
    • Check Prometheus and Grafana logs for errors.
    • Validate exporter endpoints and data freshness.