📊 Observability, Metrics & Distributed Tracing
1️⃣ Monitoring vs Observability & The Three Pillars
Definition: Monitoring batata hai ki kya system toot chuka hai (known failure modes identify karna via thresholds), jabki Observability system ke internal execution states ko uske external telemetry outputs (Logs, Metrics, Traces) se deduce karne ki ability hai (debugging unknown-unknowns).
| Pillar | Characteristics | Trade-off & Storage Cost | Standard Tools |
|---|---|---|---|
| Metrics | Numeric, aggregatable time-series data (e.g. CPU %, HTTP request rate). | Very cheap to store; constant space over time. Lacks individual request context. | Prometheus, VictoriaMetrics, Datadog, InfluxDB |
| Logs | Timestamped discrete text/JSON event strings containing execution details. | High storage volume and indexing costs; expensive to retain at scale. | Grafana Loki, Elasticsearch/OpenSearch, FluentBit |
| Traces | End-to-end distributed transaction path showing timing across microservice hops. | Requires context propagation across network RPCs; requires sampling. | OpenTelemetry, Jaeger, Tempo, Zipkin |
2️⃣ Metric Types: Counter, Gauge, Histogram & Summary
| Metric Type | Behavior | Mathematical Reset | Example Metric |
|---|---|---|---|
| Counter | Monotonically increasing value; sirf increase hota hai ya restart par 0 hota hai. | Use rate() or increase() over time windows. |
http_requests_total |
| Gauge | Numerical value jo arbitrarily upar ya neeche ja sakti hai (Current snapshot). | Instant vector reading. | memory_usage_bytes, active_db_connections |
| Histogram | Observations ko configurable buckets (e.g. le="0.1", le="0.5") me count karta hai. |
Enables server-side percentile calculation ($P_{90}$, $P_{99}$) via histogram_quantile(). |
http_request_duration_seconds_bucket |
| Summary | Client-side calculate kiye gaye percentiles ($P_{50}$, $P_{95}$, $P_{99}$) directly expose karta hai. | Accurate percentiles on single instance, but cannot be aggregated across clusters. | rpc_latency_summary |
3️⃣ Prometheus TSDB & PromQL Deep Dive
Prometheus ek pull-based time-series engine hai jo endpoints par /metrics scrape karta hai. Essential PromQL queries:
# 1. Calculate per-second request rate over 5-minute window
rate(http_requests_total{status=~"5.."}[5m])
# 2. Compute 99th Percentile Latency across all Gateway instances
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
# 3. Memory saturation alert calculation (> 85% for 10 minutes)
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 854️⃣ Distributed Tracing & W3C Context Propagation
Jab ek user request dozens of downstream services (Gateway → Auth → Order → Payment) ko touch karti hai, tab cross-service correlation maintain karne ke liye W3C Trace Context headers propagate kiye jaate hain:
# HTTP Header passed between microservices:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
| | | |
| +-- Trace ID (16 bytes, global) +-- Span ID (8B) +-- Trace Flags (01 = Sampled)
+-- Version (00)
Span vs Trace:
• Trace: Poore distributed transaction ka Directed Acyclic Graph (DAG) jo shared
• Span: Ek single service ya function ke andar execution ka timed contiguous block (Start time, End time, Span ID, Parent Span ID, Attributes aur Events).
• Trace: Poore distributed transaction ka Directed Acyclic Graph (DAG) jo shared
Trace ID se identify hota hai.• Span: Ek single service ya function ke andar execution ka timed contiguous block (Start time, End time, Span ID, Parent Span ID, Attributes aur Events).
Live Distributed Trace Propagation & Latency Flame Graph Simulator
[OTel Collector Engine]: Endpoint: otel-collector:4317 (gRPC) | Exporter: Jaeger UI. Ready to capture spans...
5️⃣ The OpenTelemetry (OTel) Standard & Pipeline
Vendor lock-in prevent karne ke liye CNCF ka open-source instrumentation framework:
[Application Process (App Code + OTel SDK)]
| (OTLP / gRPC or HTTP)
v
[OpenTelemetry Collector Daemon / Sidecar]
+----------------------------------------------------+
| 1. Receivers : otlp, prometheus, filelog |
| 2. Processors : memory_limiter, batch, tail_sample |
| 3. Exporters : otlphttp, prometheus, jaeger, loki |
+----------------------------------------------------+
|
+-------------+-------------+
v v
[Prometheus / Mimir] [Jaeger / Tempo]
Sampling Strategies:
• Head Sampling: Request start hone par hi probabilistically decide karna ki trace capture karna hai ya nahi (e.g. 10% sampling rate). High throughput par low overhead.
• Tail Sampling: Collector level par poori trace complete hone ke baad decision lena (e.g. 100% of errors capture karna aur sirf 1% of normal 200 OK traces capture karna).
• Head Sampling: Request start hone par hi probabilistically decide karna ki trace capture karna hai ya nahi (e.g. 10% sampling rate). High throughput par low overhead.
• Tail Sampling: Collector level par poori trace complete hone ke baad decision lena (e.g. 100% of errors capture karna aur sirf 1% of normal 200 OK traces capture karna).
6️⃣ High-Cardinality Bottlenecks & Structured Logging
| Anti-Pattern (High Cardinality) | Why It Crashes TSDBs | Recommended Production Practice |
|---|---|---|
Metric label me user_id ya order_id daalna:
http_requests_total{user_id="998124"} |
Millions of unique time-series create karta hai, jisse memory explode ho jaati hai. | Unique IDs ko Structured JSON Logs ya Trace Span Attributes me store karein, TSDB metrics me nahi! |
Unbounded dynamic URL paths:
http_requests_total{path="/jobs/401/apply"} |
Har dynamic job ID nayi series generate karega. | Route template use karein: path="/jobs/:id/apply". |
7️⃣ Alerting Strategy: SLI, SLO & Error Budgets
The SRE Reliability Hierarchy:
• SLI (Service Level Indicator): Quantitative real-time measurement (e.g. Good HTTP requests / Total HTTP requests).
• SLO (Service Level Objective): Reliability target agreed upon with stakeholders (e.g. 99.9% of requests respond in < 200ms over 30 days).
• Error Budget: Remaining allowed imperfection ($100\% - 99.9\% = 0.1\%$ allowed failures). Agar error budget deplete ho jaye, toh naye features deploy karna freeze karke reliability fixes par focus kiya jata hai.
• Burn Rate Alerts: Sudden fast degradation hone par alert trigger karna (e.g. 1 ghante me 14.4x burn rate par PagerDuty on-call engineer ko page karna).
• SLI (Service Level Indicator): Quantitative real-time measurement (e.g. Good HTTP requests / Total HTTP requests).
• SLO (Service Level Objective): Reliability target agreed upon with stakeholders (e.g. 99.9% of requests respond in < 200ms over 30 days).
• Error Budget: Remaining allowed imperfection ($100\% - 99.9\% = 0.1\%$ allowed failures). Agar error budget deplete ho jaye, toh naye features deploy karna freeze karke reliability fixes par focus kiya jata hai.
• Burn Rate Alerts: Sudden fast degradation hone par alert trigger karna (e.g. 1 ghante me 14.4x burn rate par PagerDuty on-call engineer ko page karna).
