OWASP Nest
Monitoring and Observability
Bringing metrics-first observability to OWASP Nest. I instrumented the Django backend, the frontend and the background workers with OpenTelemetry, stored everything in VictoriaMetrics, built Grafana dashboards on top, and wrote the Terraform to run the stack on AWS.
- 6
- Feature PRs merged
- 9
- Issues scoped
- 16
- Terraform unit tests
- 5 years
- Metric history
Demo video
Demo video coming soon
A full walkthrough of OWASP Nest will be added here.
The local stack, the Grafana dashboards and the AWS module.
Overview
OWASP Nest is the open-source platform that helps people discover OWASP projects and get involved in them. It runs a Django backend, a web frontend, background workers and AI features on AWS, and until this work it had no metrics pipeline of its own.
The Monitoring and Observability milestone, tracked as a GSoC 2026 milestone on the project, sets out to change that. I scoped the milestone into nine issues and have been delivering them one pull request at a time on the feature/o11y branch, starting with a local stack every contributor can run and ending with the same stack provisioned on AWS.
The goal is simple to state: complete visibility into service health, and a shorter path from “something is wrong” to “here is why”.
The Problem
Before this milestone, the answers to basic operational questions lived nowhere.
- The backend exported no request rate, error rate or latency
- The frontend emitted no metrics at all
- Background jobs ran with no visibility into runs or duration
- There was no dashboard to look at
- Project stars, forks and contributors had no time series
- Nothing in the AWS infrastructure could store or visualise metrics
Without that data, issues are found by users rather than by the team, and there is no baseline to compare a bad day against.
Goals of the Milestone
| Goal | How it is met |
|---|---|
| Visibility into every service | OpenTelemetry metrics from the backend, the frontend and the RQ workers. |
| One place to store metrics | A single-node VictoriaMetrics instance, locally and on AWS. |
| Dashboards anyone can reproduce | Grafana with the datasource and dashboards provisioned from version-controlled files. |
| No cost to people who do not need it | The whole stack is opt-in: a separate compose file locally and a feature flag in Terraform. |
| Never break the app | Instrumentation is off by default, guarded, and cannot block startup. |
| Production parity | The same VictoriaMetrics image is used locally and in the Terraform module. |
System Architecture
Every service pushes metrics over OTLP/HTTP straight to VictoriaMetrics, which accepts OpenTelemetry natively, so there is no collector to run and maintain. Grafana reads from VictoriaMetrics through a provisioned datasource.
| Layer | Technology | Purpose |
|---|---|---|
| Instrumentation | OpenTelemetry SDKs | Vendor-neutral metrics from Python and Node |
| Transport | OTLP over HTTP | Push metrics without a collector |
| Storage | VictoriaMetrics | Time-series storage and PromQL-compatible queries |
| Visualisation | Grafana | RED and runtime dashboards as code |
| Local orchestration | Docker Compose | An opt-in overlay with isolated volumes |
| Infrastructure | Terraform on AWS | ECS Fargate, EFS, KMS, ALB and Secrets Manager |
What I Built
Each step is a separate issue and pull request, small enough to review properly and useful on its own.
1. VictoriaMetrics, locally
A single-node VictoriaMetrics service added to Docker Compose as an opt-in overlay, with a Makefile target so only contributors who want metrics run it. Its volumes were later isolated from the main stack so the two cannot interfere.
2. Backend instrumentation
OpenTelemetry in the Django backend, exporting http.server.duration and http.server.active_requests over OTLP/HTTP.
- Manual setup from
AppConfig.ready() - Gated behind
DJANGO_OTEL_METRICS_ENABLED, off by default - Wrapped in error handling with an idempotency guard
3. Frontend instrumentation
Process and system CPU, memory and network metrics from the frontend runtime through OpenTelemetry host metrics, pushed to the same store.
4. Grafana and the RED dashboard
A Grafana service with VictoriaMetrics provisioned as its datasource, and a dashboard saved as version-controlled JSON that loads on startup.
- Rate: requests per second
- Errors: error rate
- Duration: p95 latency
- Process memory and CPU for the frontend
5. Custom metrics for background jobs
A reusable track_job decorator that records nest.job.runs and nest.job.duration, applied first to the AI reply job on the ai queue.
6. Project stats with history
An otel_export_project_metrics management command that exports stars, forks and contributors for every project. --all backfills the full daily history and the default mode exports the latest day.
Running It on AWS
The second half of the milestone moves the stack from a laptop to the project's AWS infrastructure, as a new Terraform observability module. This part is in review.
Storage that survives restarts
VictoriaMetrics runs on ECS Fargate with an EFS-backed, KMS-encrypted /data volume, so metrics outlive any single task.
Least privilege by default
The task runs as a non-root user through an EFS access point, on on-demand Fargate, as a single task.
Safe to merge
The module is registered behind enable_observability, so terraform plan shows no changes until it is turned on.
One image, two environments
The VictoriaMetrics image comes from one Dockerfile, so Dependabot tracks it and local and production stay in sync.
Tested infrastructure
A native terraform test suite with 16 mocked unit tests, plus an integration test applied against LocalStack.
Grafana and wiring
Next in line: a Grafana service with a stable VictoriaMetrics endpoint, an ALB listener, the Grafana secret in Secrets Manager, and egress rules plus OTLP environment variables so app tasks can export.
Challenges and Solutions
The interesting problems were the ones where the obvious approach quietly did not work.
| Challenge | Solution |
|---|---|
The opentelemetry-instrument auto-wrapper crashed the backend | It loads settings too early for django-configurations. I switched to manual instrumentation from AppConfig.ready(). |
| Instrumentation must never take the app down | Metric setup is feature-flagged, wrapped in error handling and guarded for idempotency, so a failure cannot block startup. |
| Job metrics were recorded but never arrived | RQ's default worker forks per job and the exporter thread does not survive fork(). The worker uses rq.SimpleWorker locally so metrics are flushed. |
| Historical stats could not be backfilled through OpenTelemetry | OTel can only stamp samples with the current time, so the export command writes to VictoriaMetrics' import API with each snapshot's own date. |
| Backfilled samples disappeared | VictoriaMetrics drops anything older than its retention window. Local retention went from 7 days to 5 years. |
| The observability stack shared volumes with the main stack | Isolated the observability volumes so running one never affects the other. |
| Metrics lost whenever a Fargate task restarts | An EFS-backed, encrypted data volume mounted through an access point. |
Status and Roadmap
| Step | Status |
|---|---|
| Single-node VictoriaMetrics locally | Merged |
| Backend OpenTelemetry metrics | Merged |
| Frontend OpenTelemetry metrics | Merged |
| Grafana with datasource and RED dashboard | Merged |
| Custom metrics for RQ workers | Merged |
| Project GitHub stats with historical backfill | Merged |
| Terraform module for VictoriaMetrics on AWS | In review |
| Grafana on AWS with a stable endpoint | In review |
| Wire app tasks to export metrics | Planned |
The wider milestone also calls for centralised logging, real-time alerts and distributed tracing. Metrics came first on purpose, because the storage, dashboards and infrastructure they need are the foundation the other signals build on.
Best Practices
Small, reviewable changes
One issue and one pull request per capability, all landing on a long-lived feature branch that is synced with main regularly.
Opt-in everywhere
A separate compose overlay, an environment flag in the backend and a Terraform variable. Contributors and environments that do not need observability pay nothing for it.
Everything as code
Dashboards, datasources and infrastructure are version-controlled and provisioned automatically, so the setup is reproducible.
Tested like application code
Unit tests for the instrumentation, the job decorator and the backfill logic, plus Terraform unit and integration tests.
Conclusion
OWASP Nest went from having no metrics to a working pipeline: three instrumented services, a time-series store with years of project history, dashboards that provision themselves and an AWS module ready to carry it to production. Doing it in the open, on a project other people depend on, meant every step had to be safe by default, easy to review and honest about what it does not cover yet.