Server API¶
sonda-server is the HTTP control plane for Sonda. It is a long-running process. You send scenarios to it over REST, then inspect or stop them. Use it when you want Sonda as a service instead of a one-shot CLI command. Common cases: integrating Sonda into CI pipelines, test harnesses, or dashboards, or keeping a synthetic-telemetry baseline alive for hours or days.
The sonda-server binary is installed next to the sonda CLI. The install script and release tarballs place both on your PATH. The Docker image runs sonda-server as its default entrypoint.
This page covers installing, configuring, and operating the server. For the request and response shapes of every endpoint, see HTTP API reference.
Networking¶
You write a scenario with url: http://localhost:8428. It works from your laptop. You send it to a containerized sonda-server and nothing arrives at your backend. That is the most common surprise when moving a YAML from one place to another. This section exists to prevent it.
The rule that matters: Sonda resolves sink URLs in the process that runs the scenario, not the process that submits it. Inside a container, localhost means the container's loopback. Outside a container, it means your host. The next section covers every realistic combination: host, Compose, Kubernetes, and external. It also covers the environment variable pattern that lets one YAML work from all of them.
Two invocation paths¶
Every sink URL is resolved inside the process that is about to write to it. Sonda has two invocation paths. They resolve localhost differently.
You run sonda run file.yaml on your laptop or a bare host. The scenario runs in the shell process on your host. http://localhost:8428 resolves to your host's loopback. It reaches whatever listens on port 8428 there. The usual target is a Compose-published port or a native install.
You send a scenario body to sonda-server running inside a container. The scenario is validated and runs inside that container's network namespace. http://localhost:8428 resolves to the container's own loopback. Nothing is there, so the request fails.
The host-side curl talks to the host's loopback (hitting the published 8080:8080 port). The scenario it carries runs one level deeper, inside the server container.
The localhost trap
A scenario with url: http://localhost:8428 works from the host CLI and fails silently when sent to a containerized sonda-server. Inside the container, localhost is the container, not your host. The POST returns 201, the sink times out, no data arrives.
Two fixes are available. Write the URL with ${VAR:-default} so one file works from both paths. Or hardcode the in-network address: a Compose service name like http://victoriametrics:8428, or the Kubernetes Service DNS http://vmsingle.monitoring.svc.cluster.local:8428.
Endpoint resolution reference¶
Pick the row that matches where sonda runs and where your backend lives.
| Process runs here | Backend runs here | Correct url: |
Why |
|---|---|---|---|
| Host CLI | Backend on host (native install) | http://localhost:<port> |
Host loopback reaches the native listener. |
| Host CLI | Backend in Compose (port-published) | http://localhost:<published-port> |
The Compose-published port is exposed on the host. |
sonda-server in Compose |
Backend in same Compose network | http://<service-name>:<port> |
Compose provides a DNS entry per service. Examples: victoriametrics, loki, kafka. |
sonda-server in Compose |
Backend on host (Docker Desktop) | http://host.docker.internal:<port> |
Docker Desktop publishes a virtual DNS name that routes back to the host. |
sonda-server in Kubernetes (same namespace) |
Service in same namespace | http://<svc>:<port> |
Kubernetes DNS resolves short names within a namespace. |
sonda-server in Kubernetes (cross-namespace) |
Service in another namespace | http://<svc>.<ns>.svc.cluster.local:<port> |
The FQDN is required for cross-namespace resolution. |
sonda-server anywhere |
External backend (SaaS, cloud) | https://<public-dns>:<port> |
Fully qualified external DNS plus TLS. |
Note
On Linux without Docker Desktop, host.docker.internal does not resolve by default. Add --add-host=host.docker.internal:host-gateway to the sonda-server container, or use the host's LAN IP.
One file, both paths: ${VAR:-default}¶
The recommended fix is ${VAR:-default} interpolation inside the YAML. The same file then runs from your host CLI on the defaults and from a containerized sonda-server on the overrides. No edit, no sed, no second copy.
sink:
type: http_push
url: "${VICTORIAMETRICS_URL:-http://localhost:8428/api/v1/import/prometheus}"
The bundled examples/docker-compose-victoriametrics.yml sets the in-network overrides on the sonda-server container. Every scenario under examples/ works untouched in both places. See the full reference for the syntax and the seven built-in variable names every example honours.
Rewriting URLs before posting¶
If a YAML file hardcodes http://localhost:<port> and you prefer not to add interpolation, rewrite the URL before posting it:
sed 's|http://localhost:8428|http://victoriametrics:8428|g; \
s|http://localhost:3100|http://loki:3100|g; \
s|http://localhost:9094|http://kafka:9092|g' \
examples/http-push-retry.yaml \
| curl -X POST -H "Content-Type: text/yaml" \
--data-binary @- \
http://localhost:8080/scenarios
The substitutions cover the Compose backends bundled with Sonda:
| Backend | Host CLI URL (published port) | Compose URL (service name) |
|---|---|---|
| VictoriaMetrics | http://localhost:8428 |
http://victoriametrics:8428 |
| Loki | http://localhost:3100 |
http://loki:3100 |
| Kafka | localhost:9094 (external listener) |
kafka:9092 (internal listener) |
Service names come from examples/docker-compose-victoriametrics.yml. Match the sed substitutions to your service names if you change the compose file.
Diff before you POST
Networking examples¶
sonda runs on your host. The Compose stack exposes VictoriaMetrics on localhost:8428. The scenario's url: uses host loopback.
Both services run in the same Compose network. The sink URL uses the VictoriaMetrics service name -- not localhost.
Starting the server¶
Start the server with the installed sonda-server binary. It listens on port 8080 by default:
Your first request loop¶
With the server running, you can drive its full lifecycle from curl in three steps. Start the server, send a scenario, then confirm it is running:
# 1. Confirm the server is up
curl http://localhost:8080/health
# {"status":"ok"}
# 2. POST a scenario — the server validates it and starts emitting
curl -X POST -H "Content-Type: text/yaml" \
--data-binary @- http://localhost:8080/scenarios <<'EOF'
version: 2
kind: runnable
defaults:
rate: 10
duration: 60s
encoder:
type: prometheus_text
sink:
type: stdout
scenarios:
- id: up
signal_type: metrics
name: up
generator:
type: constant
value: 1.0
EOF
# {"id":"a1b2c3d4-...","name":"up","state":"running"}
# 3. List running scenarios — your scenario appears with its live state
curl http://localhost:8080/scenarios
The scenario runs in the background inside the server until its duration expires, or until you stop it with DELETE /scenarios/{id}. See HTTP API reference for the full endpoint list, request bodies, and response shapes.
Server flags¶
sonda-server accepts ten flags. The first five are the network, addressing, and startup surface; the remaining five control resource limits and runtime sizing. Use the RUST_LOG environment variable to control log verbosity (default info):
| Flag | Env var | Default | Purpose |
|---|---|---|---|
--port <PORT> |
-- | 8080 |
Port the server listens on |
--bind <ADDR> |
-- | 0.0.0.0 |
Bind address |
--api-key <KEY> |
SONDA_API_KEY |
(unset) | Bearer token for /scenarios/*, /events, and metrics endpoints. See Authentication. |
--catalog <DIR> |
SONDA_CATALOG |
(unset) | Directory of scenario and pack YAML files. Lets POST /scenarios resolve pack: <name> references. See Pack references over HTTP. |
--autostart |
SONDA_AUTOSTART |
false |
Start every kind: runnable entry in --catalog when the server boots. See Autostarting a catalog. |
--workers <N> |
-- | min(available_parallelism(), 16) |
Tokio worker thread count. See Tuning resource limits. |
--max-scenarios <N> |
-- | 0 (unlimited) |
Maximum concurrent scenario rows. POSTs beyond the cap return 429 capacity_exceeded. |
--max-inflight-requests <N> |
-- | 4 * workers |
Maximum concurrent in-flight control-plane HTTP requests. |
--request-timeout <SECS> |
-- | 30 |
Per-request timeout on control-plane routes. Returns 408 request_timeout on expiry. |
--max-body-bytes <N> |
-- | 1048576 (1 MB) |
Maximum request body size on control-plane routes. Returns 413 payload_too_large when exceeded. |
When you pass --catalog, point it at a directory that holds your kind: composable pack YAML files. The path must exist. A missing directory makes the server fail at startup with a clear error.
Press Ctrl+C for graceful shutdown. The server signals all running scenarios to stop before exiting.
Autostarting a catalog¶
By default the server boots empty and waits for you to POST /scenarios. Add --autostart and it starts every kind: runnable file in --catalog itself — the same result as POSTing each of those files yourself, without the POST:
The sweep runs alongside the server, not ahead of it. /health answers as soon as the port is open, and entries appear in GET /scenarios as they start — a large catalog reads as a growing list rather than an empty one. GET /ready is the endpoint that tells you when that list is complete: it returns 503 while the sweep is starting entries and 200 once it is done. It has already flipped to 200 by the time the INFO summary below is logged, so waiting on that line and waiting on /ready never disagree. A shutdown signal that arrives mid-sweep stops the sweep at the next entry, and every scenario it did start is stopped with the rest.
--autostart requires --catalog. Passing it alone (or setting SONDA_AUTOSTART without SONDA_CATALOG) exits before the server binds a port:
Error: --autostart requires --catalog <DIR> (or SONDA_AUTOSTART requires SONDA_CATALOG): there is no scenario catalog to start from
SONDA_AUTOSTART reads like every other boolean env var: empty, 0, n, no, f, false, and off (any case) disable it; any other value enables it. An empty value is the common one — env: [{name: SONDA_AUTOSTART, value: ""}] in a pod spec, or SONDA_AUTOSTART= in a Compose .env — and it means "off", not "broken".
What starts and what does not:
| Catalog file | Autostart behaviour |
|---|---|
kind: runnable |
Started. A file with several entries under scenarios: starts one scenario per entry. |
kind: composable |
Not started. Packs exist to be referenced by pack: <name>, and runnables that reference them resolve against the same catalog directory. |
No kind: header |
Ignored — the file is not a catalog entry at all. |
Entries start in catalog-name order — the scenario_name: of the file, or its name:, or the filename stem with underscores turned into dashes (web_traffic.yaml becomes web-traffic), whichever the file declares first. That is the same order sonda list shows, so what you see in the catalog listing is the order the sweep uses.
Two things stop the server before it binds, both of them mistakes only you can fix: --autostart with no catalog, and a catalog whose entries are not addressable — two files claiming the same name, or a name that cannot be derived from the file. The error names the clash and the files that caused it:
Error: --autostart: catalog /catalog is not startable
Caused by:
catalog /catalog contains duplicate entry name "web_traffic": /catalog/a.yaml and /catalog/b.yaml
Everything else keeps the server up. A file the server cannot use — one it has no permission to open, a v1 body, a compile error, a validation failure, or a file that would push past --max-scenarios — is logged and skipped; one bad file never stops the ones after it. An entry that panics on its way in is skipped the same way, with a panicked while starting WARN naming the file:
warning: catalog: skipping unreadable file /catalog/locked.yaml: Permission denied (os error 13)
WARN sonda_server::autostart: /catalog/legacy.yaml: does not compile, skipping catalog entry origin=/catalog/legacy.yaml reason=body is not a v2 scenario. ...
WARN sonda_server::routes::scenarios: /catalog/extra.yaml: scenario cap reached origin=/catalog/extra.yaml needed=1
The first line has a different shape because it comes from the catalog scan rather than the sweep: unreadable and unparseable files are dropped while the catalog is being read, and sonda list reports them the same way.
If the catalog directory cannot be scanned — its permissions do not allow a listing, or an entry vanishes mid-scan — the server logs a WARN, starts nothing, and keeps serving so you can fix it over the API:
WARN sonda_server::autostart: autostart: catalog could not be read, starting nothing catalog=/catalog reason=failed to read catalog dir /catalog: Permission denied (os error 13)
Every started scenario gets one INFO line naming the file it came from, and the sweep closes with a count, so "pod up, serving nothing" is visible in the log rather than only in the API:
INFO sonda_server::routes::scenarios: scenario launched id=6f1c… name=cpu_usage state=running origin=/catalog/cpu.yaml
INFO sonda_server::autostart: autostart: started 3 of 4 runnable catalog entries started=3 total=4
Autostarted scenarios are ordinary scenarios: they appear in GET /scenarios, count against --max-scenarios, and are stoppable with DELETE /scenarios/{id}. A while: clause that points at a scenario in another catalog file resolves when that file's turn comes; until then the scenario sits in unresolved or pending, exactly as it would if you posted the two files in catalog-name order.
Tuning resource limits¶
The five resource-limit flags shape what the server accepts, how many scenarios it holds, and how it spends CPU. Their defaults work for a developer laptop. Production deployments should pick values per the guidance below.
--workers <N>¶
The size of the tokio multi-thread runtime. Bound: N >= 1. The default is min(std::thread::available_parallelism(), 16). On Linux, available_parallelism() queries sched_getaffinity, so the count reflects the cpuset the process is allowed to run on — which is the full set of node CPUs by default in Kubernetes. CFS CPU quotas (resources.limits.cpu) do not reduce this count; if you want the worker pool to match a quota-restricted budget, set --workers explicitly. The 16 ceiling ensures that on a 64-core host sonda-server does not spawn 64 worker threads — the scheduler benefit drops off well before that and the extra threads add context-switch overhead. Override only when you want to deviate from the default (for example, --workers 32 on a thoroughly benchmarked host that genuinely uses the parallelism, or --workers 1 to pin a workload during benchmarking).
The configured value is exported on sonda_server_worker_threads — use it to confirm the runtime sees the value you expect.
--max-scenarios <N>¶
A row cap, not a CPU-task cap. Every entry in the server's scenario map consumes one slot, including scenarios in pending, running, paused, held, unresolved, and finished state. Reaching the cap returns 429 capacity_exceeded with a by_state breakdown of where the slots went.
Pick N from the memory you are willing to dedicate. Each scenario row holds runtime state plus a per-scenario stats buffer; an order-of-magnitude estimate is ~1 MB per scenario at moderate rates. Production deployments commonly use values in the 100 to 1000 range; CI runners often run with --max-scenarios 50 so a runaway test cannot OOM the runner.
0 disables the cap entirely. The server emits a WARN log line at startup so the choice is visible:
The cap is exported on sonda_server_max_scenarios. Live row usage is in sonda_server_active_scenarios plus sonda_server_scenarios_finished_total.
Finished scenarios still occupy a slot
A scenario in finished state holds its slot until DELETE /scenarios/{id} removes it. Automation that forgets to delete completed scenarios will fill the cap and start serving 429s on every new POST. Either DELETE after a scenario completes, or set --max-scenarios high enough to absorb the backlog you expect between cleanups.
--max-inflight-requests <N>¶
A global concurrency cap across the control-plane sub-router — POST /scenarios, DELETE /scenarios/{id}, POST /events, and the list/inspect routes. The default 4 * workers keeps a healthy queue depth without overcommitting. Raise it when the server has CPU and memory headroom and you see request latency climbing under load. Lower it when control-plane work is starving scenario runners.
The cap does not apply to the observability sub-router. /metrics, /scenarios/metrics, /scenarios/{id}/metrics, and /scenarios/{id}/stats stay scrape-able under saturation — exactly when your alerts need them.
Back-pressure is implicit. tower::limit::ConcurrencyLimitLayer does not return a 503; it back-pressures via the tokio runtime, and requests queue at the listener until a slot frees. The visible symptom is rising P99 on sonda_server_request_duration_seconds, not an error spike. Alert on the histogram, not on a status code.
--request-timeout <SECS>¶
Per-request handler bound on control-plane routes. Requests that exceed the limit return 408 request_timeout. The default 30 covers the slowest legitimate POST (a large multi-scenario body with cross-POST while: resolution). Raise it when you regularly post bodies with many entries; lower it when you want a tighter SLO on the control plane.
This is a handler-execution timeout, not a TCP read timeout — slow clients shipping a body bit-by-bit are not the trigger. The flag does not apply to the observability sub-router.
--max-body-bytes <N>¶
The maximum request body size on control-plane routes. Bodies above the limit return 413 payload_too_large. The default 1 MB fits any realistic scenario YAML and most multi-scenario batches. Raise it when you post very large catalogs by-value (rare — prefer --catalog <DIR> for that). Lower it as a hardening measure on internet-facing deployments.
The flag does not apply to the observability sub-router.
Health and readiness¶
Two public endpoints answer two different questions. /health says the process is alive; /ready says it is doing what it was configured to do.
| Endpoint | Question it answers | Use it for |
|---|---|---|
GET /health |
Is the process up and serving HTTP? | Liveness probes, load-balancer checks |
GET /ready |
Has the --autostart sweep finished starting the catalog? |
Readiness probes, deploy gates, scripts that wait before asserting on data |
/health is static. It returns {"status":"ok"} with 200 as soon as the port is open and never reads server state — which is exactly what you want from a liveness probe, since a restart is the only thing a failing liveness check can do.
/ready reports the state of the startup sweep and the two counts behind it:
# While the sweep is starting entries
curl -i http://localhost:8080/ready
# HTTP/1.1 503 Service Unavailable
# {"autostart_expected":120,"autostart_started":38,"status":"in_progress"}
# Once it has finished
curl -i http://localhost:8080/ready
# HTTP/1.1 200 OK
# {"autostart_expected":120,"autostart_started":120,"status":"finished"}
status |
HTTP | Meaning |
|---|---|---|
not_configured |
200 | The server was started without --autostart, so there is nothing to wait for. |
in_progress |
503 | The sweep is still starting catalog entries. |
finished |
200 | The sweep has been through every entry. |
failed |
503 | The sweep died before it finished. An ERROR line names the cause; the scenarios it had not reached will not start. |
A server without --autostart is ready from its first request, so putting a readiness probe on /ready is safe whether or not you autostart anything.
failed is terminal — the process does not recover on its own
A single entry does not get you here: an unreadable file, one that does not compile, one the server rejects, and one that panics on its way in are all skipped, and the sweep carries on to the next entry. failed means the sweep itself died, which leaves every entry it had not reached unstarted. Nothing retries a sweep that died. The status stays failed for the life of the process, so /ready answers 503 forever and a pod behind a readiness probe stays out of the Service indefinitely. It is not restarted either: the process is still alive and serving, so the liveness probe on /health keeps passing by design — a restart would only repeat the sweep that just failed, and the API keeps answering for whatever did start. Treat a failed sweep as a page: read the ERROR line, then restart the process (kubectl delete pod, docker restart) once you know why it died. sonda_server_autostart_started sitting below sonda_server_autostart_expected is the alertable signal.
A sweep that skipped files is still ready
status: finished with autostart_started below autostart_expected means the sweep ran to the end and skipped something — a file that does not compile, one that would have pushed past --max-scenarios, or one that panicked on its way in. That is ready. One bad file in a catalog of twelve must not pull the whole server out of service; the skip is logged with the file that caused it, and the difference between the two counts is what you alert on. Both counts are exported as sonda_server_autostart_started and sonda_server_autostart_expected.
This holds even when every entry is skipped. The server reports ready with nothing running, and a Kubernetes rollout of that catalog succeeds. Readiness answers "did the sweep run", not "was the catalog any good". The counts and the WARN line for each skipped entry answer the second question. Alert on those; do not expect a probe to catch a bad catalog. A catalog that yields no runnable entries at all is the one case this comparison cannot show — see sonda_server_autostart_expected.
Both endpoints stay public when API key authentication is enabled, so probes work without credentials.
Authentication¶
You can protect scenario endpoints with API key authentication. When enabled, all /scenarios/* requests, GET /metrics, GET /scenarios/metrics, and POST /events must include a bearer token. The /health and /ready endpoints are always public, so health and readiness probes work without credentials.
Enabling authentication¶
Pass an API key with the --api-key flag or the SONDA_API_KEY environment variable:
When the server starts with a key configured, you will see:
INFO sonda_server: API key authentication enabled for /scenarios/*, /events, /metrics, and /scenarios/metrics endpoints
No key = no auth
When neither --api-key nor SONDA_API_KEY is set, the server runs without authentication. All endpoints are public. This preserves backwards compatibility with existing deployments.
Protected vs. public endpoints¶
| Endpoint | Auth required |
|---|---|
GET /health |
No -- always public |
GET /ready |
No -- always public |
POST /scenarios |
Yes |
GET /scenarios |
Yes |
GET /scenarios/{id} |
Yes |
DELETE /scenarios/{id} |
Yes |
GET /scenarios/{id}/stats |
Yes |
GET /scenarios/{id}/metrics |
Yes |
GET /scenarios/metrics |
Yes |
GET /metrics |
Yes |
POST /events |
Yes |
See Authentication conventions on HTTP API reference for request shapes, header format, and 401 response bodies.
Prometheus scraping with auth
If you enable authentication, your Prometheus scrape config must include the bearer token for /metrics, /scenarios/metrics, and /scenarios/{id}/metrics. Add a bearer_token or bearer_token_file field to your scrape_configs entry. See Authentication on Server metrics for the scrape-config shape.
Kubernetes Secrets
In Kubernetes deployments, store the API key in a Secret and reference it as an environment variable in your Deployment spec:
apiVersion: v1
kind: Secret
metadata:
name: sonda-api-key
type: Opaque
stringData:
api-key: my-secret-key
env:
- name: SONDA_API_KEY
valueFrom:
secretKeyRef:
name: sonda-api-key
key: api-key
Apply the Secret before deploying:
See API key authentication in the Kubernetes deployment guide for the full Helm chart setup.
Where to next¶
- HTTP API reference — every endpoint, request body, and response shape.
- Server metrics — the twelve
/metricsseries and the alerts that matter. - Docker — Compose stacks and host-side
docker runexamples. - Kubernetes — Helm chart, Service DNS, cross-namespace access.
- Sinks — every sink type and its
url:field. - Troubleshooting — diagnostics for connection-refused and empty-backend symptoms.