Kubernetes¶
Run sonda-server in Kubernetes when you want a continuous synthetic-telemetry baseline running inside the cluster. The service emits known metrics through your stack at all times. A flat Grafana panel then means "the data stopped," not "the scrape config broke." It is the difference between guessing whether a quiet dashboard is a real outage and knowing it in one look.
When you finish the install below, you have a sonda-server pod running in the cluster. It is reachable at a stable Service DNS name: sonda.<namespace>.svc.cluster.local:8080. It accepts scenarios sent over HTTP and exposes their metrics for Prometheus to scrape. The bundled Helm chart produces exactly that: a Deployment with health probes, a ClusterIP Service with a named http port, and optional scenario injection through a ConfigMap.
Looking for the full walkthrough?
The Synthetic Monitoring guide is the end-to-end worked example for this use case. It covers starting a local cluster, deploying the chart, submitting long-running scenarios, scraping them with Prometheus, and building Grafana dashboards. This page is the chart reference: every value, probe setting, and auth option the guide uses.
Prerequisites¶
You need a running Kubernetes cluster and these CLI tools installed:
kubectl-- configured to talk to your clusterhelm-- v3.x
If you don't have a cluster yet, the Synthetic Monitoring guide covers lightweight local options: kind, k3d, minikube, and OrbStack. It includes step-by-step setup instructions.
Install the chart¶
Wait for the pod to become ready:
You should see 1/1 Running within 15--20 seconds. The chart defaults to
ghcr.io/davidban77/sonda:<!--x-release-please-version-->1.30.3<!--x-release-please-end--> (the chart's appVersion). Pin a different version with
--set image.tag=<version>.
Deploy to a dedicated namespace
Keep Sonda isolated from your application workloads:
All kubectl commands on this page assume the default namespace. Add -n sonda if you
installed into a different one.
Chart values reference¶
The chart includes default values that work for most installs. Override any value with --set flags or a
-f values.yaml file.
Image¶
| Value | Default | Description |
|---|---|---|
image.repository |
ghcr.io/davidban77/sonda |
Container image registry and name |
image.tag |
"" (uses appVersion: 1.30.3) |
Image tag to pull |
image.pullPolicy |
IfNotPresent |
Kubernetes image pull policy |
imagePullSecrets |
[] |
Secrets for private registries |
Server¶
| Value | Default | Description |
|---|---|---|
server.port |
8080 |
Port sonda-server listens on inside the container |
server.bind |
0.0.0.0 |
Bind address |
server.autostart |
false |
Start every kind: runnable file in the mounted ConfigMap when the pod boots |
Service¶
| Value | Default | Description |
|---|---|---|
service.type |
ClusterIP |
Kubernetes Service type (ClusterIP, NodePort, LoadBalancer) |
service.port |
8080 |
Service port exposed to the cluster |
The Service exposes a named port called http. ServiceMonitor and Ingress
resources reference this name.
Resources¶
| Value | Default | Description |
|---|---|---|
resources.requests.cpu |
100m |
CPU request |
resources.requests.memory |
128Mi |
Memory request |
resources.limits.cpu |
500m |
CPU limit |
resources.limits.memory |
256Mi |
Memory limit |
These defaults are sized for light workloads: a handful of scenarios at moderate rates. If you run many concurrent scenarios or high event rates, increase the limits:
helm install sonda ./helm/sonda \
--set resources.requests.cpu=200m \
--set resources.limits.cpu=1000m \
--set resources.limits.memory=512Mi
Security¶
The container image runs as UID 65532 (the upstream "nonroot" convention used by distroless and Chainguard images). The chart ships matching podSecurityContext and securityContext defaults that satisfy the restricted Pod Security Standard and pass Kyverno or OPA Gatekeeper policies that enforce runAsNonRoot: true, allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, and capabilities.drop: [ALL] out of the box.
| Value | Default | Description |
|---|---|---|
podSecurityContext.runAsNonRoot |
true |
Reject the pod if any container would run as UID 0 |
podSecurityContext.runAsUser |
65532 |
UID the pod runs as |
podSecurityContext.runAsGroup |
65532 |
GID the pod runs as |
podSecurityContext.fsGroup |
65532 |
GID applied to mounted volumes |
podSecurityContext.seccompProfile.type |
RuntimeDefault |
Apply the container runtime's default seccomp profile |
securityContext.runAsNonRoot |
true |
Reject the container if it would run as UID 0 |
securityContext.runAsUser |
65532 |
UID the container process runs as |
securityContext.runAsGroup |
65532 |
GID the container process runs as |
securityContext.allowPrivilegeEscalation |
false |
Disallow setuid / file-capability escalation |
securityContext.readOnlyRootFilesystem |
true |
Mount / read-only; the server writes nothing outside /tmp and volume mounts |
securityContext.capabilities.drop |
[ALL] |
Drop every Linux capability |
Override individual fields if your cluster uses a different UID convention, or replace the whole block to relax a constraint. The example below pins the container to UID 1000 (a common convention for managed clusters that mount Persistent Volumes with a fixed owner):
podSecurityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
seccompProfile:
type: RuntimeDefault
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
Lists are replaced, not merged
Helm deep-merges maps but replaces lists wholesale. Scalar overrides like --set securityContext.runAsUser=1000 work key-by-key against the defaults above. Overriding capabilities.drop, however, replaces the entire [ALL] default — supply the full list you want (typically [ALL] plus any explicit exceptions) rather than the delta.
Authentication¶
| Value | Default | Description |
|---|---|---|
server.auth.enabled |
false |
Enable API key authentication on /scenarios/* endpoints |
server.auth.existingSecret |
"" |
Name of an existing Secret containing the API key |
server.auth.secretKey |
api-key |
Key within the Secret that holds the API key value |
When server.auth.enabled is true, the chart injects SONDA_API_KEY into the container
from the referenced Secret. See API key authentication for setup
instructions.
Scheduling¶
| Value | Default | Description |
|---|---|---|
replicaCount |
1 |
Number of Deployment replicas (ignored when HPA is enabled) |
nodeSelector |
{} |
Node selector constraints |
tolerations |
[] |
Pod tolerations |
affinity |
{} |
Pod affinity/anti-affinity rules |
Autoscaling (HPA)¶
| Value | Default | Description |
|---|---|---|
autoscaling.enabled |
false |
Enable HorizontalPodAutoscaler |
autoscaling.minReplicas |
1 |
Minimum replica count |
autoscaling.maxReplicas |
5 |
Maximum replica count |
autoscaling.targetCPUUtilizationPercentage |
80 |
Target CPU utilization |
autoscaling.targetMemoryUtilizationPercentage |
(unset) | Target memory utilization |
Pod Disruption Budget¶
| Value | Default | Description |
|---|---|---|
podDisruptionBudget.enabled |
false |
Enable PodDisruptionBudget |
podDisruptionBudget.minAvailable |
1 |
Minimum available pods during disruption |
podDisruptionBudget.maxUnavailable |
(unset) | Maximum unavailable pods during disruption |
Ingress¶
| Value | Default | Description |
|---|---|---|
ingress.enabled |
false |
Enable Ingress resource |
ingress.className |
"" |
Ingress class name |
ingress.annotations |
{} |
Ingress annotations |
ingress.hosts |
[{host: sonda.local, paths: [{path: /, pathType: Prefix}]}] |
Ingress host rules |
ingress.tls |
[] |
TLS configuration |
ServiceMonitor¶
A ServiceMonitor endpoint scrapes a single path. sonda-server exposes three Prometheus paths and they answer different questions — pick by the dashboard you are filling:
| Path | What it returns | When to scrape |
|---|---|---|
/metrics |
Server-process RED + saturation (sonda_server_* series). See Server metrics. |
Operational dashboards and alerts on the server itself. |
/scenarios/metrics |
Aggregate scenario data fused into one response. Supports ?label=k:v filtering. See Aggregate Prometheus scrape. |
A single scrape target covering every scenario, regardless of how it was launched. |
/scenarios/<scenario-id>/metrics |
One scenario's series. | Per-scenario debugging or a stable URL per scenario loaded from the scenarios ConfigMap. |
The chart defaults serviceMonitor.path to /metrics — the right choice for operational alerting on the server process. Switch it to /scenarios/metrics when you also want one scrape job to cover every scenario the server is running.
| Value | Default | Description |
|---|---|---|
serviceMonitor.enabled |
false |
Enable Prometheus Operator ServiceMonitor |
serviceMonitor.interval |
30s |
Scrape interval |
serviceMonitor.scrapeTimeout |
10s |
Scrape timeout |
serviceMonitor.path |
/metrics |
Metrics endpoint path. Switch to /scenarios/metrics for aggregate scenario scraping, or /scenarios/<scenario-id>/metrics for a single scenario. |
serviceMonitor.additionalLabels |
{} |
Extra labels on the ServiceMonitor resource |
One ServiceMonitor per path
A ServiceMonitor endpoint scrapes a single path. To scrape both /metrics and /scenarios/metrics, deploy two ServiceMonitor resources (or one with two endpoints entries — see the manual ServiceMonitor example below).
Scenarios (ConfigMap)¶
| Value | Default | Description |
|---|---|---|
scenarios |
{} |
Map of filename to YAML content, mounted at /scenarios |
See Configuring scenarios below.
Configuring scenarios¶
You can load scenarios into sonda-server two ways. Include them in the Helm release through a
ConfigMap, or submit them at runtime over the API.
ConfigMap (deploy-time)¶
Define scenarios under the scenarios key in a values file. Each key becomes a file mounted
at /scenarios inside the container:
scenarios:
cpu-metrics.yaml: |
version: 2
kind: runnable
defaults:
rate: 100
duration: 30s
encoder:
type: prometheus_text
sink:
type: stdout
scenarios:
- signal_type: metrics
name: cpu_usage
generator:
type: sine
amplitude: 50.0
period_secs: 60
offset: 50.0
The Deployment template includes a checksum/scenarios annotation. Changing scenario
content in your values file triggers an automatic pod rollout on helm upgrade.
You can place kind: composable metric pack YAMLs in the same scenarios map next to your runnable scenarios. When scenarios is populated, the chart points the server's --catalog flag at the mounted /scenarios directory. POST /scenarios bodies that reference a pack by name (pack: <name>) then resolve automatically. No extra configuration. See Pack references over HTTP for how that resolution works.
See Scenario Fields for the full YAML schema.
Starting the ConfigMap scenarios automatically¶
Mounting scenarios makes them available; it does not run them. Set server.autostart: true and the pod starts every kind: runnable file in the ConfigMap as soon as the server is listening — no POST, no sidecar re-posting on restart:
server.autostart: true with an empty scenarios map is rejected at template time, so the mistake surfaces on helm install rather than as a Ready pod serving nothing:
Error: execution error at (sonda/templates/deployment.yaml:2:4): server.autostart is set but scenarios is empty: there is no catalog to start. Populate .Values.scenarios or set server.autostart=false.
kind: composable packs are never started — they stay available for pack: <name> references. A scenario file the server cannot admit is logged and skipped, so one bad file cannot keep the pod from coming up; two files claiming the same name is a startup error, because the pod would otherwise come up serving nothing. See Autostarting a catalog for the full behaviour, including how skipped files are logged.
Autostarted scenarios run for the duration: in their YAML. Once they finish they stay in the scenario list in finished state until deleted — set a long duration: (or omit it for an unbounded run) when the pod is meant to emit continuously.
API (runtime)¶
Once sonda-server is running, you can submit scenarios at runtime without redeploying:
curl -X POST -H "Content-Type: text/yaml" \
--data-binary @examples/basic-metrics.yaml \
http://localhost:8080/scenarios
This is useful for long-running synthetic monitoring where you rotate scenarios over time. See Server API for the full endpoint reference and the Synthetic Monitoring guide for operational patterns.
Health probes¶
The Deployment configures a liveness probe against GET /health and a readiness probe against GET /ready:
| Probe | Path | Initial delay | Period | Timeout | Failure threshold |
|---|---|---|---|---|---|
| Liveness | /health |
5s | 10s | 3s | 3 |
| Readiness | /ready |
2s | 5s | 3s | 3 |
The two answer different questions. /health returns {"status":"ok"} with HTTP 200 whenever the process is up, and a failing liveness probe restarts the pod. /ready returns 503 while the server.autostart sweep is still starting catalog entries, and 200 once it has finished. A pod only joins the Service endpoints once the scenarios it was deployed to run are actually running. A pod that is not ready is taken out of the Service without being restarted, which is what you want while a large catalog is still starting.
With server.autostart enabled, kubectl rollout status therefore completes when the catalog is running, not when the port opens. With server.autostart disabled — the default — /ready answers 200 from the first request, so the readiness probe behaves exactly as it did against /health.
A pinned image.tag older than /ready leaves the pod permanently NotReady
The chart's default image.tag follows its own appVersion, so a helm upgrade that takes this chart takes a matching image and there is nothing to do. If you pin image.tag to a release that does not serve /ready, the probe gets a 404 and the pod never becomes Ready. Move the pin forward, or stay on the chart version you were running.
A sweep that skipped files is still ready
A sweep that finished but skipped a file still reports ready, so the pod goes into service and the rollout succeeds. That is deliberate. One bad file in a twelve-file ConfigMap must not pull the whole pod out of service. On a node drain, the same rule stops every pod coming back NotReady at once. The counts on GET /ready and on sonda_server_autostart_started are what surface the skip. This holds when every file is broken too: the pod is Ready with nothing running. Alert on the counts rather than relying on the rollout to fail.
A failed sweep is terminal — the pod stays NotReady and is never restarted
A single bad file does not get you here — those are skipped, including one that panics on its way in. failed means the sweep itself died. /ready then reports status: failed and keeps answering 503 for the life of the process, so the pod never rejoins the Service. It is not restarted either: the process is alive, so the liveness probe on /health keeps passing by design. The pod needs a human. Read the ERROR line in the log, then kubectl delete pod to start a fresh sweep. Alert on it with sonda_server_autostart_started, since a pod that is permanently NotReady is otherwise quiet.
Accessing the server¶
Port-forward¶
The quickest way to reach sonda-server from your workstation:
Then interact with the API at http://localhost:8080:
# Health check
curl http://localhost:8080/health
# Readiness check
curl http://localhost:8080/ready
# Start a scenario
curl -X POST -H "Content-Type: text/yaml" \
--data-binary @examples/basic-metrics.yaml \
http://localhost:8080/scenarios
# List running scenarios
curl http://localhost:8080/scenarios
In-cluster DNS¶
Other pods in the cluster can reach sonda-server using the Service DNS name:
For example, a Prometheus instance in the same namespace can scrape
http://sonda:8080/scenarios/<id>/metrics directly.
Prometheus scraping¶
sonda-server exposes three scrape endpoints. Each answers a different question:
| Endpoint | What it returns | When to scrape |
|---|---|---|
GET /scenarios/{id}/metrics |
One scenario's series | Per-scenario debugging |
GET /scenarios/metrics |
Aggregate scenario data | Per-process scenario view |
GET /metrics |
Server-process RED + saturation | Operational dashboards and alerts |
GET /scenarios/metrics and GET /scenarios/{id}/metrics are idempotent snapshots — one sample per (name, labels) series with no timestamp, like a node_exporter scrape. GET /metrics returns the server's own RED and saturation telemetry — see Server metrics for the twelve series and the alerts that go with them. Most Prometheus setups want a job per endpoint; the aggregate /scenarios/metrics covers every scenario and you do not need to know scenario IDs in advance. See Aggregate Prometheus scrape for the ?label=k:v AND-filter syntax.
Aggregate scrape config¶
scrape_configs:
- job_name: sonda
scrape_interval: 15s
metrics_path: /scenarios/metrics
static_configs:
- targets: ["sonda.default.svc:8080"]
Add params: {label: ["device:srl1"]} to scope a job to one device's metrics. Repeat the label value to AND-combine selectors.
Per-scenario scrape config¶
When you want one scrape job per scenario, point metrics_path at the scenario ID:
scrape_configs:
- job_name: sonda-id
scrape_interval: 15s
metrics_path: /scenarios/<SCENARIO_ID>/metrics
static_configs:
- targets: ["sonda.default.svc:8080"]
Replace <SCENARIO_ID> with the UUID returned by POST /scenarios.
ServiceMonitor¶
If you run the Prometheus Operator (typically through kube-prometheus-stack), the chart includes an optional ServiceMonitor template. Enable it with:
See the ServiceMonitor values reference for all options
(interval, scrapeTimeout, path, additionalLabels).
Manual ServiceMonitor¶
You can also apply a custom ServiceMonitor manually for full control. The example below scrapes the server's own RED metrics and the aggregate scenario view in one resource:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: sonda
labels:
release: prometheus # must match your Prometheus Operator's selector
spec:
selector:
matchLabels:
app.kubernetes.io/name: sonda
endpoints:
- port: http
interval: 15s
path: /metrics
- port: http
interval: 15s
path: /scenarios/metrics
The port: http field matches the named port on the Sonda Service. Each endpoints entry scrapes a single path. For a per-scenario route, add one entry per scenario ID with path: /scenarios/<SCENARIO_ID>/metrics.
API key authentication¶
Sonda-server supports optional bearer token authentication on /scenarios/*, /scenarios/metrics, and
/events. When enabled, clients must include an Authorization: Bearer <key> header. The
/health and /ready endpoints stay public so liveness and readiness probes work without credentials.
For the full authentication behavior (error responses, protected vs. public endpoints), see the Server API Authentication section.
Create a Secret¶
Store your API key in a Kubernetes Secret:
apiVersion: v1
kind: Secret
metadata:
name: sonda-api-key
type: Opaque
stringData:
api-key: "your-secret-key-here"
Generate a random key
Enable auth in the Helm chart¶
Point the chart at your Secret:
helm install sonda ./helm/sonda \
--set server.auth.enabled=true \
--set server.auth.existingSecret=sonda-api-key
Or in a values file:
server:
auth:
enabled: true
existingSecret: sonda-api-key
secretKey: api-key # default; change if your Secret uses a different key
The chart sets SONDA_API_KEY in the container environment from the Secret. On startup you
will see:
INFO sonda_server: API key authentication enabled for /scenarios/*, /events, /metrics, and /scenarios/metrics endpoints
Authenticated API calls¶
Once auth is enabled, include the bearer token in all /scenarios/*, /scenarios/metrics, and /events requests:
# Port-forward to reach the server
kubectl port-forward svc/sonda 8080:8080
# Start a scenario (requires auth)
curl -X POST \
-H "Authorization: Bearer your-secret-key-here" \
-H "Content-Type: text/yaml" \
--data-binary @examples/basic-metrics.yaml \
http://localhost:8080/scenarios
# Health and readiness checks (always public)
curl http://localhost:8080/health
curl http://localhost:8080/ready
Prometheus scraping with auth¶
When authentication is enabled, every scrape endpoint requires a bearer token: /metrics, /scenarios/metrics, and /scenarios/{id}/metrics. Add the token to your Prometheus scrape config:
For a ServiceMonitor, add bearerTokenSecret to the endpoint:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: sonda
labels:
release: prometheus
spec:
selector:
matchLabels:
app.kubernetes.io/name: sonda
endpoints:
- port: http
interval: 15s
path: /metrics
bearerTokenSecret:
name: sonda-api-key
key: api-key
Add a second endpoints entry with the same bearerTokenSecret to also scrape /scenarios/metrics or a per-scenario path. See Authentication on Server metrics for the full reference.
Same Secret, same namespace
The bearerTokenSecret must reference a Secret in the same namespace as the
Prometheus instance, not the Sonda namespace. If they differ, copy the Secret or use
bearer_token_file with a mounted volume instead.
Upgrading¶
Update your release after changing values or pulling a new chart version:
# Upgrade with new values
helm upgrade sonda ./helm/sonda -f my-values.yaml
# Upgrade to a new image version
helm upgrade sonda ./helm/sonda --set image.tag=<!--x-release-please-version-->1.30.3<!--x-release-please-end-->
If your values file includes scenarios, the ConfigMap checksum annotation triggers an
automatic pod rollout. No manual restart is needed.
Uninstalling¶
This removes the Deployment, Service, ConfigMap (if created), and all associated resources.
Add -n <namespace> if you installed into a non-default namespace.
What's next¶
- Synthetic Monitoring guide -- deploy Sonda on Kubernetes, submit long-running scenarios, scrape with Prometheus, and build Grafana dashboards
- Server API -- full endpoint reference for
sonda-server - Server metrics -- the twelve
/metricsseries and the PromQL alerts that matter - Docker -- Docker image and Compose stacks for local development
- Scenario Fields -- full YAML schema for scenario configuration