From Alert to Answer: An OpenSRE Usefulness Workshop — Plan & Live Demo Script

Your browser can’t run the 3D presentation. Everything below still reads top-to-bottom as a normal article — print it to PDF for the handout version.

IntroAct 1Act 2Act 3Act 4Close

2-HOUR WORKSHOP · CHAOS-DRIVEN · YOUR CLUSTERS

One rolling incident, four chaos-driven acts, twelve integrations — all live, all on your own clusters.

Kubernetes · Nginx · NATS · YugabyteDB · Aerospike · Keycloak · Grafana · Tempo · Victoria Logs · Alertmanager · Jenkins · GitHub

Goal of this workshop: replace opinions about OpenSRE with a shared experience. One incident unfolds across four acts — page, blast radius, data plus identity, fix and close — while OpenSRE investigates on the projector and every fault is injected live from the chaos kit. By the end, every attendee has seen all twelve integrations earn their place.

Format: 10 minutes intro, four 25-minute acts, 10 minutes score and backlog. Kubernetes basics assumed. The strip at the top of every slide is the spine — glance at it any time you feel lost about where we are.

IntroAct 1Act 2Act 3Act 4Close

120 MINUTES · FOUR ACTS · ONE INCIDENT

Inject. Watch. Ask. Revert.

When	Act	Fault → question it answers
0:00	Intro · 10 min	Map, LLM check, safety contract
0:10	Act 1 · The page fires · 25 min	Pod kill + flood → k8s · alertmanager · grafana · victoria
0:35	Act 2 · Blast radius · 25 min	Edge 503 + NATS flood → nginx · tempo · nats
1:00	Act 3 · Data + identity · 25 min	Slow query + login storm → yugabyte · aerospike · keycloak
1:25	Act 4 · Fix, ship, close · 25 min	Red build → green → jenkins · github · verify
1:50	Close · 10 min	12-integration scorecard + backlog

Everyone leaves with: the 4-beat loop, a green 12-integration matrix, and the chaos kit on its branch.

Every act runs the same 4-beat loop: inject (1m), symptom (1m), OpenSRE triage (3m per integration), revert plus verify (1m). The revert command is shown before every fault runs — say it out loud, then run it.

Acts chain into one story: a dying pod becomes edge 503s, which uncover queue lag, which coincide with data pressure and an auth storm, which end in a shipped fix. Nobody needs to remember the timetable; the incident carries them.

INTROSET THE STAGE · 10 MINUTES

Transition beat. “Nine o’clock, Tuesday. Checkout is about to page, the edge is about to 503, and the queue is about to drown. You will watch all of it happen — and get solved — on this screen, with faults injected live.”

IntroAct 1Act 2Act 3Act 4Close

BEFORE ANYTHING BURNS · 10 MINUTES

Pre-flight: model, matrix, safety

# 1 · the model answers from this room:
export CUSTOM_OPENAI_BASE_URL=http://192.0.2.5:8000
uv run opensre health && uv run opensre ask "are you up?"

# 2 · all twelve integrations green:
for s in kubernetes grafana alertmanager victoria-logs tempo nginx nats \
         yugabytedb aerospike keycloak jenkins github; do
  uv run opensre integrations setup $s
done

SAFETY

Faults hit stateless prod echo-services only. Never two Aerospike nodes or two YB masters — quorum loss is not a demo.

CONTRACT

Every fault script prints its revert first, previews with DRY_RUN=1, refuses live runs without --confirm. Global abort: ./abort-all.sh --confirm.

Run the health check live — the “are you up” answer is the first time the room hears the model, and it proves the whole chain (terminal, config, LAN endpoint) before any incident depends on it. Step two fills the integration matrix the Close slide scores.

Read the safety card out loud, verbatim. Then point at the two exhibits already burning: Grafana CrashLoopBackOff and 164 cert-manager-cainjector restarts. They are free triage material for Act 1 and proof that this cluster is production, not a sandbox.

IntroAct 1Act 2Act 3Act 4Close

YOUR STACK, ONE PICTURE

Everything the demo touches

Evidence Sources Around The Kubernetes Core · Twelve Integrations

Change Edge Data Observe Incident Human

Walk the map box by box, naming the real system behind each one — and read the legend out loud once, because this color language now runs through the entire deck: violet is change, amber is the edge, teal is data, blue is observation, red is the incident itself. Blue here means the OpenSRE system and its lenses — never the world being investigated.

Count to twelve with the room: Kubernetes, GitHub, Alertmanager, Nginx, Grafana, Victoria Logs, VictoriaMetrics, YugabyteDB, Aerospike, NATS, Tempo, Keycloak, Jenkins — thirteen boxes, twelve integrations, one human. The bottom row (messaging, traces, identity, CI) joins on a shared evidence bus. Land the human-in-the-loop point: OpenSRE proposes with evidence, the engineer disposes.

IntroAct 1Act 2Act 3Act 4Close

WHAT HAPPENS BETWEEN ALERT AND ANSWER · 3 MINUTES

Six stages. Read-only tools. Evidence or it didn’t happen.

The Investigation Pipeline

You will watch the Gather box work live: each tool call, each payload, narrated as it streams.

Intake normalises the alert. Resolve decides which integrations apply from configuration. Plan selects tools per stage. Gather runs the ReAct loop — this is where connector tools execute against live systems, and the only part of the pipeline the audience watches in full. Notice the color discipline: this diagram is all blue because it is all OpenSRE. Domain colors belong to the world being investigated, never to the machine doing the investigating.

Two properties to stress before the demo starts: every tool is read-only, and every failure degrades into a machine-readable payload instead of crashing the loop. Both will be visible on screen within minutes.

IntroAct 1Act 2Act 3Act 4Close

ONE INCIDENT, FOUR ACTS · PRESS → TO REVEAL

T+0, a pod dies…

The Rolling Incident — Act By Act

Reveal each beat with → as it happens in the room — the timeline doubles as the demo’s progress bar. A pod is deleted on screen, the edge starts 503ing, the page fires for real, data pressure and an auth storm pile on, and the fix ships green — while OpenSRE narrates every tool call across Kubernetes, Alertmanager, Grafana, Victoria Logs, Nginx, Tempo, NATS, YugabyteDB, Aerospike, Keycloak, Jenkins and GitHub.

Punchline at T+15: the room has watched one fault cascade through every layer — and watched each layer answer for itself with evidence. Chasing the cascade, not a single culprit, is what converts skeptics.

IntroAct 1Act 2Act 3Act 4Close

RUN-OF-SHOW · EVERY ACT RUNS THE SAME LOOP

Inject → symptom → ask → revert

1. ACT 1./pod-kill.sh --confirm + flood. Ask: what fired, which Grafana rule matches, what do victoria-logs say? Correlate all three.
2. ACT 2./scale-to-zero.sh --confirm + ./nats-flood.sh --confirm. Ask: nginx upstreams, 5xx access tail, failing Tempo trace, NATS lag — cause or casualty?
3. ACT 3Run yb-slow-query.sql + ./keycloak-login-storm.sh --confirm. Ask: slow queries, Aerospike latency, login failures — is the auth storm the outage?
4. ACT 4Push the breaking commit. Ask: which Jenkins build failed, open the fix as a GitHub PR, annotate Grafana, verify in victoria-logs + k8s. Then ./abort-all.sh --confirm.

→ Next: each of your twelve integrations gets its stage moment on the following slides.

This is the slide the facilitator rehearses from. Two projectors if possible: terminal on the left, Grafana plus Alertmanager on the right. Say every revert out loud before the fault runs — the audience should hear the safety net before they see the fire.

Rehearse the failure modes too: if the model stalls, narrate the last good evidence while it recovers; if a tool degrades, point at the degraded envelope as a feature (“watch it keep going”). Both have happened in real runs and both play well live.

IntroAct 1Act 2Act 3Act 4Close

ACTS 1–2 · EACH GETS A STAGE MOMENT

Page. Edge. Trace.

KubernetesACT 1

Show

Deleted pod recreating, restart counts, event timeline — plus the real Grafana CrashLoop.

Takeaway

Ground truth lives in pods and events — everything else interprets them.

AlertmanagerACT 1

Show

The raw firing alert JSON, silences, and which Grafana rule it matches.

Takeaway

The page is structured data — the investigation starts from the payload, not a screenshot.

GrafanaACT 1

Show

Error-rate panel climbing live, alert rules pulled via connector, deploy annotated at fix time.

Takeaway

The chart everyone trusts, pulled by the agent — not screenshotted by a human.

Victoria LogsACT 1

Show

Error spike for store-payments in the last 5 minutes, correlated to the alert window.

Takeaway

Logs confirm the page is real — and bound its blast radius to one service.

NginxACT 2

Show

Upstream health with orders at zero, 5xx access-log summary, error-log tail.

Takeaway

The edge names the failing upstream — the RCA points at a service, not a layer.

TempoACT 2

Show

One failing checkout trace, span by span, ending at the dead upstream.

Takeaway

A trace turns “the site is slow” into “this hop, this service, this second”.

→ Next: data, identity, and the fix — six more integrations.

Read a card, then read its takeaway; the pairs are written to survive being photographed. The first four cards convict the fault (something really died); the last two trace exactly where the user feels it.

Pace: three minutes per integration — one minute of symptom, two of triage. If an answer is slow, narrate the question while the loop works; never stand silent waiting for tokens.

IntroAct 1Act 2Act 3Act 4Close

ACTS 3–4 · EACH GETS A STAGE MOMENT

Data. Identity. Ship.

NATSACT 3

Show

Stream depth and consumer lag after the flood, connections holding steady.

Takeaway

Lag answers “cause or casualty” — the queue absorbed the flood, it didn’t start it.

YugabyteDBACT 3

Show

Slow-query top list with the probe visible, cluster and tserver status green.

Takeaway

The database shows its own work — pressure is measured, not guessed.

AerospikeACT 3

Show

Namespace stats, cluster size 3, write latency under load.

Takeaway

A cleared suspect is evidence too — exoneration makes verdicts stick.

KeycloakACT 3

Show

Login-failure burst for the probe user, sessions intact, no lockouts on real users.

Takeaway

Auth storms look like outages — identity evidence separates the two.

JenkinsACT 4

Show

The red build, the failing stage, then the green rebuild after the fix.

Takeaway

CI is the first witness for “did the fix work” — before users ever could be.

GitHubACT 4

Show

The fix opened as a PR, CI checks attached, merged on screen.

Takeaway

Every incident ends as a change — the loop closes where it opened.

→ Next: the payloads behind the takeaways, quoted verbatim.

This is the slide for anyone who asked “but will it cover X” — all twelve integrations now map to a timestamped moment. The back half flips the story: two cards exonerate (data, identity) while two convict-and-fix (CI, change).

End Act 3 on the Keycloak line — “auth storms look like outages” is the most-retweeted sentence of the workshop. Then break before Act 4; the fix lands better after a pause.

ACT IVFIX · SHIP · CLOSE THE LOOP · 25 MINUTES

Transition beat. “Three acts of breaking. Now watch the same machine that found everything help ship the fix — red build, pull request, annotation, green. Then we score it.”

IntroAct 1Act 2Act 3Act 4Close

NOT SLIDEWARE — REAL OUTPUT SHOWN LIVE

Ruling out is a feature

# Aerospike, mid-incident:
'node_status': 'ok', 'cluster_size': 3, 'cluster_integrity': True

# YugabyteDB, same window:
'node_count': 3, 'active': 4, 'cache_hit_ratio_percent': 0.0  # idle, not sick

# NATS, after the flood:
'stream': 'orders.events', 'lag': 4120, 'state': 'catching_up'  # absorbed it

# Keycloak, during the storm:
'login_failures': 50, 'user': 'workshop-chaos-probe', 'real_users_locked': 0

✓Yugabyte measured ✓Aerospike healthy ✓Queue absorbed it ✓Auth storm isolated  →  chase the cascade, not one culprit.

→ Next: what we refuse to build — and why restraint ships demos.

These are verbatim shapes from live fault-injection runs against real clusters — node counts, integrity flags, per-node connection counters. In the demo the same payloads arrive in the terminal in real time while the room watches. This slide is the zoomed-in proof behind the takeaways on the previous slide.

The rhetorical point to land: an SRE tool earns trust as much by clearing suspects with evidence as by naming culprits. Three healthy databases, stated plainly, is what makes the Nginx verdict believable instead of convenient.

IntroAct 1Act 2Act 3Act 4Close

SCOPE DISCIPLINE, STATED UP FRONT

What we will not build

✕cert-manager integration

Slow-burn expiry, not a 2-hour story — k8s + logs already cover it. Backlog ticket.

✕Chaos Mesh today

kubectl + hey + nats-box cover all 12 integrations. Mesh is the follow-up pilot.

✕A second RCA engine

OpenSRE reasons; nothing competes with it

✕A full dashboard in v1

One static incident page from real JSON

Each rejected item is a rabbit hole that has swallowed similar demos. The cert-manager call deserves one sentence: all certificates read True today, expiry is a slow burn that cannot fire on stage, and the failure modes are already reachable through the Kubernetes and Victoria Logs integrations.

Say this slide’s message out loud in the room: constraints are what make the demo shippable in two hours.

IntroAct 1Act 2Act 3Act 4Close

NO FAULT INJECTION REQUIRED · ALREADY BURNING

The cluster is already on fire (a little)

EXHIBIT A

Grafana CrashLoopBackOff (2/3). Ask: why is monitoring-grafana crashlooping? pods, events, last logs. Free Act 1 triage.

EXHIBIT B

cert-manager-cainjector, 164 restarts. Ask: is issuance healthy despite the restarts? certificates, issuers, recent events. The cert-manager verdict, live.

EXHIBIT C

Konnectivity refused on one node — log fetches can fail there. Narrate it as degraded evidence, not a bug.

Real production is never clean — say so, and spend five minutes on Exhibit A inside Act 1. Triaging a fault nobody injected is the most convincing minute of the workshop: no script, no safety net, just the loop against a genuinely sick pod.

Exhibit B doubles as the cert-manager discussion: restarts with all certificates True is exactly why expiry monitoring is backlog, not blockers. Exhibit C is your rehearsed demo of graceful degradation — when a log fetch fails, point at the envelope continuing, not the error.

IntroAct 1Act 2Act 3Act 4Close

THE MACHINERY · BRANCH opensre-workshop · ISSUE #6244

Eight faults. One abort.

# every script: revert first · DRY_RUN=1 previews · --confirm or refuse
workshop/chaos/pod-kill.sh            # Act 1 · delete one payments pod
workshop/chaos/scale-to-zero.sh       # Act 2 · orders → 0, --revert restores
workshop/chaos/nats-flood.sh          # Act 2 · 5000 msgs via nats-box
workshop/chaos/keycloak-login-storm.sh # Act 3 · 50 wrong-password logins
workshop/chaos/log-flood.sh           # Act 1 · short-lived log emitter
workshop/chaos/yb-slow-query.sql      # Act 3 · read-only probe
workshop/chaos/cpu-hog-job.yaml       # spare · pinned image, 5-min fuse
workshop/chaos/abort-all.sh           # global revert · rescales prod to 2

Contract, tested: dry runs touch nothing · unconfirmed live runs are refused · abort restores all four prod deploys. Seven pytest cases guard it.

Show this slide once, right before Act 4 — by now the room has seen six of these run and trusts the machinery. The point to land: chaos with a contract. DRY_RUN previews, —confirm gates, abort-all restores, and a test suite proves all three.

Invite the room to steal it: the kit plus its tests are the take-home for any team that wants game-days next quarter. That is the adoption seed — don’t sell it, just leave it visible.

IntroAct 1Act 2Act 3Act 4Close

ACT 4 · 25 MINUTES · RED TO GREEN

Ship the fix on stage

1. PUSHCommit the breaking assertion to the demo repo. Watch Jenkins go red on the projector.
2. ASKwhich build failed and why? open the minimal fix as a GitHub PR. Review the diff together.
3. MERGEMerge on screen. Jenkins rebuilds green. Annotate the deploy in Grafana.
4. VERIFYError rate falls in victoria-logs, rollout healthy in k8s. Run ./abort-all.sh --confirm. Incident closed.

→ Next: score all twelve, file the backlog.

Act 4 is the emotional payoff — the same loop that spent an hour finding faults now helps ship. Slow down at the merge: let the room see the PR, the checks, the green build, the annotation. Verification is the sermon — “we don’t believe it’s fixed, we query that it’s fixed.”

Run abort-all live as the closing gesture. Prod rescales, chaos-load drains, pods return. The incident ends the way it should: with the cluster visibly whole.

IntroAct 1Act 2Act 3Act 4Close

CLOSE · 10 MINUTES · TWELVE FOR TWELVE

Scorecard

ACT 1 · PAGE

k8s · alertmanager · grafana · victoria

Pod kill triaged, alert correlated, logs bound the blast radius

ACT 2 · EDGE

nginx · tempo · nats

Upstream named, trace pinned the hop, queue cleared itself

ACT 3 · DEPTH

yugabyte · aerospike · keycloak

Pressure measured, suspects exonerated, storm isolated

ACT 4 · SHIP

jenkins · github

Red to green on stage, PR merged, deploy annotated + verified

Backlog: cert-manager ticket · Chaos Mesh pilot in poc · weekly game-days. Kit lives on branch opensre-workshop (#6244).

→ Next: what Monday looks like.

Read the four cards fast — the room just lived them. The scorecard’s job is a photograph: twelve integrations, four verdicts, one page of proof. Ask for a show of hands on which act surprised them most; that answer tells you which team adopts first.

Name the backlog out loud and assign nothing yet — assignment happens on the next slide, with a named owner.

CLOSESCORE + BACKLOG · 10 MINUTES

Transition beat. “You have watched production break on purpose — and get fixed faster than most real incidents get triaged. Here is how those two hours become a decision.”

IntroAct 1Act 2Act 3Act 4Close

LEAVING WITH MORE THAN PHOTOS

Take-home + 30/60/90

TAKE-HOME

In your hands

Green 12-integration matrix · the 4-beat loop · chaos kit branch (#6244) · this deck as the runbook

ROADMAP

What follows

30 — first game-day on a real service · 60 — Chaos Mesh pilot + cert-manager ticket · 90 — MTTR trend vs today, expand services

Two questions to close the room: which service gets the first game-day? and who owns it?

The integration matrix from pre-flight is the adoption artifact: twelve rows, all green, witnessed live. The 4-beat loop is the method every future game-day repeats. The chaos kit branch is the seed — any team can run DRY_RUN previews this week.

End with a named service and a named owner for the first game-day. Workshops that close on a question get answers; workshops that close on a summary get applause and nothing else.

The integration matrix from pre-flight is the adoption artifact: twelve rows, all green, witnessed live. The 4-beat loop is the method every future game-day repeats. The chaos kit branch is the seed — any team can run DRY_RUN previews this week.

End with a named service and a named owner for the first game-day. Workshops that close on a question get answers; workshops that close on a summary get applause and nothing else.

← → navigate  ·  H help  ·  P console  ·  F5 fullscreen  ·  print for the article version