Back to tags
#sre
slos, error budgets, and the pager rotation that makes them real.
3 blog posts · 1 TIL.
Posts about the part of the job that is measured in pages rather than features. The burn-rate alerting post is the argument for paging on the error budget instead of the CPU graph, with the two-window rule that stops an alert firing for an hour after the incident ended. The JA4 post is the incident that argument exists for: one fingerprinting rule, thirty percent of production, reconstructed from the timeline. The crash-loop triage is the 3am version, five causes separated by an exit code and a reason field.
Blog posts
Threshold alerts page you for causes that may not matter. Multi-window burn-rate alerts page you once, for the symptom, when the error budget is genuinely…
CrashLoopBackOff is a symptom with five common causes. The exit code and the last event narrow it to one in under a minute, before you open the…
Deployed a TLS fingerprinting rule that seemed reasonable. Blocked every Chrome 119 user on Windows. The incident report was not fun to write.
Today I learned
Rotating ciphers exploded our TLS client metrics to 50k unique fingerprints. JA4's split format dropped that to under 200 without losing detection.