jul 23, 20269 min read
Your model isn't crashing, your probe is
A model server that takes five minutes to load and a liveness probe that gives it ten seconds is a crash loop waiting to happen. Probes, drains, and safe…
I tinker with infrastructure and write about what broke. The good stuff is at the top; the rest is in the archive.
A model server that takes five minutes to load and a liveness probe that gives it ten seconds is a crash loop waiting to happen. Probes, drains, and safe…
Same GPUs, same model, same replica count. Swap round-robin for prefix-cache-aware routing and the fleet gets 2.3x faster. The router was throwing the…
Idle GPUs at six dollars an hour are a bonfire. Scaling to zero saves the money, but the first user back waits minutes unless you kill the cold start.
The GPU dashboard says 92% busy and users are waiting eight seconds for the first token. Monitoring an LLM server means watching the queue, not the…
Past one node the network becomes the machine. InfiniBand vs RoCE, gang scheduling, FSDP and Megatron, and why a 16k-GPU cluster fails every three hours.
Eight GPUs in one server behave like a small network. NVLink vs PCIe, reading nvidia-smi topo -m, NCCL transports, the ACS trap, and fitting a 70B model.
A GPU pod sits on a dozen layers from silicon to scheduler, and each one fails its own way. Drivers, the container toolkit, MIG, DCGM, and the metrics…
Ten years of git distilled into eight daily aliases: gst, glola, gd, gcam, an fzf branch picker, and the weekly branch-pruning ritual that keeps repos…
Excalidraw is fast, but everything I make in it looks the same. Seven tools that promise visuals with attitude, one diagram, three I'd keep.
Ivanti made everyone re-read their VPN architecture in January 2024. Tailscale, Cloudflare Tunnel, and WireGuard in one afternoon.
Your company's MDM dropped managed-settings.json and the network team wedged Claude through an AI gateway. Here's how each leash works.
Snowflake taught everyone what happens when an infostealer runs on a contractor's personal Mac. The laptop is the perimeter.
Four DNS records that close the entire phishing-impersonation class: SPF, DKIM, DMARC, and CAA, plus two monitors, set up correctly in one afternoon.
Identity, network access, default credentials, attestation, and audit logs: the unsexy DevSecOps controls that close most of the gap Parts 1 and 2 left…
Hardening GitHub Actions for small teams: SHA pinning against tag rewrites, OIDC over long-lived secrets, dependency cooldowns, and the trigger to never…
Startup-grade defense against npm supply-chain attacks, for Future You at 3am. Chainjacking, postinstall scripts, smallest install, most leverage.
Self-hosted SimpleLogin with Docker, Postfix, and Brevo for $3/month. The TLS gotcha that ate two hours of my Sunday, written down so you skip it.
Tried booking a flight. Got blocked. Turns out Akamai thinks my 21 security extensions make me look like a hacker. They're not wrong.
500 GB of logs, 7 days, same hardware. VictoriaLogs vs Loki: 94% lower query latencies, 37% smaller storage, half the CPU and RAM.
Netlify suspended five free-tier sites of mine one Tuesday night. The 15-minute migration to Dokploy on a €3/month VPS that bought everything back.
Deployed a TLS fingerprinting rule that seemed reasonable. Blocked every Chrome 119 user on Windows. The incident report was not fun to write.
What I rewatch when the day's debugging is done: That 70's Show, Arrested Development, HIMYM, and the detective canon from Holmes to Poirot.
Two years running GitHub Actions and GitLab CI across 50 microservices side by side. Here's the CI/CD comparison data and which one I'd reach for and when.
A practical guide to setting up Prometheus and Grafana for production monitoring. No theory, just battle-tested configurations that work.
Five Kubernetes debugging tricks from real 3 AM pages: kubectl logs --previous, ephemeral debug containers, scheduler events, and fixes that saved…
Your containers are probably insecure. Here's how I learned to harden Docker containers the hard way, and the security mistakes that almost cost us.
Our AWS bill hit $50k/month. Here's exactly how EC2 rightsizing, Reserved Instances, S3 lifecycle rules, and CloudWatch retention cut it to $20k in six…
Terraform mistakes I made so you don't have to: hardcoded AMIs, lost local state, a 2000-line main.tf, an unpinned provider, and a stray terraform destroy.