sep 2, 20268 min read
Why your Docker build caches locally and never in CI
A CI runner starts with an empty layer store, so every build is a first build. Cache mounts, instruction order, and an exported cache fix it in about ten…
I tinker with infrastructure and write about what broke. The good stuff is at the top; the rest is in the archive.
A CI runner starts with an empty layer store, so every build is a first build. Cache mounts, instruction order, and an exported cache fix it in about ten…
Threshold alerts page you for causes that may not matter. Multi-window burn-rate alerts page you once, for the symptom, when the error budget is genuinely…
CrashLoopBackOff is a symptom with five common causes. The exit code and the last event narrow it to one in under a minute, before you open the…
A Kubernetes namespace isolates the API, not the silicon. Under time-slicing two teams share a physical GPU with no memory wall. Quotas, isolation, and…
A model server that takes five minutes to load and a liveness probe that gives it ten seconds is a crash loop waiting to happen. Probes, drains, and safe…
Same GPUs, same model, same replica count. Swap round-robin for prefix-cache-aware routing and the fleet gets 2.3x faster. The router was throwing the…
Idle GPUs at six dollars an hour are a bonfire. Scaling to zero saves the money, but the first user back waits minutes unless you kill the cold start.
The GPU dashboard says 92% busy and users are waiting eight seconds for the first token. Monitoring an LLM server means watching the queue, not the…
Past one node the network becomes the machine. InfiniBand vs RoCE, gang scheduling, FSDP and Megatron, and why a 16k-GPU cluster fails every three hours.
Eight GPUs in one server behave like a small network. NVLink vs PCIe, reading nvidia-smi topo -m, NCCL transports, the ACS trap, and fitting a 70B model.
A GPU pod sits on a dozen layers from silicon to scheduler, and each one fails its own way. Drivers, the container toolkit, MIG, DCGM, and the metrics…
Ten years of git distilled into eight daily aliases: gst, glola, gd, gcam, an fzf branch picker, and the weekly branch-pruning ritual that keeps repos…
Excalidraw is fast, but everything I make in it looks the same. Seven tools that promise visuals with attitude, one diagram, three I'd keep.
Ivanti made everyone re-read their VPN architecture in January 2024. Tailscale, Cloudflare Tunnel, and WireGuard in one afternoon.
Your company's MDM dropped managed-settings.json and the network team wedged Claude through an AI gateway. Here's how each leash works.
Snowflake taught everyone what happens when an infostealer runs on a contractor's personal Mac. The laptop is the perimeter.
Four DNS records that close the entire phishing-impersonation class: SPF, DKIM, DMARC, and CAA, plus two monitors, set up correctly in one afternoon.
Identity, network access, default credentials, attestation, and audit logs: the unsexy DevSecOps controls that close most of the gap Parts 1 and 2 left…
Hardening GitHub Actions for small teams: SHA pinning against tag rewrites, OIDC over long-lived secrets, dependency cooldowns, and the trigger to never…
Startup-grade defense against npm supply-chain attacks, for Future You at 3am. Chainjacking, postinstall scripts, smallest install, most leverage.
How to self-host SimpleLogin with Docker, Postfix and Brevo on a $3/month VPS. DNS, reply-from-alias, and the TLS gotcha that ate two hours of my Sunday.
Tried booking a flight. Got blocked. Turns out Akamai thinks my 21 security extensions make me look like a hacker. They're not wrong.
VictoriaLogs vs Loki, benchmarked on 500 GB of logs over 7 days on identical nodes: 94% lower query latency, 37% less storage, under half the CPU and RAM.
Netlify suspended five free-tier sites of mine one Tuesday night. The 15-minute migration to Dokploy on a €3/month VPS that bought everything back.
Deployed a TLS fingerprinting rule that seemed reasonable. Blocked every Chrome 119 user on Windows. The incident report was not fun to write.
What I rewatch when the day's debugging is done: That 70's Show, Arrested Development, HIMYM, and the detective canon from Holmes to Poirot.
GitHub Actions vs GitLab CI after two years running both across 50 microservices: build times, free-tier minutes, Docker support, debugging, and which to…
A practical guide to setting up Prometheus and Grafana for production monitoring. No theory, just battle-tested configurations that work.
Five Kubernetes debugging tricks from real 3 AM pages: kubectl logs --previous, ephemeral debug containers, scheduler events, and fixes that saved…
Your containers are probably insecure. Here's how I learned to harden Docker containers the hard way, and the security mistakes that almost cost us.
Our AWS bill hit $50k/month. Here's exactly how EC2 rightsizing, Reserved Instances, S3 lifecycle rules, and CloudWatch retention cut it to $20k in six…
Terraform mistakes I made so you don't have to: hardcoded AMIs, lost local state, a 2000-line main.tf, an unpinned provider, and a stray terraform destroy.