Observability

Story behind this blog post A couple of months ago, a friend of mine approached me with a challenging problem: building a service that can identify Iranian ISPs from IP addresses at an extreme scale. The requirements were brutal: around 500,000 requests per second with a P99 latency of under 5 milliseconds. As a reference for the data load, Iran has about 10,000,000 (Ten Million) IPs allocated across roughly 50,000 CIDR ranges, and the system was gonna be in the hot path of a very high-load environment. ...

In the early days of software development, many teams measured the health of their applications through a single lens: logs that printed cryptic messages whenever something went wrong (or occasionally when something went right). Over time, as systems grew more complex and needs broadened, engineers embraced additional approaches, like metrics that could highlight performance trends and spikes. Eventually, logs and metrics alone were no longer enough. Distributed systems, microservices, containers, and orchestrators demanded a whole new way of understanding application behavior. This is how the idea of Observability started to take shape—an effort to see right into the beating heart of the system and know exactly what’s going on, without guesswork. ...