Caching and Performance

Caching is storing the result of an expensive operation so you can reuse it instead of recalculating it every time. When a user requests their profile, you query the database once, cache the result in memory, and serve the cached version to the next 1000 requests. Caching trades memory for speed and freshness for performance. Every cache introduces a new problem: when does the cached data become stale, and how do you invalidate it?

There are cache layers at every level of the stack. The browser caches static assets like images and CSS. A CDN (content delivery network) caches entire pages at edge locations near users. Your application server caches database query results in memory. Your database caches frequently accessed rows. Each layer has different invalidation semantics and different costs. Understanding which layer to cache at determines whether your optimization works or creates bugs.

The two hardest problems in computer science are cache invalidation, naming things, and off-by-one errors. Cache invalidation is hard because deciding when cached data is no longer valid requires understanding all the ways data can change. If you cache a user's profile and they update their email, you must invalidate the cache. If you forget to invalidate, users see stale data. If you invalidate too aggressively, you lose the performance benefit.

Time-to-live (TTL) is the simplest invalidation strategy. Cached data expires after a fixed duration — 60 seconds, 5 minutes, 1 hour. The cache automatically refreshes when the TTL expires. This works well when you can tolerate stale data for the duration of the TTL. It fails when you need immediate consistency — if a user changes their password, you cannot wait 5 minutes for the cache to expire before enforcing the new password.

Write-through and write-behind are strategies for keeping caches in sync with the database. Write-through updates both the cache and the database simultaneously — slower writes but guaranteed consistency. Write-behind updates the cache immediately and writes to the database asynchronously — faster writes but risk of data loss if the system crashes before the database write completes. The choice depends on whether you prioritize speed or durability.

The cache hit ratio measures how often requests are served from the cache versus hitting the underlying data source. A 95% hit ratio means 19 out of 20 requests avoid the database. A 50% hit ratio means caching is barely helping. Low hit ratios indicate either the wrong things are being cached or the cache is too small and evicting frequently accessed data. Monitoring hit ratios is how you know if your caching strategy works.

Thundering herd is the problem that occurs when a popular cache entry expires and suddenly thousands of requests hit the database simultaneously trying to regenerate it. If the database cannot handle the spike, it crashes, making the problem worse. The solution is cache stampede protection: the first request to detect the expired cache locks the key, regenerates the value, and updates the cache while other requests wait for the updated value instead of querying the database.

Not everything should be cached. Data that changes frequently or is unique per user often costs more to cache than it saves. Caching a user-specific recommendation feed that is regenerated every time they log in provides no benefit — you still have to compute it. Caching a homepage that changes once per day and is viewed by millions provides massive benefit. The rule is: cache what is expensive to compute and accessed frequently by many users.