Prompt caching: 10x cheaper LLM tokens, but how?
A far more detailed explanation of prompt caching than anyone asked for: how tokens, embeddings, and attention make cached LLM tokens 10x cheaper and faster.
3 of 358 posts
A far more detailed explanation of prompt caching than anyone asked for: how tokens, embeddings, and attention make cached LLM tokens 10x cheaper and faster.
Route AI requests to OpenAI, Anthropic, Google, and self-hosted models through one endpoint, with automatic failover, key rotation, and CEL-based selection.
We integrated Coraza into our Traffic Policy system, ran it against every request to ngrok.com, and blocked 1.2% of traffic. Here's how we designed it and what attack patterns we're seeing.