17 of 800 posts
An agent can call the wrong tool, or call the right tool with the wrong arguments. This guide covers three ways to test each failure mode, a Python harness that grades both, and how to run the same test cases against several tool-capable...
A golden eval dataset is a curated set of production inputs with reviewed expected outputs, versioned in Git and run before every deploy.
An agent's behavior can change when you edit a prompt, swap a model, change a tool schema, or change what retrieval returns.
If you already have the image a video should start from, the model choice comes down to what has to happen after that frame.
We audited our own OpenRouter account and found over 1,000 active API keys for 85 employees, 168 of them hadn't been used in months.
Qwen 3.8 is Alibaba's August 2026 model generation. Four models share the name and they differ in whether you can download them, which license applies, whether they read images and video, and whether you can turn reasoning off.
ByteDance's Seedance video models are available through hosted APIs, and we found no downloadable checkpoint or model license for any of them.
Kimi K3 ships public weights under Moonshot AI's own Kimi K3 License, which is not an OSI-approved open-source license. This post explains the difference, what the license grants and requires, what the checkpoint contains, and how to call...
A method for putting Jev to work on a new problem, worked end to end on marketplace listing moderation: what stays in code, what Jev sees, how to phrase the questions, and how to turn its probabilities into publish, hold, or reject.
An embedding model decides what your retrieval system can find. We shortlisted the embedding models in our catalog for English RAG, multilingual retrieval, code search, text-and-image retrieval, and low-cost indexing, sent live requests to...
Nemotron 3.5 Lightning is NVIDIA's open-weight 30B mixture-of-experts model with about 3B active parameters per token, built for the high-volume execution calls in an agent run.
Claude Opus 5 leads OpenRouter's classification task ranking by spend. We sent the same 3,080 Banking77 utterances to it and to Jev 1.13 through the Decisions API.
Send a whole workload in one POST, collect the results within 24 hours, and typically pay half the per-token price. Across 230k+ batches that completed over our two week beta period, the median finished in 7 minutes.
Jev reads text and returns typed answers with probabilities instead of prose. Here's what that means, three live API calls, how to read the numbers, and how to call it with an OpenRouter key.
An LLM judge writes a verdict. Jev returns a probability. We graded the same labeled answers and expert-rated summaries with both, and the difference decides which one you should use for a given rubric.
For Descript, testing a new model was a couple of hours of work and a week of waiting. The team removed the waiting. Evaluations run in an hour or two now, and nobody has to ask an engineer for time.
Jev returns typed judgments with probabilities. An LLM returns prose. We benchmarked both on 100 support cases through OpenRouter, then wired them together so Jev routes and the LLM writes.
Blogsby Glyph