Call this before every inference request
Decide if the call is worth it, where to send it, and how to optimise latency and cost across 15 providers.
GET /v1/should-call
Unknown prices and stale data are explicit. Ranking scores are not probabilities.
What is InferenceLatency.com?
InferenceLatency.com is a real-time AI inference monitoring platform that continuously tests 15 enabled LLM API providers — OpenAI, Anthropic, Groq, OpenRouter, Google Gemini, DeepSeek, Cohere, Together AI, Fireworks AI, Cerebras, SambaNova, xAI (Grok), Mistral AI, Perplexity, Nvidia NIM — measuring time to first token (TTFT), throughput, cost, and reliability. Google Gemini is currently rate-limited, so individual tests may report it as unavailable. Results are available via free JSON APIs with no authentication required.
The primary endpoint is GET /v1/should-call — an advisory routing decision using saved measurements for the currently configured models. It filters fresh, successful samples against your latency and cost limits, then returns a provider, model, eligible alternatives and the evidence behind the ranking. Reading a decision does not trigger paid provider tests. Savings and task quality are not guaranteed.
Measurements use a standardised 1-token prompt sent simultaneously to all providers via their official APIs. Timing is millisecond-precise from request start to first token received (TTFT). Results are stored in a rolling 48-hour database to compute statistically reliable P50, P95, and P99 latency percentiles — the metrics that matter most for SLA planning and production reliability.
Quick Tests
Monitored Providers
Provider Recommender
Ask in plain English which provider to use. Results are live-tested.
Core Performance
?priority=speed|cost|balanced|reliability and ?use_case=chat|code|reasoning for tailored results.Advanced Monitoring
Developer Resources
Connect InferenceLatency tools
Cursor / Windsurf (MCP)
Add the HTTP URL in your IDE MCP settings. Streamable HTTP transport — no auth required.
https://inferencelatency.com/mcp
Claude Desktop (MCP)
Add to your claude_desktop_config.json. Uses SSE transport via mcp-proxy.
{
"mcpServers": {
"inferencelatency": {
"command": "mcp-proxy",
"args": [
"https://inferencelatency.com/sse"
]
}
}
}
Smithery Registry
Find and connect via the Smithery MCP registry. No configuration needed.
https://smithery.ai/server/inferencelatency
Quick Start — curl
# Get fastest provider right now curl https://inferencelatency.com/api/fastest # Full latency ranking (all 15 providers) curl https://inferencelatency.com/latency # Custom benchmark with your prompt curl "https://inferencelatency.com/benchmark?prompt=Explain+RAG&max_tokens=50" # Cost optimizer curl https://inferencelatency.com/cost-optimizer
Agent Discovery
https://inferencelatency.com/mcphttps://inferencelatency.com/openapi.jsonAbout InferenceLatency.com
What is InferenceLatency.com?
InferenceLatency.com is a real-time AI inference monitoring and routing platform. It continuously tests 15 enabled LLM API providers — OpenAI, Anthropic, Groq, OpenRouter, Google Gemini, DeepSeek, Cohere, Together AI, Fireworks AI, Cerebras, SambaNova, xAI (Grok), Mistral AI, Perplexity, Nvidia NIM — measuring time to first token (TTFT), throughput, cost, and reliability, and exposes that data via free JSON APIs. Google Gemini is currently rate-limited, so individual tests may report it as unavailable. No authentication required for most endpoints.
Who is it for?
AI agents and automated pipelines that need to route requests to the optimal provider before each inference call. Developers benchmarking which LLM API to use for a given workload. DevOps and platform teams building AI reliability pipelines and SLA dashboards. Researchers tracking inference performance trends across providers, regions, and models over time.
How are measurements taken?
Every test sends a standardised short prompt with a 1-token output limit to each provider simultaneously using their official APIs. Timing is measured in milliseconds from the moment the HTTP request is sent until the first token is received — this is the TTFT (time to first token) metric. Results are stored in a rolling 48-hour database and used to compute P50, P95, and P99 latency percentiles, giving a statistically reliable picture of both typical and worst-case performance. Providers are tested from a consistent geographic location so results are directly comparable.
What is time to first token (TTFT)?
TTFT is the duration from sending a request until the first token of the response arrives. It is the most important latency metric for interactive AI applications — it determines how quickly users see the response begin. A lower TTFT means a faster-feeling application. Throughput (tokens per second) matters more for long-form generation tasks. InferenceLatency.com tracks both.
How do I use this in my agent pipeline?
The primary integration point is GET /v1/should-call — read-only routing advice with a provider and model, observed short-request latency, verified pricing when available, eligible alternatives and measurement evidence. It does not measure streaming TTFT or guarantee task quality. Keep your configured fallback policy when no eligible choice is available. /api/fastest is a separate live-test ranking endpoint.
FAQ
cost_sensitivity and latency_sensitivity settings to get a routing recommendation that automatically balances these factors based on live data.GET /v1/should-call returns an advisory provider/model choice, observed short-request latency, verified model pricing when available, measurement age, eligible alternatives and exclusion reasons. ranking_score is a heuristic; the legacy confidence_score is null, not a success probability. Optional parameters include task type, sensitivities and explicit latency/cost limits. Unknown prices cannot meet a cost cap; cost_sensitivity=low permits them with warnings. No authentication required for up to 30 requests per day.