Mock LLM API server for developers stress-testing LLM app infrastructure
Teams building LLM-powered apps need to load-test their own gateways, queues, and streaming paths but can't afford to burn real token spend on the provider APIs. They end up hand-rolling mock servers. A drop-in, API-compatible fake LLM endpoint with configurable streaming, latency, errors, and rate limits would let them test infra for free.
The problem
When developers stress-test their LLM app infrastructure, hitting the real OpenAI/Claude APIs turns a scalability test into a token-spending test. They currently have to build and maintain their own mocking service from scratch, duplicating effort across every company.
What could be built
A self-hostable or hosted API-compatible mock endpoint for OpenAI/Anthropic that returns deterministic canned responses, simulates streaming, configurable latency/time-to-first-token, token counts, tool calls, 429/5xx errors, and malformed streams—so teams can load-test everything around the model without paying for inference.
Who it's for
Who is talking（1）
Related topics
Summaries are AI-generated. The original words are in the threads above.