InferGauge

InferGauge measures how AI and LLM endpoints behave under load: latency, time to first token, inter-token latency, throughput, error rates, goodput, response quality and cost.

With InferGauge in ChatGPT you can:

• Describe a load, stress, spike or endurance test in plain language and get a validated InferGauge configuration, plus the command to run it.
• Ask how your latest test went and get the score, SLA results and main finding, with a link to the full report.
• Find where your application saturates: the concurrent-user level where latency starts to climb.
• Check whether a release regressed, by comparing a run with the previous run of the same test.
• Generate a GitHub Actions workflow that fails pull requests when performance gets worse than a threshold you set.

Tests run on your own machine or CI runner with the InferGauge CLI, never inside ChatGPT. Prompts, model responses, API keys and private endpoint URLs stay on your infrastructure. Only metrics and findings sync to your InferGauge console, and ChatGPT sees only your own account's results.