Zenhub gates every model swap on 12,000 real prompts, moving to Amazon Bedrock
Amazon BedrockZenhub, the project-management platform built for software teams, runs LLM-powered features in production. VeUP put a governed gateway in front of every model call, built an automated model-evaluation pipeline into the team’s CI from thousands of real production prompts, and is moving the estate onto Amazon Bedrock as the unified model channel — so swapping models becomes a configuration change backed by head-to-head results instead of a code change backed by hope.
The challenge
Zenhub’s AI features call frontier models directly from application code — and not from one vendor. Anthropic, OpenAI and Google models run in parallel in production, each reached through its own provider API, each billed separately. That left no unified metering or observability of model usage and cost, no alerting or diagnosis path when calls failed, billing fragmented across three vendors, and no way to compare candidate models — or adopt a newer, cheaper, or faster one — without engineering work in the application itself. The team had no evidence-based way to answer the recurring question every multi-model product team faces: should we switch?
The engagement
Observability and guardrails.VeUP deployed a LiteLLM gateway routing all model calls, with native tagging, OpenTelemetry export into the team’s observability platform, gateway fallbacks, and budget enforcement. Slack alerting is wired to a structured diagnosis runbook — alert, root cause, resolution — so an engineer receiving an alert has a verified path instead of raw logs. The full workstream was deployed and validated in production, with live alerts flowing through the runbook during the engagement.
Evaluation in CI.VeUP moved the team’s model evaluation into GitHub Actions and made it real: roughly 12,000 production prompts mined from thirty days of live logs — scrubbed of sensitive content — form the evaluation dataset. A 30-test suite runs candidate models head-to-head against the incumbent, an LLM judge scores reasoning quality and tracks hallucination rates, and automated pull-request comments post latency, estimated cost difference, judge scores, and hallucination rates for every comparison. Automated merge-to-main runs are pinned to a low-cost model so continuous evaluation does not itself become a cost problem, and open-source models were benchmarked on the same suite.
Amazon Bedrock as the model channel. The engagement moves Zenhub off direct-to-provider API calls and onto Amazon Bedrock as the unified channel, consolidating multi-provider usage onto a single AWS bill and making model swaps configuration-only at the gateway — non-disruptive to the application. Because the gateway already meters every call, the cutover is sequenced behind a pre-migration cost baseline, so the change in run-rate is measured rather than discovered.
Outcomes
| KPI | Result |
|---|---|
| Governed model traffic | Every model call metered, tagged, and budget-enforced through one gateway with fallbacks — deployed and validated in production, with Slack alerts flowing through a structured diagnosis runbook. |
| Evaluation pipeline | A 30-test evaluation suite in GitHub Actions over ~12,000 real production prompts, with LLM-judge scoring, hallucination-rate tracking, and automated PR scorecards posting latency, cost delta, and quality results on every comparison. |
| An evidence-based non-switch | The harness’s first head-to-head found a candidate model ran ~14% faster but showed no reasoning-quality win over the incumbent — so the team kept its model, on evidence. The same harness now gates every future switch. |
| Unified model channel | Amazon Bedrock adopted as the single model channel — multi-provider usage consolidating onto one AWS bill, with model swaps reduced to configuration changes at the gateway. Cutover is sequenced behind a pre-migration cost baseline taken through the gateway’s own metering. |
| Handover | A six-week engagement closing out with documentation across all three workstreams, so Zenhub’s own engineers operate the gateway, the diagnosis runbook, and the evaluation pipeline themselves. |
AWS IAM
AWS Marketplace