The decision sits in front of every platform engineering leader once a third team asks for an OpenAI key. The first two requests were one-offs. The third is a pattern. Within ninety days the company will have five teams making LLM calls, three different ways of handling secrets, and no shared view of what is being spent. The question is not whether to centralize. It is what to centralize. A thin gateway gets billing, keys, and observability under one roof, then leaves teams to build their own apps. An internal LLM platform adds shared RAG infrastructure, a prompt registry, an evaluation harness, and policy enforcement on top. Both are right answers. They are right at different stages, and the cost of choosing the larger one too early is a platform team that owns infrastructure no one is using.
This guide is the framework we walk through with mid-market clients (50-500 people, typically Series B through pre-IPO) when they ask us to help them decide. It assumes you have already made the centralization call. The question now is the scope.
Why the "just put a gateway in front" advice undersells the decision
Most public guidance reduces the choice to "use LiteLLM and move on." That works as a starting tactic. It fails as an architecture decision because it ignores the four variables that actually drive the answer: how many teams are calling LLMs, whether they share a knowledge corpus, what your regulatory posture is, and how mature your internal AI practice has become. We have seen companies put LiteLLM in production in a week, then spend the next eighteen months bolting on RAG infrastructure, a prompt versioning system, and an eval harness piecemeal, because every team rebuilt those pieces in incompatible ways. The cost of the rebuild is real. So is the cost of a platform team of two trying to operate a platform that needs six engineers to keep healthy.
The framework below names the two endpoints clearly, defines the staged path between them, and gives you a self-assessment so you can place your organization honestly. The cost and timeline figures throughout are drawn from roughly two dozen AI enablement engagements we have run over the last eighteen months across mid-market SaaS, financial services, and regulated healthcare. They are reference points, not benchmarks; your variance against them will be driven mostly by existing platform maturity.
The Optivulnix LLM Platform Maturity Framework
We use a four-stage model. The names matter because they map to staffing decisions and quarterly roadmaps, not just architecture diagrams.
Stage 1: Direct Access (no central infrastructure)
Each team uses provider SDKs directly. Keys live in team-owned secret stores. Spend shows up on the provider invoice with no per-team attribution. This is fine for organizations with one or two teams running LLM workloads at under five thousand dollars a month combined.
What it is: the absence of a platform decision.
What it is not: a long-term position. The moment a third team joins, you have a governance gap.
Outputs: working applications, no shared telemetry, no consolidated billing.
Failure modes: keys leaked into client-side bundles, no rate-limit coordination during outages, finance unable to attribute spend to business units.
Stage 2: Thin Gateway (the LLM proxy decision)
A single proxy sits in front of all model providers. Teams call the gateway, the gateway calls OpenAI, Anthropic, Google, Bedrock, or self-hosted endpoints. The gateway handles authentication, per-team rate limits, cost attribution by tag, request and response logging, fallback chains, and basic caching. Teams still own their applications, their prompts, their evaluation, and their data retrieval.
What it is: centralized control of the request path.
What it is not: a platform. The gateway does not know what a prompt template is, does not host vector indexes, does not run evaluations, and does not enforce content policy beyond what you script into a middleware hook.
Outputs: unified observability, per-team billing, a single point to swap providers or revoke keys, an audit log that your security team can actually use.
Failure modes: teams build five different RAG stacks because the gateway does not solve retrieval; prompts live in application code and cannot be versioned or A/B tested across teams; the gateway becomes the bottleneck during incidents because no one designed it for high availability.
Realistic build cost: a two-person platform team can stand up a self-hosted gateway in three to six weeks, with another four to six weeks to wire it into existing applications. Operating cost is typically one engineer at twenty to thirty percent of their time. Infrastructure for a single-replica deployment serving moderate traffic can run under one thousand dollars a month, but a properly redundant deployment (two gateway replicas, an HA Postgres pair, managed Redis) on most major clouds lands closer to two-and-a-half to four thousand dollars a month before egress. The volume ceiling is workload-dependent; we have seen the same baseline configuration handle anything from low single-digit to low double-digit millions of tokens per day, depending on prompt size and concurrency.
Stage 3: Platform Foundation (gateway plus shared services)
The gateway is now one component in a small set of shared services. You add a managed vector database (typically one production instance shared across teams with namespace isolation), a prompt registry where prompts are versioned and tied to evaluation results, and a centralized evaluation harness that teams can plug into. Governance shifts from "we log everything" to "we enforce policies before requests reach providers."
What it is: a thin platform with three or four shared services, owned by a small platform team, consumed by application teams.
What it is not: a hyperscale internal AI cloud. There is no fine-tuning service, no model registry for custom-trained models, no agent orchestration layer.
Outputs: shared knowledge corpora that multiple teams query against; a registry where you can answer "which application is using which version of which prompt against which model"; the ability to run an evaluation suite on every prompt change in CI.
Failure modes: the platform team is staffed at two people and tries to operate four services; teams refuse to migrate off their own RAG implementations because the shared one is missing a feature they already built; the evaluation harness becomes a science project that no one runs.
Realistic build cost: this is where teams underestimate. Across the Stage 3 buildouts we have run or audited in the last eighteen months, a credible delivery has needed a four-to-six-person platform team for three to six months, plus another two quarters to drive real adoption. Steady-state operation runs three to six engineers. Infrastructure in the eight to twenty thousand dollars a month range is typical for the buildouts we have seen, driven mostly by vector storage footprint and reserved inference capacity. Smaller organizations land lower; teams that over-provision for projected growth land higher.
Stage 4: Internal AI Platform (the full build)
Everything in Stage 3, plus self-hosted inference for at least one open-weight model family, an agent orchestration layer, fine-tuning infrastructure, model evaluation that runs nightly against a golden dataset of meaningful size, and policy enforcement that includes PII redaction, content filtering, and per-use-case approval workflows. Most mid-market companies should never reach this stage. By the time the staffing and infrastructure profile below is justified, the organization has often already crossed out of the mid-market band we focus on (50-500 people) into the lower enterprise tier, and the planning conversation changes accordingly. Those that commit while still mid-market usually have a specific regulatory or cost driver, not an aspirational one.
What it is: a small-scale internal AI cloud.
What it is not: a competitive advantage on its own. The advantage comes from the applications, not the platform.
Outputs: full control of the inference path, per-token cost reduction at scale, the ability to host models that providers cannot offer for compliance reasons.
Failure modes: the platform team ships a stack that is two years behind the managed provider ecosystem; self-hosted inference utilization sits at fifteen percent because workload demand is bursty; the agent orchestration layer is rebuilt three times in two years as the orchestration framework debate replays.
Realistic build cost: in the two Stage 4 buildouts we have advised on directly and the three we have observed from adjacent engagements, the consistent shape has been six to twelve engineers, twelve to eighteen months to credible production, and infrastructure spend that starts around thirty thousand dollars a month and rises with self-hosted GPU utilization. We flag this as a small sample; the floor in particular is sensitive to whether the team buys reserved GPU capacity or runs on-demand.
The four-variable assessment
Place your organization on each axis before deciding which stage you should occupy in the next two quarters.
Variable 1: Team count using LLMs
One or two teams: Stage 1 or 2. The overhead of a thin gateway is hard to justify until a third team joins.
Three to six teams: Stage 2 is the right answer for almost everyone. This is the modal mid-market company we work with.
Seven or more teams: Stage 3 starts to pencil out, but only if the next two variables align.
Variable 2: Shared RAG corpus or not
If teams are answering questions against materially different knowledge bases (engineering docs, sales collateral, HR policies), they do not share a corpus. A shared vector database is a coordination tax, not a benefit. Stay at Stage 2 and let teams pick their own vector store.
If two or more teams need to query the same set of source documents (a customer knowledge base, a product catalog, a policy library), the duplication cost becomes real. Each team builds its own ingestion pipeline, its own chunking strategy, its own embedding refresh schedule. A shared retrieval service is the highest-leverage Stage 3 component for these organizations.
Variable 3: Regulatory posture
Low-regulation environments (most B2B SaaS, internal productivity use cases): Stage 2 with provider data processing agreements is usually sufficient.
Regulated data regimes (financial services under regional banking regulators, healthcare workloads touching protected health information under HIPAA, anything in scope for GDPR or DPDPA-style personal data law): the policy enforcement layer that comes with Stage 3 becomes load-bearing. The specific controls differ by regime, but the common requirement is pre-request enforcement: PII or PHI redaction before requests leave your network, request logging with the right retention period, and the ability to route different use cases to different model endpoints based on data classification. See our LLM governance framework for the controls these environments require.
Heavily regulated or sovereignty-constrained environments (defense, certain healthcare segments, government): Stage 4 becomes plausible because self-hosted inference may be the only compliant option. Even here, validate that managed providers cannot meet your residency requirements before committing to a self-hosted stack.
Variable 4: Internal AI maturity
Use the AI Center of Excellence model from our AI CoE structure guide as a proxy. Organizations without a functioning CoE, without a published AI usage policy, and without a model risk owner are not ready for Stage 3. The platform investment will outpace the organizational capacity to use it.
Organizations with a CoE in place, an evaluation discipline that survived first contact with production, and at least one application generating measurable business value can absorb Stage 3 investment. The platform amplifies what is already working.
Reading the assessment
Score each variable on the stage it points to (1, 2, 3, or 4) and treat the most common value as the ceiling for the next two quarters. If you get four different values across the four variables, anchor on the lower of the two middle values; the conservative choice is almost always right because under-building is recoverable in a quarter and over-building usually is not. Worked example: a company scoring Stage 2, Stage 2, Stage 3, Stage 4 has Stage 2 as the most common value and stays at Stage 2. The Stage 4 driver is real but it is solved through targeted exceptions, not through a full platform build.
We have never seen a mid-market organization succeed by building two stages ahead of where this assessment places them.
Honest read on the OSS gateway landscape
If the assessment puts you at Stage 2, the next question is whether to self-host LiteLLM, Helicone, or Portkey, or to use the managed versions. As of mid-2026, here is how we see the three options when we are evaluating them for clients. The descriptions below combine direct operating experience in our engagements with the broader pattern we observe across the mid-market ecosystem; we have flagged the distinction where it matters.
LiteLLM
The most adopted self-hosted option among the mid-market teams we work with directly. Strong provider coverage (more than a hundred upstream providers, with the model count higher because most providers expose multiple models), straightforward Python and proxy deployments, an active project. The proxy server is the part most teams care about: it handles routing, virtual keys, budgets, and fallback chains. The tradeoff is operational maturity. In our engagements, teams encounter rough edges in the admin UI, the database schema occasionally changes in ways that require care during upgrades, and high-availability deployments require deliberate work on Redis, Postgres, and load balancer configuration. The project has also begun shipping Model Context Protocol and agent-primitive support in the upstream proxy (see https://github.com/BerriAI/litellm), which narrows the gap between the gateway layer and what used to be a separate orchestration concern.
Best fit: platform teams of two-to-four who want full control and are willing to absorb operational ownership.
Helicone
Originally an observability-first product, now a credible proxy with strong cost-tracking, logging, and caching primitives. The important context for any team evaluating Helicone in 2026: Helicone announced its acquisition by Mintlify in March 2026 (see https://www.helicone.ai/blog, "Helicone Joins Mintlify"). The product is operating and the OSS components remain available, but the medium-term roadmap is now tied to Mintlify's product direction rather than Helicone's standalone trajectory. For Stage 2 teams choosing today, this means the observability features are real and usable, but we recommend treating long-horizon roadmap commitments (deep policy primitives, advanced routing) as uncertain until Mintlify publishes a combined product direction. Provider coverage is narrower than LiteLLM regardless. If observability is the primary problem you are solving and you can absorb the acquisition-related uncertainty, Helicone is still often the fastest path to value.
Best fit: teams whose first problem is "we cannot see what we are spending and what is being asked" rather than "we need policy enforcement and fallback routing," and who are comfortable with the post-acquisition roadmap risk.
Portkey
Strongest among the three on the governance and routing primitives: guardrails, semantic caching, prompt management, and conditional routing are first-class concepts rather than added later. Portkey publicly cites coverage across forty-five-plus providers and the broader LLM catalog those providers expose (Portkey markets this as 1,600-plus LLMs), so the comparison with LiteLLM is genuinely close once you normalize provider count versus model count. Portkey's stack splits cleanly into two paths. The MIT-licensed open-source gateway (see https://github.com/Portkey-AI/gateway) self-hosts via npm, Docker, or Cloudflare Workers at no license cost and is a legitimate Stage 2 option in its own right, particularly for teams that want first-class routing and guardrail primitives without the LiteLLM operational footprint. The separate Enterprise tier adds the prompt registry, advanced governance, multi-tenant controls, and the support model that Stage 3 deployments typically need. Like LiteLLM, Portkey has shipped MCP and agent-framework integration in the gateway in recent releases, which we factor into the buy-vs-build math for organizations otherwise looking at a separate orchestration layer.
Best fit: teams that want the governance primitives early (OSS gateway at Stage 2) or are clearly heading to Stage 3 within the next two quarters and want the prompt registry and policy primitives without rebuilding them (Enterprise tier).
A note on managed providers
For a Stage 2 deployment serving fewer than five teams at under twenty thousand dollars a month in provider spend, the managed version of any of these tools is almost always cheaper than self-hosting once you account for engineering time. The build-it-yourself math starts to work when provider spend crosses fifty thousand dollars a month or when regulatory constraints force a self-hosted deployment regardless of cost. We cover the deeper cost-comparison work in our LLM API cost optimization guide.
The staged path from gateway to platform
The most common mistake we see is treating Stage 2 and Stage 3 as separate projects. The teams that get this right treat Stage 2 as the foundation on which Stage 3 components are layered, one at a time, with a clear adoption criterion before the next component is added.
Months 1 to 3: stand up the gateway
Deploy a self-hosted or managed gateway. Wire two or three high-value applications through it. Get billing attribution, request logging, and rate limiting working. Define the tagging convention (team, application, use case, data classification) that every other component will inherit. Do not build anything else until two consecutive months of clean per-team billing reports have gone out and at least one application team has used the observability data to optimize a workload.
Months 4 to 6: solve the highest-pain shared component
Pick the single Stage 3 component that solves the most acute pain across your application teams. For most clients it is one of: a shared vector database, a prompt registry, or a centralized evaluation harness. Build or adopt that one component. Do not start the second until the first has at least two teams using it in production.
Months 7 to 12: extend deliberately
Add the second and third Stage 3 components on the same cadence. Re-run the four-variable assessment at month nine. If the assessment now places you at Stage 4, that is a board-level conversation about staffing and infrastructure spend, not a quiet roadmap addition.
When to slow down
If at any quarterly checkpoint adoption of a shared component is below thirty percent of eligible teams, stop adding new components. The problem is not platform completeness. It is platform-product fit, and adding more surface area will make the adoption problem worse.
Where this framework breaks
It assumes a relatively coherent engineering organization. In companies with strong divisional autonomy, where each business unit has its own platform team and its own technology choices, a single shared platform is often the wrong answer regardless of the assessment. The right answer there is a federated model: shared standards (tagging, observability schema, policy primitives) with team-owned implementations.
It also assumes the LLM provider ecosystem stays roughly as it is. If a single provider's economics or capability set shift dramatically, the build-vs-buy line moves. We update this framework when the underlying market moves.
Finally, it scopes agent orchestration as a deliberately separate decision. The major gateways now ship MCP and agent-framework primitives, so the line between gateway and orchestrator is genuinely blurry in 2026. We still treat orchestration as its own decision because the staffing, governance, and evaluation work for agentic systems is large enough that bundling it into the gateway scope tends to under-resource both. The framework above gets you the request path and the observability; the orchestration layer needs its own analysis even when the gateway can technically host it.
FAQ
When does a thin gateway stop being enough?
When more than two application teams have independently built RAG infrastructure, when prompt versioning across teams is impossible to reconstruct during an incident, or when regulatory requirements force pre-request policy enforcement that the gateway cannot express. Until at least one of these is true, the gateway is enough.
Can we skip Stage 2 and go straight to a platform?
We have not seen it work in the mid-market segment. The Stage 2 work establishes the tagging, observability, and provider-abstraction primitives that every Stage 3 component depends on. Skipping it means rebuilding those primitives inside each Stage 3 component, which is how organizations end up with inconsistent telemetry across services they thought were unified.
How many engineers do we need to operate a Stage 2 gateway?
A self-hosted gateway serving five to ten application teams typically requires twenty to thirty percent of one platform engineer's time once it is in steady state. The total stand-up window we quote is two to three months end-to-end: roughly three to six weeks for a two-person team to ship the gateway itself and another four to six weeks to wire existing applications through it. High availability and disaster recovery add another half-engineer of attention during the first two quarters.
Is LiteLLM production-ready for regulated environments?
It can be, but the burden is on you. The project itself does not ship with the audit-logging schema, retention controls, or PII-handling primitives that regulated deployments need. You will need to wire those in through middleware and through your surrounding infrastructure. Several of our financial services clients run LiteLLM in production; none of them treat it as a turnkey solution.
When does building an internal AI platform actually pay off?
When the answer to all four of these is yes: you have more than seven teams making LLM calls, at least two of them share a knowledge corpus, your regulatory posture requires controls a gateway cannot enforce, and you have a CoE plus published AI policy already operating. If any one is no, the platform investment outpaces the organizational capacity to use it.
What should we do about agent orchestration in this framework?
Treat it as adjacent even though the gateways can now technically host MCP and agent primitives. The framework gets you the request path, the observability, and the policy layer. Agent orchestration sits above that and needs its own decision about whether to build, buy, or adopt an open-source framework. Most mid-market clients we work with run agents through the same gateway as their non-agent applications, then make the orchestration decision per-use-case rather than platform-wide.
Where this goes next
If you are at Stage 1 or 2 today and trying to decide whether to invest in a Stage 3 platform in the next two quarters, the framework above is the conversation we have with clients in week one of an AI enablement engagement. The honest answer for most mid-market organizations is "not yet, and here is what would have to be true for that to change." The exceptions are real but they are exceptions. Knowing whether you are one of them is worth the half-day of structured assessment work.
