AI Center of Excellence structure for 50-500 person companies

An AI Center of Excellence (CoE) at mid-market scale is a small coordinating function — typically 1-4 people — that maintains shared AI capabilities, supports product teams building AI applications, and represents AI in technical decisions. The dedicated team model from large-enterprise AI CoEs does not fit at mid-market. The right structure is usually a guild or hub-and-spoke model run by people with other day jobs.

Why mid-market needs a different model

Large-enterprise AI Centers of Excellence are typically dedicated organizations of 15-50 people: research leads, ML engineers, applied scientists, governance specialists, ethics reviewers. They exist because the cost of coordination across hundreds of teams justifies the overhead, and because regulatory environments require formal review functions.

At mid-market (50-500 person companies), this structure does not fit:

• The headcount budget for a 15-person dedicated AI team is usually not available
• The number of teams that need coordination is small enough (typically 5-20 product teams) that lighter coordination works
• The governance complexity is real but bounded; a full review function is overkill

What works at mid-market is a smaller coordinating function — usually 1-4 people — operating in one of three structural models. The choice depends on company size, AI ambition, and existing engineering organization.

This piece walks the three models, where each fits, and how to evolve from one to another as the company grows. It is the structural companion to our AI enablement roadmap pillar.

The three models

We do not recommend the large-enterprise CoE model (15+ dedicated headcount) at mid-market. It is over-built for the coordination need and under-utilized for the company's actual AI ambition. The companies that try it usually rebuild as one of the three above within 12-18 months.

Model 1: Guild

The Guild model is the lightest-touch structure. There is no separate AI team; AI capability lives in the product engineering teams. The guild is a coordinating fabric across them.

Structure:

• One coordinator (often a senior engineer, principal engineer, or staff engineer with explicit time allocation — 25-40%)
• 4-8 engineers from product teams who self-identify as AI-active
• Recurring guild meeting (bi-weekly works) to share learnings, coordinate tooling, surface decisions

What the guild does:

• Maintains shared infrastructure (default LLM provider relationship, evaluation tooling, common libraries)
• Reviews proposals for new AI applications (lightweight; informational, not gating)
• Documents patterns and anti-patterns from team experience
• Represents AI in cross-cutting decisions (procurement, architecture, security)

What the guild does not do:

• Build AI applications directly. Applications are built by product teams; the guild supports them.
• Make decisions for product teams. The guild advises; teams own.
• Maintain a separate engineering roadmap. Guild work happens in members' product team contexts.

Best for:

• 50-150 person companies
• Companies where AI use cases are distributed across multiple product areas
• Companies where the engineering culture favors team autonomy over central platform teams
• Earlier-stage companies where dedicated AI headcount cannot be justified yet

Where the guild model breaks:

• When AI applications proliferate to the point that "shared infrastructure" needs full-time owners
• When governance requirements (regulatory, security) need formal review functions
• When the coordinator role exceeds 50% time for one person; at that point, dedicating the role becomes more efficient than splitting

Concrete signal it's time to evolve: when the coordinator's 25-40% allocation is consistently exceeded for two consecutive quarters, or when the guild surface area extends beyond what bi-weekly meetings can cover. At that point, move to Hub-and-Spoke.

Model 2: Hub-and-spoke

Hub-and-spoke combines a small dedicated coordinating team (the hub) with embedded engineers in product teams (the spokes).

Structure:

• 2-4 dedicated AI/platform engineers in the hub
• Engineers in product teams who own their team's AI applications, with a dotted-line relationship to the hub
• Weekly hub-spoke sync; recurring guild-style meetings for the broader engineering org

What the hub does:

• Owns shared platform: LLM provider relationships, evaluation infrastructure, common libraries, RAG infrastructure (if relevant)
• Owns governance: vendor evaluations, acceptable-use policy maintenance, audit trails
• Provides direct support to product teams building AI applications; embedded for specific projects
• Represents AI in technical leadership (architecture, security, procurement)

What the spokes do:

• Build AI applications in their product context
• Maintain their applications post-launch
• Use the hub's shared infrastructure rather than building parallel
• Surface gaps in the hub's tooling for hub to address

Best for:

• 150-350 person companies
• Companies with multiple AI applications in production needing ongoing support
• Companies where governance is becoming a real workload (vendor reviews, policy questions, incident response)
• Companies ramping AI ambition where dedicated capacity is justified but a separate organization is overbuilt

Where hub-and-spoke breaks:

• When the hub becomes a bottleneck — too many spokes need too much hub attention. Solution: more hub headcount or better self-service tooling, depending on which gap is bigger.
• When the dotted-line relationship becomes ambiguous about ownership. Specifically: when an AI application breaks, who's on the hook? The hub for the platform or the spoke team for the application? Resolve this in the model design, not at incident time.

Concrete signal it's time to evolve: when the hub team's roadmap has more demand than capacity for two consecutive quarters and you cannot grow the hub without making it a separate organizational function. At that point, consider the small dedicated team model.

Model 3: Small dedicated team

A small dedicated team is a separate organizational unit — typically reporting to a CTO, VP of Engineering, or a Chief AI Officer in larger mid-market companies — that owns AI capability development and supports product teams.

Structure:

• 4-8 dedicated team members: AI/ML engineers, applied scientists if appropriate, governance/policy lead, sometimes a product manager
• Embedded relationships with product teams that have heavy AI usage
• Quarterly planning aligned with broader engineering planning

What the dedicated team does:

• Owns the AI platform end-to-end: infrastructure, tooling, observability, governance
• Builds AI capabilities (shared services, evaluation frameworks, fine-tuning pipelines if applicable)
• Provides consulting and embedded support to product teams
• Represents AI in executive-level decisions

What the dedicated team does not do:

• Take over product team AI applications. Product teams remain the owners; the dedicated team is the platform and capability support.
• Build features that compete with vendor offerings. Build vs buy stays an explicit decision per use case.

Best for:

• 350-500 person companies
• Companies where AI is a stated strategic priority (board-level, executive-stated)
• Companies with regulatory or compliance requirements that need formal AI governance
• Companies acquiring smaller AI-capable companies and needing to integrate

Where this model breaks:

• When the team becomes a compliance bottleneck — every AI initiative routes through them and wait times grow
• When the team builds for hypothetical future need rather than current product team need
• When the team's roadmap diverges from the company's product strategy

Concrete signal it's working: product teams ship AI features faster with the team than they would alone, governance questions are resolved within reasonable cycle time, the team's own retention and morale are stable.

Choosing between models

A simple decision framework:

When the answers point to different models, pick the higher-overhead one. The cost of being slightly over-structured is recoverable; the cost of being under-structured shows up as coordination friction that erodes engineering momentum.

The exception: pre-Series-B companies with the headcount of mid-market but the resource constraints of startup. Stay with Guild even if other indicators point higher; the headcount investment in Hub-and-Spoke or Dedicated may not pay back at that growth stage.

Common structural mistakes

Patterns we see at mid-market:

Hiring a Head of AI without a clear scope. When the role is "do AI things" without specifying what or for whom, the role flounders. Define the scope first (what stage are you at, what's the year-one mandate, what's the relationship to product teams) before hiring.

Building a centralized AI team that owns AI applications. This pattern fights product team autonomy. Product teams resist; the central team becomes a bottleneck. The fix is to move ownership back to product teams with the central team as a platform/support function.

Naming a "CoE" without dedicating headcount. A CoE that exists on paper but has no allocated time produces no work. Either dedicate the time or use a lighter Guild structure with explicit allocation.

Mixing the platform and governance functions. When the same team builds AI infrastructure and governs AI use, the conflict-of-interest dynamic distorts both. At mid-market, the team is small enough that the conflict is manageable; document the explicit decision rules to keep it healthy.

Importing the Big Tech CoE model. Models from Google, Meta, Microsoft AI organizations are designed for hyperscale; they don't fit mid-market. Don't copy organizational structure from companies 1000x your size.

Funding the team

Funding an AI capability is its own conversation with finance. Three approaches we've seen work:

Embedded budget. AI capability work happens within product engineering budgets; no separate AI line item. Works for Guild and early Hub-and-Spoke. Easy to justify; harder to track total AI investment.

Centralized AI budget. A separate AI capability line item, funded by reallocation from product engineering or by net-new investment. Standard for Hub-and-Spoke and Dedicated. Easier to track; harder to justify in years where AI doesn't show clear top-line impact.

Chargeback to product teams. AI infrastructure costs are billed back to the product teams using them. Works for mature operations; adds accounting overhead. Mid-market typically does not need this until later.

The right approach depends on how AI ROI is measured at the company. If AI is treated as a product capability, embedded budget works. If AI is treated as a strategic priority, centralized budget signals that. If AI is large enough to be a recurring conversation with finance, chargeback adds the discipline.

Reporting structure

Where the AI function reports affects what it can do.

For mid-market, CTO/VP-Engineering is the most common right answer for Guild and Hub-and-Spoke. Chief Product Officer or a dedicated Chief AI Officer becomes appropriate at the larger end (Small Dedicated team) when AI ambition justifies the executive role.

Evolving the model over time

The natural progression as companies grow:

Guild -> Hub-and-Spoke -> Small Dedicated -> (eventually) Enterprise CoE

The transitions are not sudden. A Guild can have one mostly-dedicated coordinator before becoming Hub-and-Spoke. Hub-and-Spoke can have a hub that grows from 2 to 4 to 6 before splitting into a Dedicated team plus product platform. Each transition is a 6-12 month period.

The mistake to avoid: skipping a stage. A company moving from Guild directly to Small Dedicated without the Hub-and-Spoke transition often discovers that the dedicated team and the product teams haven't built the working relationships to collaborate effectively. The intermediate stage isn't optional; it's where the coordination patterns get learned.

Where this framework breaks

Honest limitations:

• It assumes a centralized engineering function. Companies with autonomous business units (acquired companies, multiple product lines with separate engineering orgs) need an overlay model. We have not codified that here.
• It assumes English-speaking primary work environment. Multi-region companies with multi-language requirements add structural complexity, particularly for governance.
• It assumes the company has a strategic posture on AI. Companies where AI is being adopted reactively (because customers are asking, because competitors do it) may not need any of these structures yet.

FAQ

Q: We have a Head of Data Science. Do we need a separate AI function? Often, yes, at the mid-market scale. Data science teams are typically focused on analytics, ML modeling, and metrics; AI/LLM application development is a different skill set and a different operational shape. The two functions can coexist, with the AI function focused on LLM-powered application capabilities and the data science function focused on analytics and traditional ML. Combining them works only at smaller scale.

Q: Should our AI function build models in-house? At mid-market, almost never. The cost-effectiveness of frontier API providers (Anthropic, OpenAI, Google) and open-weights models (Llama, Mistral, Qwen) means in-house model training rarely pays back unless you have a domain-specific moat (proprietary data of meaningful scale, regulatory requirement for on-prem, specific custom model that vendors can't approximate).

Q: How do we measure the AI function's success? By the success of the AI applications product teams ship, not by the function's internal metrics (number of platform features, headcount, etc.). The AI function's job is to make product teams successful with AI; that's what should be measured.

Q: Should the AI function own customer-facing AI features? We strongly recommend product teams own customer-facing AI features. The AI function provides platform and support. The exception is highly regulated AI applications (financial advice, medical diagnosis adjacent) where the AI function may need formal sign-off; even then, ownership stays with the product team.

Q: What's the right title for the leader of the AI function? At Guild, no separate title needed; "AI Coordinator" or similar functional designation works. At Hub-and-Spoke, "Director of AI Engineering" or "Head of AI Platform" is common. At Dedicated team scale, "VP of AI" or "Chief AI Officer" depending on how strategic AI is to the company. The title matters less than the scope and authority.

*For broader AI enablement context, see our pillar on the AI enablement roadmap for mid-market. For the governance practices these structures need to enforce, see our LLM governance framework.*