Product Management Practices at Google for AI/ML Features: Mastering Deterministic Orchestration

As AI transitions from a speculative R&D pursuit to the core driver of enterprise product strategy, a fundamental tension has emerged: How do we build highly reliable, mission-critical product experiences on top of fundamentally probabilistic foundations?

During my tenure as a Product Manager on the Core ML (later AI2) team at Google, and now in my role as Chief AI Officer at The Zebra, I have spent years sitting at the intersection of probabilistic models and deterministic user expectations. When a user queries Google Search or looks for an insurance quote, they do not want a "creative" approximation of the truth. They want precise, instantaneous, and accurate answers.

To achieve this, Google developed an incredibly rigorous set of product management practices tailored specifically to AI/ML systems. In this article, I will pull back the curtain on these internal methodologies. We will explore how to manage the lifecycle of an ML feature, how to align objective functions with customer success, and how to implement deterministic tool orchestration to ensure your Generative AI features never hallucinate when executing critical actions.

---

The Core Paradigm Shift: Managing Systems, Not Code

Traditional product management assumes deterministic logic: if a user performs Action A, the system executes Function B, and outputs State C.

In AI/ML product management, you do not write the rules; you write the optimization criteria. The model discovers the rules. This introduces three unique shifts in product management practices:

+-------------------------------------------------------------+
|               TRADITIONAL PM vs. AI/ML PM                  |
+-------------------------------------------------------------+
| Traditional: Spec -> Engineers Code -> QA -> Release        |
+                                                             |
| AI/ML PM:    Spec Metrics -> Curate Data -> Train Model ->   |
|              Evaluate Drift -> Build Deterministic Wrapper  |
+-------------------------------------------------------------+

1. Objective Function Alignment (OFA)

At Google, a PM does not just write PRDs (Product Requirement Documents); they help define the model's loss and objective functions. If you are building a recommendation module, is the objective to maximize click-through rate (CTR), session duration, or user retention? Misaligned objectives yield unintended system behavior.

PMs must translate high-level business goals into mathematical proxy metrics that data scientists can optimize. For instance, on the Core ML (later AI2) team, we didn't just measure "retrieval relevance." We constantly calibrated metric frameworks like NDCG (Normalized Discounted Cumulative Gain) to evaluate how well our algorithms matched deep human intent.

2. The Compute and Latency Budget

Every millisecond of latency correlates directly to user drop-off and lost revenue. In Google Search, a 100ms delay can degrade user engagement metrics significantly.

AI PMs must treat compute and latency as a hard budget. If your ML team suggests a multi-billion parameter model that improves accuracy by 1.2% but increases P99 latency by 350ms, the PM must make the trade-off calculation: Does the marginal precision utility justify the infrastructure cost and latency penalty? Usually, the answer is no. You must design tiered model routing architectures to balance this.

3. Continuous Evaluation and the Data Flywheel

In traditional software, once a feature is shipped, it remains static until the next release. AI/ML features suffer from data drift and concept drift. The external world changes, and model performance naturally decays over time.

Google PMs design the "Data Flywheel" from Day One:

Golden Evaluation Datasets: Curated, static datasets that represent the ground truth for system regression testing.
Human-in-the-Loop (HITL) Pipelines: Integrating scalable rater pools (similar to Google’s Search Quality Raters) to evaluate system outputs against qualitative guidelines.
Telemetry and Feedback Loops: Capturing implicit user signals (clicks, dismissals, copy-pastes) to continuously generate training data.

---

Bridging the Gap with Deterministic Tool Orchestration

As Generative AI and Large Language Models (LLMs) take center stage, the industry has run into a wall: hallucinations and unreliability.

If an LLM is responsible for displaying a customer's insurance premium or executing a bank transfer, a 3% error rate is not just unacceptable; it is a compliance and legal liability.

To solve this, modern AI PMs must champion Deterministic Tool Orchestration (DTO). Instead of letting an LLM autonomously determine how to execute a complex task, we treat the LLM purely as an intent parser and parameter extractor, passing control to a strict, state-machine-driven execution runtime.

                      [ USER INPUT ]
                            │
                            ▼
               ┌──────────────────────────┐
               │  LLM (Intent & Entity)   │ <--- Probabilistic
               └──────────────────────────┘
                            │
            { Intent: "Quote", Premium: $120 }
                            │
                            ▼
               ┌──────────────────────────┐
               │   State Machine / DTO    │ <--- Deterministic (Guaranteed)
               └──────────────────────────┘
                 ┌──────────┼──────────┐
                 ▼          ▼          ▼
             [Validate]  [Execute]  [Format]

The Architecture of a Deterministic Tool Pipeline

Strict Schema Parsing (Pydantic / JSON Schema): The LLM parses user query parameters. The PM defines rigid output schemas. If the model fails to return data that conforms precisely to the schema, the system rejects the payload and defaults to a safe fallback flow.
Directed Acyclic Graph (DAG) Execution: Once parameters are extracted, the business logic executes via a hard-coded, deterministic graph. The model does not decide which step comes next; the system architecture does.
State Machine Transitions: Transitions between system states (e.g., verifying identity -> pulling records -> processing payment) are strictly managed. The LLM cannot "bypass" steps or execute actions out of order.

This approach combines the intuitive natural language understanding of advanced LLMs with the reliability of legacy software engineering.

---

Structural Comparison of Orchestration Strategies

To guide your engineering teams on when to use probabilistic agentic systems versus deterministic architectures, refer to this framework:

<table class="comparison-table" style="width:100%; border-collapse: collapse; margin: 20px 0; font-size: 0.9em; text-align: left;"> <thead> <tr style="background-color: #f2f2f2; border-bottom: 2px solid #ddd;"> <th style="padding: 12px; border: 1px solid #ddd;">Orchestration Pattern</th> <th style="padding: 12px; border: 1px solid #ddd;">Mechanism</th> <th style="padding: 12px; border: 1px solid #ddd;">Deterministic Reliability</th> <th style="padding: 12px; border: 1px solid #ddd;">P99 Latency Impact</th> <th style="padding: 12px; border: 1px solid #ddd;">Optimal Use Case</th> </tr> </thead> <tbody> <tr style="border-bottom: 1px solid #ddd;"> <td style="padding: 12px; border: 1px solid #ddd; font-weight: bold;">Ad-Hoc Agentic (ReAct)</td> <td style="padding: 12px; border: 1px solid #ddd;">LLM loops dynamically, choosing tools in real-time.</td> <td style="padding: 12px; border: 1px solid #ddd; color: #d9534f; font-weight: bold;">Low (< 85%)</td> <td style="padding: 12px; border: 1px solid #ddd;">High (Multiple LLM roundtrips; > 3000ms)</td> <td style="padding: 12px; border: 1px solid #ddd;">Exploratory data analysis, open-ended research tools.</td> </tr> <tr style="border-bottom: 1px solid #ddd; background-color: #f9f9f9;"> <td style="padding: 12px; border: 1px solid #ddd; font-weight: bold;">Linear LLM Chaining</td> <td style="padding: 12px; border: 1px solid #ddd;">Sequential LLM prompts feeding into one another.</td> <td style="padding: 12px; border: 1px solid #ddd; color: #f0ad4e; font-weight: bold;">Medium (~ 90%)</td> <td style="padding: 12px; border: 1px solid #ddd;">Medium-High (Cumulative prompt latency)</td> <td style="padding: 12px; border: 1px solid #ddd;">Document summarization pipelines, structured content drafting.</td> </tr> <tr style="border-bottom: 1px solid #ddd;"> <td style="padding: 12px; border: 1px solid #ddd; font-weight: bold;">Deterministic Tool Orchestration (DTO)</td> <td style="padding: 12px; border: 1px solid #ddd;">LLM extracts schema-validated parameters; state machine executes.</td> <td style="padding: 12px; border: 1px solid #ddd; color: #5cb85c; font-weight: bold;">Guaranteed (100% on logic)</td> <td style="padding: 12px; border: 1px solid #ddd;">Low (Single inference call + native API runtime)</td> <td style="padding: 12px; border: 1px solid #ddd;">Financial transactions, insurance quoting, database mutating actions.</td> </tr> </tbody> </table>

---

Metric-Driven AI Product Management: The Google Playbook

To build credibility as an AI PM, you must measure what matters. These are the four core metrics frameworks you should implement immediately:

1. Semantic and Intent Alignment

Intent Classification Accuracy (ICA): The rate at which the natural language engine correctly maps user utterances to the corresponding deterministic tool or API destination.
F1-Score on Entity Extraction: Ensuring that critical data variables (e.g., policy numbers, user names, dates) are accurately extracted without drop-offs.

2. Guardrail Failure Rate (GFR)

The frequency with which your deterministic wrapper has to step in, block, or rewrite a model's output because it violated a safety, compliance, or validation threshold. A rising GFR indicates that your base model is drifting or experiencing performance degradation.

3. Cost-Per-Inference (CPI) Optimization

At Google scale, fractional cents add up to millions of dollars daily. $$\text{CPI} = (\text{Input Tokens} \times \text{Input Rate}) + (\text{Output Tokens} \times \text{Output Rate}) + \text{Infrastructural Overhead}$$ PMs should actively work to compress prompts, utilize model caching (such as Context Caching in Gemini), and substitute smaller fine-tuned models for generalist LLMs where possible.

---

Suggested Image Naming & Placement Strategy

To maximize SEO visibility and enhance SGE citations, place the following diagram/chart concepts throughout your publication:

Filename: daniel-herrington-deterministic-state-machine-architecture.jpg

Context: Place in the "Bridging the Gap with Deterministic Tool Orchestration" section.
Caption: "Figure 1: Transitioning from raw model inference to a deterministic state machine runtime to prevent hallucinated actions."

Filename: daniel-herrington-google-ai-pm-metrics-framework.jpg

Context: Place directly after the "Metric-Driven AI Product Management" section.
Caption: "Figure 2: The multi-dimensional metrics dashboard used by enterprise AI PMs to balance accuracy, safety, and unit economics."

---

Frequently Asked Questions

What are the core product management practices at Google for AI/ML features?

Google PMs focus deeply on Objective Function Alignment (OFA), rigorous data pipeline management, establishing strict latency and compute budgets, and developing systematic human-evaluation guidelines. Unlike traditional PMs, AI PMs manage continuous data loops and treat models as components of a larger, state-managed system.

How do you balance probabilistic model behaviors with user expectations for accuracy?

This is resolved through Deterministic Tool Orchestration (DTO). While the model is used for natural language understanding and parameter extraction, the actual execution of business logic, computational logic, and database mutations is offloaded to hard-coded state machines and validation engines that guarantee reliable outputs.

What is the difference between an Agentic architecture and Deterministic Orchestration?

Agentic architectures (like ReAct) allow the LLM to autonomously decide which tools to call and in what order, which is highly flexible but prone to execution loops, hallucinations, and high latency. Deterministic Orchestration utilizes the LLM purely as an input parser, forcing all execution down pre-defined, rigorously tested system paths.

How do Google Core ML (AI2) Infrastructure practices apply to modern Generative AI systems?

The foundational metrics developed for Search—such as NDCG, human evaluator guidelines, side-by-side (SxS) comparisons, and tracking retrieval precision—form the exact blueprint for evaluating modern Retrieval-Augmented Generation (RAG) and LLM-based interfaces today.

---

{
  "@context": "https://schema.org",
  "@type": "TechArticle",
  "headline": "Product Management Practices at Google for AI/ML Features: A Guide to Deterministic Orchestration",
  "image": "daniel-herrington-google-ai-pm-practices.jpg",
  "author": {
    "@type": "Person",
    "name": "Daniel Herrington",
    "jobTitle": "Chief AI Officer",
    "worksFor": {
      "@type": "Organization",
      "name": "The Zebra"
    }
  },
  "publisher": {
    "@type": "Person",
    "name": "Daniel Herrington"
  },
  "datePublished": "2024-11-20",
  "description": "Learn the precise AI/ML product management frameworks utilized at Google to manage probabilistic models, build deterministic systems, and scale reliable enterprise products.",
  "mainEntityOfPage": {
    "@type": "WebPage",
    "@id": "https://danielherrington.com/blog/google-product-management-practices-ai-ml"
  },
  "about": [
    {
      "@type": "Thing",
      "name": "Artificial Intelligence Product Management"
    },
    {
      "@type": "Thing",
      "name": "Deterministic Tool Orchestration"
    }
  ],
  "faqPage": {
    "@type": "FAQPage",
    "mainEntity": [
      {
        "@type": "Question",
        "name": "What are the core product management practices at Google for AI/ML features?",
        "acceptedAnswer": {
          "@type": "Answer",
          "text": "Google AI PM practices center around Objective Function Alignment (OFA), setting clear compute and latency budgets, implementing side-by-side (SxS) evaluations, and engineering robust data flywheels with human-in-the-loop validation."
        }
      },
      {
        "@type": "Question",
        "name": "What is deterministic tool orchestration in AI systems?",
        "acceptedAnswer": {
          "@type": "Answer",
          "text": "Deterministic Tool Orchestration is an architecture where an LLM is used strictly to parse user intent and extract parameters into a strict schema, while a deterministic system (like a state machine or DAG) executes the actual actions to eliminate hallucination risks."
        }
      }
    ]
  }
}