Building Live Voice Agents. Deploying Real-Time Multimodal AI Systems to Production Heiko Hotz, Sokratis Kartakis

Dostępność produktu

Produkt nie został jeszcze oceniony pod kątem ułatwień dostępu lub nie podano żadnych informacji o ułatwieniach dostępu lub są one niewystarczające. Prawdopodobnie Wydawca/Dostawca jeszcze nie umożliwił dokonania walidacji produktu lub nie przekazał odpowiednich informacji na temat jego dostępności.

Spis treści książki

• Preface
  • Who Should Read This Book
  • Goals of This Book
  • What This Book Is Not
  • Navigating This Book
  • Conventions Used in This Book
  • Using Code Examples
  • OReilly Online Learning
  • How to Contact Us
  • Acknowledgments
• I. Foundations of Modern AI Systems
• 1. Intelligent Agents and Collaborative AI
  • The Birth of Agents
    • Stage 0: Legacy Agents
    • Stage 1: The Foundation Models and Prompts
    • Stage 2: Foundation Models and Context Retrieval
    • Stage 3: Modern AI Agents: Foundation Models, Context Retrieval, and Actions
    • Stage 4: Multi-Agent Systems
  • What Is an AI Agent?
  • Defining Actions: How Agents Become Doers Using Tools
  • Orchestrating the Core of an Agent
    • Setting Up the Core and Sending the Request
    • The Models Response: A Call to Action
    • From Request to Execution: Your Codes Role
    • Closing the Loop: Execution and Final Response
    • Summary of the Execution Loop
    • Moving to Multi-Turn Agents
      • The stateful execution loop
      • Memory architectures: Short-term versus long-term
  • Agents Calling Agents: The Power of Multi-Agent Systems
  • Conclusion
• 2. Understanding Multimodality: Beyond Text
  • The Sensory World of AI: From Pipelines to Native Multimodal Interfaces
    • The Cascaded Pipeline Approach
    • Native Multimodal Interfaces
    • Tokens, Embeddings, and Shared Representation Spaces
  • Giving AI Ears: The Mechanics of Speech and Audio
    • Acoustic Encoders and Audio Representations
    • Native Audio Generation
  • Giving AI Eyes: The Mechanics of Computer Vision
    • Patches and Visual Representations
    • Processing Video Through Time
  • The Work of Fusion: Processing Simultaneous Inputs
  • Generating Multimodal Outputs
    • Human-Facing Media
    • Machine-Facing Structured Data
  • Conclusion
• II. Building Live, Interactive Experiences with Real-Time Multimodal APIs
• 3. Architecting for Real-Time AI Interaction
  • The Real-Time Voice Loop
  • Hands-On: Building Your First Live Voice Application
    • Architecting for Real-Time AI Interaction
    • Why a Web App? Browser-Based Echo Cancellation
    • Gemini Live API
    • Our Interaction Model: Hot Mic, Mute, and Interruption
      • The user interface
      • How the conversation works: VAD and interruption
    • Prerequisites and Project Setup
    • Step 1: Configure the Static Frontend
    • Step 2: Capture PCM Audio in the Browser
    • Step 3: Open the Gemini Live API Session Through the Proxy
    • Step 4: Return Audio and Transcript Events
    • Step 5: Stop Queued Playback on Interruption
    • Checkpoint: The Full Conversation
    • Local Code Checks
  • Final Review
  • What This Architecture Does and Does Not Solve
  • Conclusion
• 4. Advanced Live Interactions: Video, Tools, and System Instructions
  • Guiding the Session with Instructions and Voice
    • System Instructions and Voice Configuration
    • Hands-On: Configuring Instructions and Voice
      • Step 1: Define session behavior in system instructions
      • Step 2: Choose a voice and update the configuration
      • Checkpoint: A guided assistant
  • Adding Visual Context
    • Camera and Screen Context
    • Hands-On: Integrating Camera and Screen Context
      • Step 1: Update the user interface
      • Step 2: Create the MediaHandler class
      • Step 3: Capture and send frames
      • Step 4: Wire everything together in the browser
      • Step 5: Update the backend proxy
      • Checkpoint: Visual context
  • Adding Controlled Tool Use
    • A Local Architecture for Tools
    • Hands-On: Building and Integrating Local Tools
      • Step 1: Write the tool logic
      • Step 2: Teach the AI about the tools
      • Step 3: Wire up execution in the proxy
      • Step 4: Preserve provenance
      • Checkpoint: Tool use and provenance
  • Testing the Responsive Interface
    • Responsive Interface Continuity
    • Hands-On: Testing the Responsive UI Locally
    • Scope Check: Local Only
  • Conclusion
• III. Architecting and Implementing Agentic Systems
• 5. Core Components of an Agent Framework
  • Runtime: Keeping the Agent Moving
    • The Starting Loop
    • The Execution Loop
    • State Management
    • Lifecycle Management: Pause, Resume, and Stop
    • Nonblocking Execution
    • Control Flow and Routing
    • Concurrency and Parallelism
  • Memory: Keeping the Right Context Alive
    • Short-Term Memory: The Conversational Workspace
    • Long-Term Memory: The Persistent Knowledge Base
    • How Short-Term and Long-Term Memory Work Together
  • Tools: Turning Intent into Bounded Action
    • From Ad Hoc Calls to Tool Schemas
    • Tool Definition and Registration
    • Tool Selection Through Function Calling
    • Secure Execution
    • Framework Patterns for Tools
  • Coordination: Dividing Work Without Losing Control
    • Subagents, Agent Teams, and Coordination Topologies
    • Delegation: Passing Work to a Specialist
    • Information Sharing: Passing Context
    • Coordination Patterns
  • Instructions: Giving the Agent Its Charter
    • Structured Instructions
      • Dynamic context injection
      • Instruction layers and reusable guidance
      • Persona and role definition
    • Tool Schema Integration
    • Instructions Are Necessary but Not Sufficient
  • Choosing a Framework Style
  • Conclusion
• 6. Designing and Building Agents
  • Designing Your Agent
    • Step 1: Define the Agents Purpose and Tools
    • Step 2: Determine the Required Context
    • Step 3: Design the Core Instructions
    • Step 4: Provide Examples of Execution
  • Environment Initialization
  • Building the Basic Agent
    • Instantiating an Agent
    • Interacting with an Agent
  • Adding Tools to the Basic Agent
    • Creating the Tools
    • Querying the Agent
  • Transforming Our Math Agent into a Live Agent
    • Improving Agent Implementation
      • The agents capabilities: tools.py
      • The agents persona: context.py
      • The golden path: examples.py
      • The agents constitution: prompt.py
      • Assembling the agent: agent.py
    • From Live-Streaming Models to Live Agents
    • Live Agent Deep Dive
    • Running Your Live Agents Backend and Frontend
  • Improving and Evaluating the Agent
  • Conclusion
• 7. Multi-Agent Systems: Collaboration and Orchestration
  • Design Your Multi-Agent System
    • Split Agents Like Microservices
    • Choose a Collaboration Pattern
    • Design the Context Exchange
  • Implement Your First Multi-Agent System
    • Prerequisites
    • Implement the Specialist Agents
    • Multi-Agent Sequential Pattern
    • Callbacks for Control and Robustness
      • To load initial context (in the orchestrator)
      • To enforce guardrails (in the subagents)
    • Context Exchange
    • Interact with the Teaching Assistant Agent
  • Transform Your Multi-Agent System to Live
  • Conclusion
• IV. Enabling Interoperability with Open Protocols
• 8. The Agent2Agent Protocol
  • A2A Principles and Capabilities
    • Opaque Execution
    • Async First
    • Modality Agnostic
    • Enterprise Ready
    • Simple and Consistent
  • The Main Actors in A2A
  • Agent Card: The Business Card of an Agent
  • A2A Main Components
    • The Task: The Unit of Work
    • Artifacts Versus Messages
    • Parts: The Atoms of Content
  • A2A Communication Mechanisms
    • Synchronous Communication (Request/Response and Polling)
    • Asynchronous Streaming (Server-Sent Events)
    • Asynchronous Push (Webhooks)
  • A2A and Security
    • Authentication: Who Are You?
    • Transport Security: Is the Line Secure?
    • Authorization and RBAC: What Are You Allowed to Do?
    • Opaque Execution: Security by Design
    • Webhook Verification: Securing the Return Trip
  • Conclusion
• 9. Implementing A2A-Compliant Agents
  • The A2A Adapter Pattern
  • Setting Up the Local Code
  • Reference Implementation 1: Calling Across Frameworks
    • Server Side: Publish a LangGraph Agent as an A2A Service
    • Client Side: Call the Remote Agent Without Importing LangGraph
  • Reference Implementation 2: Wrap an Existing Agent Without Rewriting It
  • Reference Implementation 3: Compose Specialists Through A2A
    • Research Specialist Services
    • Building the Research Coordinator
  • Client Response Contracts
    • Handling File Artifacts
    • Structured Data Payloads
    • Debugging A2A Interactions
  • Conclusion
• 10. The Model Context Protocol: Standardizing Tool Interaction
  • Architectural Limitations of Native Function Calling
    • Constraint 1: Glue Code and Provider Adaptation
    • Constraint 2: Security and Control
    • Constraint 3: The Ad Hoc State Machine
    • Constraint 4: The Static Toolset Problem
  • The MCP Architecture: A Dynamic Host-Server Boundary
    • The Server: The Capability Specialist
    • The Host: The Trusted Orchestrator
    • The Client: The Protocol Session
    • Initialization and Discovery
    • Configured Capability Discovery
  • MCP Primitives: The Protocol Vocabulary
    • Server Primitives
      • Tools: Executable capabilities
      • Resources: Readable context
      • Prompts: Reusable interaction templates
    • Client Primitives
      • Roots: Filesystem boundaries advertised by the client
      • Sampling: Advanced server-requested model assistance
      • Elicitation: Conditional server-requested user input
  • The Transport Layer: Communication Channels
    • stdio: Local Subprocess Communication
    • Streamable HTTP: Remote and Shared Servers
  • Conclusion
• 11. Implementing MCP-Enabled Agents and Tool Servers
  • The Stock Data Agent Scenario
  • Setting Up the Chapter 11 Code
  • Building the MCP Stock Server
    • Normalizing Tickers
    • Exposing a Quote Tool
    • Returning an Image
    • Running the Server by Itself
  • Enabling the ADK Agent as an MCP Client
  • Defining the Stock Data Agent
  • Running a Query Through the Agent
  • The Execution Flow
  • Handling Image Payloads
  • Verifying the Server and Client
  • Optional Manual Debugging with the MCP Inspector
  • Production Considerations
  • Conclusion
• V. Synthesizing Live, Multi-Agent, Multi-Protocol Systems
• 12. Reference Architecture: An Integrated Live Agent System
  • Establishing Service Boundaries
  • Tracing the Request Lifecycle
  • Implementing Service Interfaces
    • MCP Tool Server
    • A2A Specialists
    • Host, Backend, and Browser
  • Run and Verify Locally
  • Deploy to Cloud Run and Open the App on Your Phone
    • Build the Runtime Image
    • Store the Model Key
    • Deploy the Tool and Specialist Services
    • Deploy the Live Backend and Frontend
    • Verify Laptop and Phone Access
  • Simplifications and Antipatterns
  • Conclusion
• 13. Asynchronous Operations and Long-Running Tasks in an Integrated Agent System
  • The Core Pattern: From Submission to Completion
  • Responsibilities Across ADK, A2A, and MCP
    • Layer 1: Protecting the Local Process
    • Layer 2: Persistent Tasks and Short-Lived Connections
  • ADK and Long-Running Tools
  • A2A: Delegating Long-Running Work to Remote Agents
  • MCP: Long-Running Tool Work and Protocol Boundaries
  • Human-in-the-Loop as Asynchronous Suspension
    • Using ADK Tool Confirmation as a Suspension Point
    • Authentication as Suspension
    • Distributed Input Requests
  • Operationalizing the Task Lifecycle
    • Frontend Responsiveness and State Consistency
    • Progress Reporting
    • Cancellation and Cleanup
    • Timeouts, Expiration, and Ghost Tasks
    • Idempotency: The Hidden Requirement
    • Observability Across Agent, Agent, and Tool Boundaries
    • Security in Long-Running Workflows
  • Tracing the End-to-End Asynchronous Workflow
  • Antipatterns to Avoid
  • Production Checklist
  • Conclusion
• 14. Designing User Experience for a Live, Multimodal, Multi-Agent System
  • Why Live Multi-Agent UX Is Different
  • From Live Chat to Live Workspace
  • Voice-First, Visual-Augmented Design
  • State Legibility and Interruption
    • Designing Live System States
    • Turn Taking, Interruptions, and Barging In
    • Showing Agent Activity Without Exposing Machinery
  • Structuring the Asynchronous Workspace
    • Designing for Asynchronous Tasks
    • Generative UI: When the Agent Sends the Components
  • Trust, Approval, and Recovery
    • Provenance, Sources, and Confidence
    • Human-in-the-Loop UX: Approval, Authentication, and Structured Input
    • Error Presentation and Recovery
  • Personalization and Inclusive Design
    • Personalization and Contextual Adaptation
      • Make personalization visible
      • Define the scope
      • Make personalization reversible
    • Accessibility and Inclusive Multimodal Design
  • End-to-End UX Walkthrough
  • UX Patterns and Antipatterns
  • Production UX Checklist
  • Conclusion
• VI. Productionizing
• 15. The Birth of AgentOps: Introduction to an Agent Operationalization Platform
  • From MLOps to AgentOps: Defining the Shift for Agentic Systems
    • The Foundation: DevOps and MLOps
    • The Rise of Model Creators and Consumers
    • The Application Era: GenAIOps
    • The Leap to Autonomy: AgentOps
  • The Essentials of MLOps and GenAIOps: People, Processes, and Technology
    • MLOps Foundations
      • People and process: Data scientists collaborate with other teams
      • Technology
    • The Shift to GenAIOps
      • People and processes: Select and evaluate models
        • Choosing the right model
        • Preparing your data: From prompt engineering to a robust prompt catalog
        • Defining success: Finding the right evaluation metrics
        • Conduct rigorous model evaluation
        • Choosing a model based on business priorities
        • Building a robust backend and user-friendly frontend
      • Technology: Architecting a GenAIOps Platform
  • AgentOps: Productionizing Agents at Scale
    • A Quick Recap: How an Agent Is Built and Run
    • People and Processes: Core AgentOps Processes and Challenges
      • Agent evaluation and optimization
      • Tool orchestration and governance: The tool registry
      • Agent orchestration and governance: The agent registry
      • Memory and data governance
    • Technology: The Unified AgentOps Platform
  • Unifying MLOps and GenAI/AgentOps
  • Conclusion
• 16. Evaluation and Testing of Live Agent Systems
  • Classic Agent Evaluation
    • Single-Turn Evaluation
      • Step 1: Create unit tests
      • Step 2: Create the golden dataset
      • Step 3: Evaluate tool selection (trajectory)
      • Step 4: Evaluate final response
      • Step 5: Evaluate operational metrics
    • Multiturn Evaluation
      • Step 1: Single-turn evaluation steps
      • Step 2: Create tool chains
      • Step 3: Evaluate the final response
      • Step 4: Evaluate memory
    • Evaluating Long-Term Memory and Retrieval
    • Simulators: Generating the Golden Dataset
    • A Unified Taxonomy of Evaluation Metrics
    • Integrating Evaluation into the Development Lifecycle
  • Live Voice Agent Evaluation
    • Transcript-Based Evaluation: Turning Audio into Text
    • The Limits of Text: Losing the Human Element
    • Multimodal Native Evaluation: Listening to the Raw Audio
    • Evaluating Interruptions: The Chaos of Human Conversation
    • Closing the Loop: From Evaluation to Live Monitoring
    • The Evaluation Lifecycle: When to Test and How to Adapt
  • Conclusion
• 17. Managing Tools at Scale
  • The Landscape of Tools
  • The Tool Registry: A Single Source of Truth
  • Tool Selection Strategies: Generalist Versus Specialist
  • Distributed Tool Environments and the Centralized Tool Registry
  • Standardizing the Repository
  • Consolidating Tools with API Abstraction
  • Standardizing with Model Context Protocol
  • Tool Registry and Organizational Structure
    • Functional (Decentralized) Organization
    • Centralized Organization
    • Federated (Hybrid) Organization
    • How to Choose the Right Tool Management Model
  • Conclusion
• 18. Sharing and Governing Agents as Services and Reusable Components
  • The Agent Template Catalog: Standardization at the Source Code
    • The Cookie-Cutter Approach
    • Anatomy of an Agent Template: The Standardized Folder Structure
    • Implementing the Agent Template Catalog
    • From Catalog to Codebase: The Instantiation Workflow
    • Customizing the Blueprint
  • The Agent Registry: From Repository to Reusable Service
    • Distinguishing the Agent Catalog from the Registry
    • Building the Agent Registry: Beyond the Agent Card
      • The registry record pattern
      • Registry hygiene: Keeping the directory accurate
      • Agent registry interaction in an enterprise environment
  • The Agent Gateway: From Who Exists? to Who Has Access?
    • Defining Access Policies
    • Governance Models: Who Holds the Keys?
    • Implementing the Agent Gateway
    • Gateway Operations: Who Maintains the Guardrails?
    • Real-World Scenario: The Developers Journey
    • The Identity Gap: Looking Ahead to Authentication
  • The Complete Lifecycle: Putting It All Together
    • The Creation Lifecycle (AI Engineer)
    • The Governance Lifecycle (Admin and Owner)
    • The Usage Lifecycle (End User)
  • Conclusion
• 19. Security, Privacy, and Responsible AI
  • The Eight Layers of Defense: A Bottom-Up Approach
    • Layer 1: Infrastructure Security
      • Physical hardening: Guards, gates, and mantraps
      • Hardware integrity: Trusting the silicon
      • Compute isolation: Confidential computing and air gaps
    • Layer 2: Network Security
      • Isolation
      • Traffic control
      • Private model access and on-premises hosting
      • Advanced interconnectivity: VPC peering and cross-cloud connections
    • Layer 3: Data Security
      • Encryption: Who holds the keys?
      • Sanitization: Cleaning the input and output
      • Data governance: Lineage and retention
    • Layer 4: AI Application Security
      • The threat landscape: OWASP Top 10 for LLMs
      • The firewall for AI: Input and output guardrails
      • Model security and supply chain
      • Observability and the kill switch
    • Layer 5: AI Agent Security
      • Context security: Protecting the short-term memory
      • Tool security: Securing the hands
      • Framework security: Hardening the nervous system
    • Layer 6: Identity and Access Management (IAM)
      • The evolution of credentials: Why tokens win
      • The standard protocol: OAuth 2.0
        • The human experience: Three-legged OAuth (authorization code flow)
        • The agent experience: Two-legged OAuth (client credentials flow)
        • Two-legged versus three-legged: Code for getting the token
      • Advanced identity architecture
    • Layer 7: Logging and Monitoring
      • The standard: OpenTelemetry
      • The new metrics: Tokens, cost, latency, and resources
      • Chain of thought observability
      • From logs to action: Alerts and mitigation
    • Layer 8: Governance, Risk, and Compliance (GRC)
      • Third-party audits and certifications
      • Human-in-the-loop protocols
      • Combatting shadow AI
  • Responsible AI for Live Multimodal Agents
    • The Foundation: Text-Based Guardrails
      • Agent guardrails: The developers responsibility (Layer 5)
        • Callbacks: The first line of defense
        • Scaling security with plugins
      • Model guardrails: The administrators responsibility (Layer 4)
    • The Evolution: Multimodal Guardrails (The Gateway Pattern)
      • The latency myth
      • Securing live agents: The streaming challenge
      • The transformation strategy: Turning audio into text
      • The gatekeeper mechanism
      • The security gap: Reactive versus proactive approaches
        • Proactive input gating (the privacy buffer)
        • Parallel multimodal guardrails (true independence)
  • Solving the Identity Gap: User Identity Propagation
    • Approach 1: The Standard On-Behalf-Of (OBO) Flow
    • Approach 2: Agent Authorization Authority (Centralized Broker)
    • Approach 3: Strengthening the Agent Authorization Authority with Token Exchange (RFC 8693)
      • The multi-provider challenge
      • How incremental authorization works with multiple providers
      • The switching process
      • Why Approach 3 wins over Approach 2
    • Extending the Authority: Managed Agent Identities (Secretless Agents)
  • Conclusion
• 20. Deployment and Scaling the Integrated System
  • The CI/CD Foundation: From Code to Container
  • Deploying to Development: The Agentic Pipeline
  • From Staging to Production: Quality Gates and Automated Governance
    • The Staging and AI-on-AI Testing
    • Registration as Code
    • Continuous Monitoring and the Feedback Loop
  • Release Strategies for Production
  • The Infinity Loop: Orchestrating Multiple Lifecycles
    • Decoupling Teams and Architectures
    • Standardized Contracts: MCP and A2A
    • Versioning and Lifecycle Retention
  • Conclusion
• Index

pokaż cały spis treści

ukryj recenzje