Data Engineering for Multimodal AI. Architecting Scalable Systems for Next-Generation AI Applications Vasundra Srinivasan

Dostępność produktu

Produkt został zweryfikowany pod kątem ułatwień dostępu więcej przeczytasz w deklaracji dostępności.

Spis treści książki

• Preface
  • Why I Wrote This Book
  • How This Book Was Built
  • Who Should Read This Book
  • What You Will Get Out of This Book
    • Navigating This Book
  • Conventions Used in This Book
  • Using Code Examples
  • OReilly Online Learning
  • How to Contact Us
  • Acknowledgments
  • A Note Before You Begin
• 1. The Multimodal AI Data Engineering Landscape
  • The Engineering Context of Multimodal AI Systems
  • Data Architecture Components in a Multimodal AI System
  • Data Contracts: The Foundation of Trust
  • Building the Semantic Layer of a Multimodal AI System
    • From Principle to Practice: Granites Layered Architecture
    • Entity Resolution and Why Its Hard
    • Building the Semantic Layer: A Step-by-Step Summary
  • Key Challenges in Building Multimodal AI Pipelines
    • Integration and Synchronization
    • Processing at Scale
    • Operational Resilience
  • The Stages of the Multimodal AI Data Lifecycle
    • Stage 1: Data Ingestion
    • Stage 2: Staging and Organization
    • Stage 3: Core Processing
    • Stage 4: Feature Engineering
    • Stage 5: Model Serving
    • Stages 68: Continuous Learning and Deployment
    • Stage 9: Monitoring and Analysis
  • Playbook: Building a Minimal Semantic Layer Pipeline
    • Step 1: Data Exploration
      • ChestMNIST image data
      • Synthetic clinical notes
      • Vital signs time series
    • Step 2: Initial Setup and Data Preparation
    • Step 3: Data Ingestion Pipeline
    • Step 4: Starting the Semantic API Server
  • Takeaways
• 2. Data Architectures for Multimodal AI Systems
  • Core Patterns for Multimodal Data Architectures
    • Vector-Native Storage
    • Composable Data Lakehouse Architecture
    • Complementary Strategies: Hybrid Storage and Real-Time Processing
  • Embeddings and Retrieval in Multimodal Data Architectures
  • Containerization and Orchestration for Multimodal AI
  • Reliability and Scale in Multimodal AI Data Architecture
    • Streaming Ingestion
    • Scaling Strategies
    • Production Patterns for Reliability and Scale
    • Monitoring and Observability
  • Data Fabric and Mesh Architectures for Multimodal Systems
    • Data Fabric Fundamentals
    • Data Mesh Fundamentals
  • Playbook: Building a Multimodal Feature Store for StreamBuy
    • Step 0: Data Exploration
    • Step 1: Project Setup and Architecture
    • Step 2: Data Models
    • Step 3: Feature Encoders
    • Step 4: Database Clients
    • Step 5: Background Vector Processing
    • Step 6: API Layer
    • Step 7: Running Tests
    • Step 8: Running the Full Stack
    • Extending This Pipeline
  • Takeaways
• 3. Platforms and Patterns for Multimodal AI Data Engineering
  • Setting the Context: Multimodal Context Intelligence
  • Context Intelligence in Action: Revaults Onboarding Flow
  • Platform Foundations for Multimodal Context Infrastructure
    • Managed Multimodal AI Platforms
    • Composable Multimodal Context
    • Real-Time Streaming Infrastructure for Multimodal Context
    • Choosing Between Archetypes: A Cost and Complexity Framework
  • Core System Patterns for Multimodal Context Engineering
    • Multimodal Representation Layer
    • Temporal State Assembly
    • Selective Context Retrieval
    • Cross-Cutting Concerns: Observability, Embeddings, and Agentic Orchestration and Context Flow
  • Playbook: Building a Serverless Document Retrieval Pipeline
    • Stage 1: Document Parsing and Representation
    • Stage 2: Embedding Generation
    • Stage 3: Vector Indexing
    • Stage 4: Selective Retrieval
    • Scaling for Production
    • Playbook Conclusion
  • Takeaways
• 4. Ingestion and Transformation for Multimodal AI Systems
  • Why Is Multimodal Ingestion Fundamentally Harder?
  • Multimodal Ingestion Pipelines
    • Traditional Approach
    • A Hybrid AI Approach
    • The Agentic AI Approach
  • Operationalizing ELT for Multimodal AI
    • ELT Versus ETL
    • Hybrid ETL and ELT Architectures
    • Privileging ELT over ETL at ByteEats
      • Decision 1: Where and how to ingest raw data
      • Decision 2: When to apply AI-driven transformations
      • Decision 3: Where to join and fuse modalities
      • Decision 4: What triggers actions in the system
      • Decision 5: Designing for continuous feedback
      • Decision 6: Governing schema evolution across modalities
    • Balancing Cost, Latency, and Fidelity
  • Validation, Transformation, Observability, and Annotation Across Modalities
    • What Happens When Validation Fails
    • From Validation to Representation
    • Observability: Tracing Signals Through the Pipeline
    • Annotation: Where Multimodal Systems Mature or Collapse
    • Ingestion as Context Engineering
  • Playbook: Building a Core Multimodal AI Data Operations Pipeline
    • Step 1: Order Ingestion
    • Step 2: Robot Prep Trigger
    • Step 3: Feature Extraction from Videos to Completion Trigger
    • Step 4: QA Testing
    • Step 5: Vector Embeddings
    • Step 6: Similarity Search
  • Takeaways
• 5. Feature Engineering and Management for Multimodal Data
  • Feature Engineering Techniques for Multimodal Data
    • A Modality-Aware Feature-Engineering Pipeline
    • Preprocessing as Feature Engineering
    • Extracting Features Across Modalities
      • Text data
      • Image data
      • Audio data
      • Video data
      • Spatial data (lidar)
      • Temporal alignment
    • Feature-Engineering Core Techniques
  • Feature Selection and Fusion in Multimodal Contexts
    • Why Fusion Is Not a Join
    • Fusion Architectures
      • Encoder-decoder fusion
      • Attention-based fusion
      • Graph-based fusion
      • Generative fusion
      • Constraint-based fusion
    • Cross-Cutting Considerations in Fusion
  • Implementing and Managing Cloud-Based Multimodal Feature Stores
    • Why Feature Stores Are Needed
    • Four Patterns for Feature Store Architecture
      • Dual-layer pattern
      • Vector-augmented hybrid pattern
      • Metadata-governed feature mesh
      • Graph-integrated feature store
    • Feature Store as a Service
    • Feature Store Implementation: BayGos Two Approaches
  • Playbook: Building an End-to-End Multimodal Feature-Engineering Pipeline
    • Getting Familiar with the Dataset
    • Step 1: Verify Data Structure
    • Step 2: Understand the Main Pipeline Code
    • Step 3: Run the Multimodal Feature Pipeline
    • Step 4: Examine the Generated Outputs
    • Step 5: Understand the Generated Outputs
  • Takeaways
• 6. Data Storage and Retrieval for Multimodal AI
  • Storage Solutions for Multimodal ML Workloads
    • Infrastructure Layer
    • Data Intelligence Layer
    • AI Service Layer
    • Application Layer
  • Databases for Cross-Modal Applications
    • Vector Databases
      • Core concepts of vector databases
      • Design decisions for vector databases
    • Graph Databases and Knowledge Graphs
      • Design decisions for graph databases
      • Graph database tools
    • Hybrid Storage and Retrieval
      • How hybrid storage and retrieval systems work
      • Hybrid retrieval tools
  • Caching Strategies and Inference on the Edge
    • Caching Zones
    • Static Caching
    • Dynamic Caching
    • A Comparison of Static and Dynamic Caching
    • Caching in Action: A Field Scenario
    • Caching Design Decisions
  • Playbook: Engineering Graph-Augmented Retrieval and Edge-Aware Caching for Multimodal AI
    • Sample Data Exploration
    • Overview
    • Step 1: Environment Setup and Data Preparation
    • Step 2: Embedding Generation
    • Step 3: Graph Construction
    • Step 4: Hybrid Retrieval Test
    • Step 5: Edge-Aware Cache Test
    • Step 6: Full System Test
    • Step 7: Streamlit UI
    • Playbook Conclusion
  • Takeaways
• 7. Streaming and Adaptive Pipelines for Multimodal AI
  • Designing Streaming Pipelines for Multimodal Inference
    • Event-Time Alignment
    • From Lambda to Event Driven: An Architectural Shift
    • The Reference Pipeline
    • Modality-Specific Preprocessing
    • Windowed Fusion
    • Inference Routing
    • Back-Pressure Control
    • Fanning Out, Sinks, and System Outputs
    • Tuning the Pipeline for Production
  • Adaptive Multimodal AI Systems
    • Why Does Adaptation Even Matter?
    • Drift Types, Detection, and Engineering Response
      • Input drift
      • Cross-modal drift
      • Latency-induced drift
      • Semantic drift
      • Contextual drift
      • Layered monitoring across the pipeline DAG
      • Feedback loops and triggering retraining
      • Fine-grained, modality-aware adaptation
  • Edge Computing for Multimodal AI Data Processing
    • Partitioning Logic: What Runs Where and Why
      • The edge
      • The cloud
      • Key considerations
    • Edge-Centric Pipeline Architecture Walkthrough
      • Step 1: Edge preprocessing
      • Step 2: Edge-to-cloud transfer
      • Step 3: Fusion and window join
      • Step 4: Inference router
      • Step 5: Fan-out and observability
      • Step 6: Feedback and adaptive loops
    • Federated Learning
    • CI/CD and Rollback for Edge-Deployed Models
  • Playbook: Building a Real-Time Multimodal Inference Router
    • Step 1: Data Exploration, Preprocessing, and Normalization
    • Step 2: Stream Fusion and Inference Routing
      • Data fusion
      • Inference routing
    • Step 3: Inference Serialization and Dispatch
    • Step 4: DAG Orchestration and Monitoring
• 8. Applying MLOps for Multimodal AI Applications
  • An End-to-End MLOps Lifecycle
  • Continuous Integration and Continuous Delivery for Multimodal AI
  • Data Versioning and Experiment Tracking for Multimodal AI
  • Automated Retraining Pipelines for Multimodal AI
  • Continuous Data Integration for Multimodal Systems
    • Streaming and Batch Ingestion
    • The Feature Store Interface
  • Testing and Validation for Multimodal Models
    • Validation Architecture
    • Observability for Multimodal Systems
  • Collaborative Dynamics and Best Practices for Multimodal AI Teams
  • From Data Foundations to Feedback: Eight Lessons in Operating Multimodal ML Systems
  • Playbook: Implementing an End-to-End MLOps Pipeline
    • Data Exploration
    • Step 1: Data Ingestion
    • Step 2: Feature Extraction
      • IMU and INS
      • IMU or INS: A critical choice
    • Step 3: Model Training
    • Step 4: Model Evaluation and Prediction
    • Step 5: Model Serving and Deployment
    • Step 6: Setting Up Model Monitoring
      • DAG-based orchestration
      • Running and monitoring the pipeline
      • Making the pipeline resilient and reproducible
    • Playbook Conclusion
  • Takeaways
• 9. Scaling Cloud-Native Multimodal AI Data Pipelines
  • Distributed Data Processing for Multimodal Data
    • Stream-Processing Engines: Apache Beam on Dataflow
    • Batch Feature-Engineering Frameworks: Spark on Kubernetes
    • Distributed AI Runtimes: Ray + KubeRay + Inference Engines
    • High-Throughput Data Loaders: The Training Input Layer
    • Fault Tolerance and Recovery
    • Distributed DAGs and Bottlenecks in Multimodal Pipelines
    • Unifying the Stack: A Composable Runtime for Multimodal AI
  • Auto-Scaling Strategies for Multimodal Workloads
    • Modality-Specific Scaling Profiles
      • Radiology imaging: Scheduled batch + GPU burst
      • ICU telemetry: Streaming in real time + horizontal elasticity
      • Clinician notes: Latency-tolerant, memory-constrained
      • Generative inference: Latency-driven scaling
    • Cross-Modal Dependency Orchestration
      • Stream processing + batch framework: Fused readiness coordination
      • Training + serving: Model lifecycle synchronization
      • Visual-text fusion: Inference dependency guardrails
  • Cost Optimization for Cloud-Based Multimodal AI Infrastructure
    • Preemptible Training and Opportunistic Compute
    • Batching and Inference Tiering
    • Dynamic Model Parking with Checkpoint Streaming
    • Storage-Aware Cost Control
    • Token-Efficient Inference
  • Performance Tuning and Optimization Techniques
    • Memory-Tiered Sharding in GPU Workloads
      • Memory-tiered sharding
      • Shard-aware scheduling
    • Tuning the Inference Engine for Latency
      • Adaptive batching
      • Queue-aware scheduling
      • Inference engine selection
    • Batch Executor Optimization in Spark
      • Semantic-aware repartitioning
      • Shuffle avoidance via broadcast joins
      • Speculative execution and tail rebalancing
    • Data Loader Tuning for Training Throughput
  • Playbook: Deploying a Serverless Multimodal Inference Service with On-Demand GPU Scaling
    • Understanding the Sample Data
    • Step 1: Prerequisites and Setup
    • Step 2: Deploying the Pipeline
    • Step 3: Running a Clinical Workload Simulation
      • Key observations
      • From reactive elasticity to predictive scaling
    • Step 4: Running the Predictive Autoscaler Demo
  • Takeaways
• 10. Model Deployment and Serving for Multimodal AI
  • Deploying Multimodal Models for Real-Time Inference
    • The Inference Triangle
    • The Inference Framework: Modal Contracts
    • A Production Example at Eighteen Labs
    • Scene Text Recognition in Multimodal Pipelines
  • Handling Multi-Stage Inference Pipelines
    • Engineering Challenges
    • Eighteen Labs: Data Center Operations
  • Monitoring and Managing Deployed Multimodal Models
    • Monitoring Architecture
    • Detecting Drift
    • Feedback Loops
    • Evaluation Framework and Retraining
    • Causal Attribution and Adversarial Resilience
  • Scalable Multimodal Serving
    • Concurrency and Synchronization
    • Batching Across Modalities
    • Model Optimization for Deployment
    • Hardware and Accelerator Scheduling
    • Edge-Cloud Inference Topology
    • Tiered Routing
    • Cost and Sustainability
  • Continuous Deployment and Lifecycle Orchestration for Multimodal Models
    • Version Control for Composite Models
    • Canary Deployment
    • CI/CD Orchestration
    • Rollback
    • Large Behavior Models
    • Dual-Path Architecture
    • Sequence-Aware CI/CD
    • Governance
  • Playbook: Building and Managing a Multimodal Inference Service
    • Understanding the Data: RLBench
    • Step 1: Validating the Model Configuration
    • Step 2: Implementing the Model
    • Step 3: Web Service Implementation
    • Step 4: Canary Deployment and Containerization
  • Takeaways
• 11. Governance, Security, and Responsible Practices for Multimodal AI
  • Foundations of Governance in Multimodal AI
    • Lineage and Provenance Tracking
    • Metadata Catalogs
    • Compliance Across Modalities
    • Data Sovereignty
    • Governance Architecture
    • Example: Warehouse Governance in Practice
  • Securing Multimodal AI Pipelines
    • Protecting Data Across the Pipeline
      • Ingestion
      • Processing
      • Inference
    • Privacy-Preserving Machine Learning
    • Federated and Distributed Learning
    • Adversarial Threats to Multimodal LLMs
    • Example: Data Center Security in Practice
  • Responsible and Transparent AI
    • Bias Detection Across Modalities
    • Explainability for Multimodal Systems
    • Adversarial Robustness
    • Production Observability
    • Example: Responsible AI in Practice
  • Playbook: Operationalizing Governance, Security, and Responsible AI for Multimodal Pipelines
    • What Youll Build
    • The Datasets
    • Step 1: Design Pipeline Architecture
    • Step 2: Data Ingestion Module
    • Step 3: Security Module
    • Step 4: Inference Module
    • Step 5: Responsible AI Module
    • Step 6: Running the End-to-End Pipeline
    • Cost Governance
    • Playbook Conclusion
  • Takeaways
• 12. Putting It All Together: Building End-to-End Multimodal AI Pipelines
  • Designing and Architecting Complete Multimodal AI Systems
  • StreamKit in Practice: High-Volume Incident Surge
    • Intelligent Handling of a Product Incident
      • A customer interacts with the contact center
      • Data flows into the fusion bus
      • The agent makes inferences
    • Key Considerations
  • Integration Patterns for Multimodal Data Pipelines
    • Pattern 1: On-the-Wire Contracts
    • Pattern 2: Transport Selection
    • Pattern 3: Back-Pressure and Fault Handling
    • Pattern 4: Scaling and Elasticity
    • Pattern 5: Multimodal RAG
  • Workflow Orchestration for Multimodal AI
    • Orchestration Responsibilities
    • Implicit Versus Explicit Orchestration
    • Edge Orchestration in Action
    • Schema Validation at the Edge
    • Adaptive Resource Control
    • The Edge-Cloud Handoff
  • Monitoring and Troubleshooting End-to-End Multimodal Pipelines
    • Signal Granularity: Instrument Each Lane
    • Cross-Modal Correlation: Detect Cascading Failures
    • Actionable Diagnostics: From Alert to Fix
    • StreamKits Three Observability Layers
    • Automated Runbooks for Common Failures
    • Benchmarking Production Readiness
  • Evolving and Scaling Production Multimodal Pipelines
    • Shadow Mode and Safe Promotion
    • Scaling Across Sites: Session-Aware Fusion
    • Dual-Support Deprecation: The Sentiment Analyzer Upgrade
    • Async Upgrade Bridge: Bridging Embedding Incompatibilities
    • Resilient Operation Under Modality Failure
    • Observability Binds the Whole Picture
    • Evaluation and Feedback Loops
  • Playbook: Building a Multimodal Root Cause Analysis Pipeline
    • Data Exploration
    • What We Will Build
    • Pipeline Architecture
    • Running the Playbook
    • What to Look For in the Output
    • Extending the Playbook
  • Takeaways
• A. References and Datasets
  • Chapter 1. The Multimodal AI Data Engineering Landscape
  • Chapter 2. Data Architectures for Multimodal AI Systems
  • Chapter 3. Platforms and Patterns for Multimodal AI Data Engineering
  • Chapter 4. Ingestion and Transformation for Multimodal AI Systems
  • Chapter 5. Feature Engineering and Management for Multimodal Data
    • Dataset
  • Chapter 6. Data Storage and Retrieval for Multimodal AI
  • Chapter 7. Streaming and Adaptive Pipelines for Multimodal AI
  • Chapter 8. Applying MLOps for Multimodal AI Applications
  • Chapter 9. Scaling Cloud-Native Multimodal AI Data Pipelines
  • Chapter 10. Model Deployment and Serving for Multimodal AI
    • Dataset
  • Chapter 11. Governance, Security, and Responsible Practices for Multimodal AI
    • Datasets
  • Chapter 12. Putting It All Together: Building End-to-End Multimodal AI Pipelines
    • Dataset
• Index

pokaż cały spis treści

ukryj recenzje