Cost-Effective Data Pipelines

Spis treści ebooka

• Preface
  • Who This Book Is For
  • What You Will Learn
  • What This Book Is Not
  • Running Example
  • Conventions Used in This Book
  • Using Code Examples
  • OReilly Online Learning
  • How to Contact Us
  • Acknowledgments
• 1. Designing Compute for Data Pipelines
  • Understanding Availability of Cloud Compute
    • Outages
    • Capacity Limits
    • Account Limits
    • Infrastructure
  • Leveraging Different Purchasing Options in Pipeline Design
    • On Demand
    • Spot/Interruptible
    • Contractual Discounts
    • Contractual Discounts in the Real World: A Cautionary Tale
  • Requirements Gathering for Compute Design
    • Business Requirements
    • Architectural Requirements
    • Requirements-Gathering Example: HoD Batch Ingest
      • Data
      • Performance
      • Purchasing options
  • Benchmarking
    • Instance Family Identification
    • Cluster Sizing
    • Monitoring
      • Cluster resource utilization
      • Data processing engine introspection
  • Benchmarking Example
    • Undersized
    • Oversized
    • Right-Sized
  • Summary
  • Recommended Readings
• 2. Responding to Changes in Demand by Scaling Compute
  • Identifying Scaling Opportunities
    • Variation in Data Pipelines
    • Scaling Metrics
    • Pipeline Scaling Example
  • Designing for Scaling
  • Implementing Scaling Plans
    • Scaling Mechanics
    • Common Autoscaling Pitfalls
      • Scale-out threshold is too high
      • Flapping
      • Over-scaling
  • Autoscaling Example
  • Summary
  • Recommended Readings
• 3. Data Organization in the Cloud
  • Cloud Storage Costs
    • Storage at Rest
    • Egress
    • Data Access
  • Cloud Storage Organization
    • Storage Bucket Strategies
    • Lifecycle Configurations
  • File Structure Design
    • File Formats
    • Partitioning
    • Compaction
  • Summary
  • Recommended Readings
• 4. Economical Pipeline Fundamentals
  • Idempotency
    • Preventing Data Duplication
    • Tolerating Data Duplication
  • Checkpointing
  • Automatic Retries
    • Retry Considerations
    • Retry Levels in Data Pipelines
  • Data Validation
    • Validating Data Characteristics
    • Schemas
      • Creating schemas
      • Validating with schemas
      • Keeping schemas up to date
  • Summary
• 5. Setting Up Effective Development Environments
  • Environments
    • Software Environments
    • Data Environments
    • Data Pipeline Environments
    • Environment Planning
      • Design
      • Costs
      • Environment uptime
  • Local Development
    • Containers
      • Container lifecycle
      • Container composition
        • Running local code against production dependencies
        • Using environment variables
        • Sharing configurations
        • Consolidating common settings
    • Resource Dependency Reduction
    • Resource Cleanup
  • Summary
• 6. Software Development Strategies
  • Managing Different Coding Environments
    • Example: A Multimodal Pipeline
      • Notebooks
      • Web UIs
  • Example: How Code Becomes Difficult to Change
  • Modular Design
    • Single Responsibility
    • Dependency Inversion
      • Supporting multicloud
      • Plugging in other data sinks
      • Testing
    • Modular Design with DataFrames
  • Configurable Design
  • Summary
  • Recommended Readings
• 7. Unit Testing
  • The Role of Unit Testing in Data Pipelines
    • Unit Testing Overview
    • Example: Identifying Unit Testing Needs
  • Pipeline Areas to Unit-Test
    • Data Logic
    • Connections
    • Observability
    • Data Modification Processes
    • Cloud Components
  • Working with Dependencies
    • Interfaces
    • Data
  • Example: Unit Testing Plan
    • Identifying Components to Test
    • Identifying Dependencies
  • Summary
• 8. Mocks
  • Considerations for Replacing Dependencies
    • Placement
    • Dependency Stability
    • Complexity Versus Criticality
  • Mocking Generic Interfaces
    • Responses
    • Requests
    • Connectivity
  • Mocking Cloud Services
    • Building Your Own Mocks
    • Mocking with Moto
  • Testing with Databases
    • Test Database Example
    • Working with Test Databases
  • Summary
  • Further Exploration
    • More Moto Mocks
    • Mock Placement
• 9. Data for Testing
  • Working with Live Data
    • Benefits
    • Challenges
  • Working with Synthetic Data
    • Benefits
    • Challenges
    • Is Synthetic Data the Right Approach?
  • Manual Data Generation
  • Automated Data Generation
    • Synthetic Data Libraries
      • Customizing generated data
      • Distributing cases in test data
    • Schema-Driven Generation
      • Mapping data generation to schemas
      • Example: catching schema change impacts with CI tests
  • Property-Based Testing
  • Summary
• 10. Logging
  • Logging Costs
    • Impact of Scale
    • Impact of Cloud Storage Elasticity
  • Reducing Logging Costs
  • Effective Logging
  • Summary
• 11. Finding Your Way with Monitoring
  • Costs of Inadequate Monitoring
    • Getting Lost in the Woods
    • Navigation to the Rescue
      • Job metrics
      • Autoscaling events
      • Job runtime alerting
      • Error metrics
  • System Monitoring
    • Data Volume
    • Throughput
    • Consumer Lag
    • Worker Utilization
  • Resource Monitoring
    • Understanding the Bounds
    • Understanding Reliability Impacts
  • Pipeline Performance
    • Pipeline Stage Duration
    • Profiling
    • Errors to Watch Out For
      • Ingestion success and failure
      • Stage failures
      • Validation failures
      • Communication failures
      • Stage timeouts
  • Query Monitoring
  • Minimizing Monitoring Costs
  • Summary
  • Recommended Readings
• 12. Essential Takeaways
  • An Ounce of Prevention Is Worth a Pound of Cure
    • Reign In Compute Spend
    • Organize Your Resources
    • Design for Interruption
    • Build In Data Quality
  • Change Is the Only Constant
    • Design for Change
    • Monitor for Change
  • Parting Thoughts
• A. Preparing a Cloud Budget
  • Its All About the Details
    • Historical Data
    • Estimating for New Projects
    • Changes That Impact Costs
      • Data landscape
      • Load
      • Infrastructure
  • Creating a Budget
    • Budget Summary
    • Changes Between Previous and Next Budget Periods
    • Cost Breakdown
  • Communicating the Budget
  • Summary
• Index

pokaż cały spis treści

ukryj recenzje