Building Resilient Distributed Systems. Patterns and Practices for Stable Software Sam Newman

Dostępność produktu

Produkt został zweryfikowany pod kątem ułatwień dostępu więcej przeczytasz w deklaracji dostępności.

Spis treści książki

• Preface
  • Who This Book Is For
  • Prerequisite Knowledge
  • What You Will Learn
  • Navigating the Book
    • Part I: Foundations
    • Part II: Dealing with Failure
    • Part III: People, Process, and Culture
  • Writing Process and Use of AI
  • Conventions Used in This Book
  • OReilly Online Learning
  • How to Contact Us
  • Acknowledgments
• 1. What Is Resiliency?
  • Defining Resiliency
    • Technology
    • Social
  • Why Resilience Matters
  • What Is a Distributed System?
    • How Distributed Systems Can Fail (Us)
    • The Three Golden Rules of Distributed Systems
      • You cannot beam information instantaneously between two points
      • Sometimes you cant reach the thing you want to talk to
      • Resource pools are not infinite
    • Distributed Systems Arent Just Computers
  • Failure Is Inevitable
  • From Safety Management to Resilience Engineering
  • The Four Concepts for Resilience
    • Robustness
    • Rebound
    • Graceful Extensibility
    • Sustained Adaptability
  • How Resilient Do You Need to Be?
  • Summary
• I. Foundations
• 2. Fundamentals of Observability
  • Observability Versus Monitoring
  • The Three Pillars of Observability?
    • Metrics
    • Logs
    • Traces
    • Three Pillars Not Sufficient?
  • OpenTelemetry
    • Slicing and Dicing
  • The Golden Signals
    • Saturation
    • Latency
    • Error Rates
    • Traffic
    • Correlation Is Key
  • SLAs and SLOs and SLIs, Oh My!
    • Service Level Agreements
    • Service Level Objectives
    • Service Level Indicators
    • Error Budgets
  • Alerts
    • Some Problems Are Worse Than Others
    • Alert Fatigue
    • Toward Better Alerting
    • Separation of Responsibility Considered Harmful?
  • Testing Observability
  • Observability to the Rescue of AI?
  • Summary
• 3. Timeouts
  • The Problem with Time
  • Pattern: Timeout
    • How It Works
    • When to Use It
  • Setting a Timeout
  • Analyzing Existing Response Times
    • Tail Latencies
    • Tight Latency Bounds
  • How Many Requests Can You Handle?
  • Using User Expectations
  • Fault Injection and Testing Timeouts
  • Timeouts and Call Chains
  • Pattern: Timeout Budget Propagation
    • How It Works
    • When to Use It
  • Other Timeout Considerations
  • Case Study: AdvertCorp
    • Overload
    • Customer-Driven Denial of Service
    • Page Load Times
    • Missing Timeouts
    • A Confluence of Events
  • Summary
• 4. Retries and Idempotency
  • Pattern: Retry Remote Operation
    • How It Works
    • When to Use It
  • Should You Always Retry?
    • A Better Type of Error
  • How Many Retries Are Appropriate?
    • Fixed Retry Count
    • Dynamic Retry Limit
  • Delays Between Retries
    • Exponential Backoff
    • Jitter
  • Is Retrying Safe?
  • Pattern: Idempotent Operation
    • How It Works
      • How to respond to repeated attempts
      • What happens when state changes between attempts
      • Handling concurrent duplicate requests
    • When to Use It
  • Making Operations Idempotent
    • Pattern: Client Idempotency Key
      • How it works
      • When to use it
    • Pattern: Server-Side Request Fingerprinting
      • How it works
      • Where to use it
    • Using Client Idempotency Keys Together with Server-Side Fingerprinting
  • Testing Retries and Idempotency
  • Further Reading
  • Summary
• 5. Thundering Herds
  • What Is a Thundering Herd?
  • Load Balancer Domino
    • Example: Pet Service
    • Mitigations
      • Overprovision
      • Reduce work
      • Use desired state management
      • Reduce failure rate
  • Cache Collapse
    • Example: ShopCorp Catalog Cache Failure
    • Mitigations
      • Cache in service instances
      • Use a more resilient cache
      • Improve recovery time
      • Disable the service
  • Mismatched Capacities
    • Example: BankCo Pricing Grid
    • Scaling Up
    • A Lucky Break
    • Mitigations
      • Better load testing
      • Gradual ramp-up
      • Rate limiting
      • Improve communication
  • Roaring Success
    • Example: Twitter
    • Example: Healthcare.gov
    • Mitigations
      • More realistic load testing
      • Scaling your way out of trouble
      • Limiting sign-ups
      • Better alignment
  • Distributed Denial of Service Attacks
    • Examples
    • Mitigations
      • Keeping things patched
      • Scale up
      • Multiple networks
      • Degradation functionality and blackhole routing
      • Rate limiting
      • Web application firewalls and content delivery networks
  • Summary
• 6. Rate Limiting
  • Ways to Handle Having Too Much Work
    • Just Let It Fall Over
    • Throw Away Some of the Work (Load Shedding)
    • Reduce the Work Being Sent (Backpressure)
    • Queue Up the Work
    • Dynamically Provision More Resources
  • Pattern: Load Shedding
    • How It Works
      • Choosing what to track
      • Setting thresholds
      • Communicating load shedding to clients
      • Setting where to trigger
      • Handling inconsistent load from work
      • Prioritizing work
    • When to Use It
  • Backpressure
    • Client-Only Backpressure
    • Accord-Based Backpressure
  • Pattern: Circuit Breaker
    • How It Works
      • Case study: AdvertCorp
      • For client-only or accord-based backpressure
      • Issues with circuit breakers
        • Too late to the party
        • Boom and bust
        • Partial failure
    • When to Use It
  • Reducing Versus Stopping Traffic
    • Boom and Bust Cycle
    • A Delicate Balance
  • Pattern: Leaky Bucket Rate Limiting
    • How It Works
      • Challenges
      • Variations
      • Existing implementations
    • When to Use It
  • Simulating the Use of a Circuit Breaker and a Token/Leaky Bucket
  • Config, Not Code
  • Stopping the Load in the First Place
  • Summary
• 7. Resiliency and AI
  • Using AI for Software Delivery and Operations
    • Software Development
    • Understanding Existing Code
    • Interacting with Powerful Tools
    • Automated Anomaly Detection
    • Incident Response
  • Resiliency of AI-Powered Software
    • Model Provider Resiliency
    • Model Changes
    • Observability
  • AI Challenges
    • Nondeterminism
    • Correctness
    • Infrastructure Availability
    • Model Changes
    • Vendor Instability
    • Cost
    • Yet Another Financial Bubble?
  • How to Handle Uncertainty
    • Multimodel, Multivendor
    • Cost Transparency
    • More Frequent Vendor Analysis
  • Summary
• II. Dealing with Failure
• 8. Handling Machine Failure
  • Service Discovery
    • Pattern: Hardcoded IP Resolution
      • How it works
      • When to use it
    • Pattern: Dynamic Service Discovery
      • How it works
        • Domain Name System (DNS)
        • Consul, etcd, et al
      • When to use it
  • Load Balancing
    • Pattern: Server-Side Load Balancing
      • How it works
        • Managing the worker pool
        • Distributing load
        • Fault tolerance
      • When to use it
    • Pattern: Client-Side Load Balancing
      • How it works
        • Finding server endpoints
        • Distributing load
        • Using a service mesh
      • When to use it
  • Pattern: Desired State Management
    • How It Works
    • When to Use It
  • Stateful Versus Stateless Processing
    • Stateless Processing
    • Stateful Processing
      • Work routing
      • Handling instance failure
      • General guidance
  • Testing Machine Failure
  • Summary
• 9. Message Brokers
  • Message Broker Basics
    • Queues Versus Topics
    • Point-to-Point Versus Broadcast
    • Consumer Groups
  • Why Use a Message Broker?
    • Temporal Decoupling
    • Delivery Guarantees
    • Scaling
  • How Brokers Work (in a Nutshell)
    • Maintaining Durability
    • Maintaining Availability
    • Durability and High Availability Not Needed?
    • Fleeting Messages
  • Types of Delivery Guarantees
    • At Least Once
    • At Most Once
    • Exactly Once
    • Guaranteed Delivery Versus Guaranteed Processing
  • Useful Patterns
    • Pattern: Transactional Outbox
      • How it works
        • Failure in message sending
        • Retaining sent messages
      • When to use it
    • Pattern: Consumer-Managed Inbox
      • How it works
      • When to use it
    • Pattern: Producer Heartbeat Message
      • How it works
      • When to use it
    • Pattern: Dead Letter Channel
      • How it works
      • When to use it
  • Self-Hosted Versus Fully Managed Brokers
  • Summary
• 10. Disaster Recovery
  • Cloud Region, Data Center, or Site?
  • How Good Does Your DR Need to Be?
    • Recovery Point Objective
    • Recovery Time Objective
    • You Cant Have Everything
    • DR Objectives and SLOs
  • Disaster Recovery Patterns
    • Pattern: Backup and Restore
      • How it works
      • When to use it
    • Pattern: Pilot Light Failover
      • How it works
      • When to use it
    • Pattern: Warm Failover
      • How it works
      • When to use it
    • Pattern: Multisite Active-Active
      • How it works
      • When to use it
  • Infrastructure Availability
  • Multisite Data Replication
    • Impact of Latency
    • Active-Passive
    • Active-Active
    • Implementation Options
      • Relational databases
      • Vector databases
      • Cassandra
      • Public cloud global tables
      • Message brokers
  • Further Reading
  • Testing DR
  • Summary
• 11. Progressive Collapse
  • What Is Progressive Collapse?
  • Real-World Examples
  • In Distributed Systems
  • Case Study: AWS US-EAST-1 Outage, October 2025
    • Context
    • What Happened
    • Impact
  • Mitigating Progressive Collapse
    • Reduce Hazards
    • Strengthen Individual Components
    • Reduce Interconnection
  • Patterns for Avoiding Progressive Collapse
  • Pattern: Bulkhead
    • How It Works
    • When to Use It
  • Pattern: Graceful Degradation
    • How It Works
    • When to Use It
  • Pattern: Distribute Across Failure Planes
    • How It Works
      • Creating and managing the failure planes
      • Distributing work
      • Synchronizing data
      • Increasing isolation
    • When to Use It
  • Pattern: Cell-Based Architecture
    • How It Works
      • Cells
      • Cell router
      • Cell gateway
      • Cell-local routing
      • Control plane
      • Cell resiliency
      • Case studies
    • When to Use It
  • Pattern: Critical Standby System
    • How It Works
    • When to Use It
  • Summary
• 12. Using Formal Theorems to Understand Trade-Offs
  • CAP Theorem
    • Pick TwoSort Of
    • Example: Online Record Shop
      • Sell the item!
      • Dont sell the item
      • In the real world
    • Example: Concert Ticketing
      • Sell the ticket!
      • Dont sell the ticket
      • In the real world
    • Its Not All or Nothing
  • PACELC
    • Latency Versus Consistency
    • Record Shop and Ticketing Examples Revisited
    • Latency Versus Consistency and Resiliency
  • Littles Law
    • A (Contrived) Real-World Example
    • Littles Law and Ticketing Systems
    • Limitations of Littles Law
  • Summary
• III. People, Process, and Culture
• 13. The Sociotechnical Systems
  • Understanding Sociotechnical Systems
    • The Hexagonal Model for Sociotechnical Systems
    • People
    • Culture
    • Goals
    • Technology and Infrastructure
    • Processes
    • External Factors
  • The Therac-25 Radiation Machine
  • Summary
• 14. Incident Management and Post-Incident Response
  • What Do You Mean, Incident?
  • Not Linear
  • Making Incident Response Easier
    • Right People, Right Place
    • Clear Lines of Ownership
    • Make Systems Easily Observable
    • Implement Fast Rollback
  • Incident Roles
  • Key Activities in Incident Response
    • Communication
    • Delegation
    • Escalation
    • Coordination
    • Saturation
  • Post-Incident Recovery
  • Post-Incident Review
    • Do You Need One?
    • Who Is Involved?
    • Gather Data
    • Learning, Not Blame
    • Root Cause(s)?
      • The five whys
      • Thinking broadly
      • Visualizing root causes
    • Action-Oriented
    • Share a Draft
    • Share the Final Results
  • Summary
• 15. Constant Learning and Adaptation
  • Psychological Safety
    • What Is Psychological Safety?
    • Why Psychological Safety Is Vital
    • Its Not Universal
  • AI and the Rise of Cognitive Debt
  • Build Learning into the Day Job
  • Collaborative Working
  • Learning from Incidents
  • Naive, Purposeful, and Deliberate Practice
  • Game Days
  • Online Learning Resources
  • Architectural Katas
  • Chaos Engineering
  • Summary
• 16. Conclusion
  • Distributed Systems
  • Resilience Engineering
  • Sociotechnical Systems
  • Resilience and AI
  • Patterns, Practices, and Concepts
    • Observability
    • Timeouts
    • Retries and Idempotency
    • Thundering Herds
    • Rate Limiting
    • Machine Failure
    • Message Brokers
    • Disaster Recovery
    • Progressive Collapse
    • Formal Methods
    • Incident Management
    • Constant Learning
  • Final Thoughts
• A. Pattern Index
• Glossary
• Bibliography
• Index

pokaż cały spis treści

ukryj recenzje