Fundamentals of Data Observability

Spis treści ebooka

• Preface
  • Overview of the Book
  • Who Should Read This Book
  • Conventions Used in This Book
  • Using Code Examples
  • OReilly Online Learning
  • How to Contact Us
  • Acknowledgments
• I. Introducing Data Observability
• 1. Introducing Data Observability
  • Scaling Data Teams
    • Challenges of Scaling Data Teams
    • Segregated Roles and Responsibilities and Organizational Complexity
    • Anatomy of Data Issues and Consequences
    • Impact of Data Issues on Data Team Dynamics
    • Scaling AI Roadblocks
  • Challenges with Current Data Management Practices
    • Effects of Data Governance at Scale
    • Data Observability to the Rescue
    • The Areas of Observability
  • How Data Teams Can Leverage Data Observability Now
    • Low Latency Data Issues Detection
    • Efficient Data Issues Troubleshooting
    • Preventing Data Issues
    • Decentralized Data Quality Management
    • Complementing Existing Data Governance Capabilities
    • The Future and Beyond
  • Conclusion
• 2. Components of Data Observability
  • Channels of Data Observability Information
    • Logs
    • Traces
    • Metrics
  • Observations Model
    • Physical Space
    • Server
    • User
    • Static Space
      • Data source
      • Schema
      • Lineage
      • Application
      • Application repository
      • Application version
    • Dynamic Space
      • Application execution
      • Lineage execution
      • Data metrics
  • Expectations
    • Rules
      • Explicit rules
      • Assisted rules
      • Connections with SLAs/SLOs
    • Automatic Anomaly Detection
    • Prevent Garbage In, Garbage Out
      • Pre- and post-conditions
      • Circuit breaker
  • Conclusion
• 3. Roles of Data Observability in a Data Organization
  • Data Architecture
    • Where Does Data Observability Fit in a Data Architecture?
      • The data-observable system
      • The data observability platform
    • Data Architecture with Data Observability
  • How Data Observability Helps with Data Engineering Undercurrents
    • Security
    • Data Management
      • Data governance
        • Discoverability
        • Accountability
        • Data quality
        • Master data management and data modeling
        • Data Integration and Interoperability
        • Data lifecycle management
        • Ethics and privacy
      • DataOps
        • Automation
        • Observability
        • Incident response
      • Orchestration
      • Software engineering
  • Support for Data Meshs Data as Products
  • Conclusion
• II. Implementing Data Observability
• 4. Generate Data Observations
  • At the Source
  • Generating Data Observations at the Source
  • Low-Level API in Python
    • Description of the Data Pipeline
    • Definition of the Status of the Data Pipeline
    • Data Observations for the Data Pipeline
    • Generate Contextual Data Observations
    • Generate Data-Related Observations
    • Generate Lineage-Related Data Observations
    • Wrap-Up: The Data-Observable Data Pipeline
    • Using Data Observations to Address Failures of the Data Pipeline
  • Conclusion
• 5. Automate the Generation of Data Observations
  • Abstraction Strategies
    • Event Listeners
    • Aspect-Oriented Programming
      • Code enrichment
      • Runtime manipulation
        • Monkey patching
        • Bytecode instrumentation
      • Code generation
        • AST manipulation
        • Annotations, decorators, and attributes
        • Macros
  • High-Level Applications
    • No-Code Applications
    • Low-Code Applications
      • Code generators
      • Workflows
  • Differences Among Monitoring Alternatives
  • Conclusion
• 6. Implementing Expectations
  • Introducing Expectations
    • Shift-Left Data Quality
    • Corner Cases Discovery
    • Lifting Service Level Indicators
    • Using Data Profilers
  • Maintaining Expectations
  • Overarching Practices
    • Fail Fast and Fail Safe
    • Simplify Tests and Extend CI/CD
  • Conclusion
• III. Data Observability in Action
• 7. Integrating Data Observability in Your Data Stack
  • Ingestion Stage
    • Ingestion Stage Data Observability Recipes
    • Airbyte Agent
  • Transformation
    • Transformation Stage Data Observability Recipes
    • Apache Spark
      • Spline
      • Going beyond the compliance use case
    • dbt Agent
  • Serving
    • Recipes
    • BigQuery in Python
    • Orchestrated SQL with Airflow
  • Analytics
    • Machine Learning Recipes
    • Business Intelligence Recipes
      • Databricks
      • Superset
  • Conclusion
• 8. Making Opaque Systems Translucent
  • Data Translucence
  • Opaque Systems
    • SaaS
    • Dont Touch It; It (Kinda) Works
    • Inherited Systems
  • Strategies for Data Translucence
    • Strategies
      • Data governance APIs
      • Log analytics
      • Code and binary analyzers
      • Data scanning
    • The Data Observability Connector
    • Example: Building a dbt Data Observability Connector (SaaS)
  • Conclusion
• Afterword: Future Observations
  • Unification of Processing
  • Generative Milestones
  • Trustable Expanded Creativity
  • Conclusion
• Index

pokaż cały spis treści

ukryj recenzje