Kafka: The Definitive Guide. Real-Time Data and Stream Processing at Scale (ebook)(audiobook)(audiobook)

Spis treści

• Foreword
• Preface
  • Who Should Read This Book
  • Conventions Used in This Book
  • Using Code Examples
  • OReilly Safari
  • How to Contact Us
  • Acknowledgments
• 1. Meet Kafka
  • Publish/Subscribe Messaging
    • How It Starts
    • Individual Queue Systems
  • Enter Kafka
    • Messages and Batches
    • Schemas
    • Topics and Partitions
    • Producers and Consumers
    • Brokers and Clusters
    • Multiple Clusters
  • Why Kafka?
    • Multiple Producers
    • Multiple Consumers
    • Disk-Based Retention
    • Scalable
    • High Performance
  • The Data Ecosystem
    • Use Cases
      • Activity tracking
      • Messaging
      • Metrics and logging
      • Commit log
      • Stream processing
  • Kafkas Origin
    • LinkedIns Problem
    • The Birth of Kafka
    • Open Source
    • The Name
  • Getting Started with Kafka
• 2. Installing Kafka
  • First Things First
    • Choosing an Operating System
    • Installing Java
    • Installing Zookeeper
      • Standalone Server
      • Zookeeper ensemble
  • Installing a Kafka Broker
  • Broker Configuration
    • General Broker
      • broker.id
      • port
      • zookeeper.connect
      • log.dirs
      • num.recovery.threads.per.data.dir
      • auto.create.topics.enable
    • Topic Defaults
      • num.partitions
      • log.retention.ms
      • log.retention.bytes
      • log.segment.bytes
      • log.segment.ms
      • message.max.bytes
  • Hardware Selection
    • Disk Throughput
    • Disk Capacity
    • Memory
    • Networking
    • CPU
  • Kafka in the Cloud
  • Kafka Clusters
    • How Many Brokers?
    • Broker Configuration
    • OS Tuning
      • Virtual Memory
      • Disk
      • Networking
  • Production Concerns
    • Garbage Collector Options
    • Datacenter Layout
    • Colocating Applications on Zookeeper
  • Summary
• 3. Kafka Producers: Writing Messages to Kafka
  • Producer Overview
  • Constructing a Kafka Producer
  • Sending a Message to Kafka
    • Sending a Message Synchronously
    • Sending a Message Asynchronously
  • Configuring Producers
    • acks
    • buffer.memory
    • compression.type
    • retries
    • batch.size
    • linger.ms
    • client.id
    • max.in.flight.requests.per.connection
    • timeout.ms, request.timeout.ms, and metadata.fetch.timeout.ms
    • max.block.ms
    • max.request.size
    • receive.buffer.bytes and send.buffer.bytes
  • Serializers
    • Custom Serializers
    • Serializing Using Apache Avro
    • Using Avro Records with Kafka
  • Partitions
    • Implementing a custom partitioning strategy
  • Old Producer APIs
  • Summary
• 4. Kafka Consumers: Reading Data from Kafka
  • Kafka Consumer Concepts
    • Consumers and Consumer Groups
    • Consumer Groups and Partition Rebalance
  • Creating a Kafka Consumer
  • Subscribing to Topics
  • The Poll Loop
  • Configuring Consumers
    • fetch.min.bytes
    • fetch.max.wait.ms
    • max.partition.fetch.bytes
    • session.timeout.ms
    • auto.offset.reset
    • enable.auto.commit
    • partition.assignment.strategy
    • client.id
    • max.poll.records
    • receive.buffer.bytes and send.buffer.bytes
  • Commits and Offsets
    • Automatic Commit
    • Commit Current Offset
    • Asynchronous Commit
    • Combining Synchronous and Asynchronous Commits
    • Commit Specified Offset
  • Rebalance Listeners
  • Consuming Records with Specific Offsets
  • But How Do We Exit?
  • Deserializers
    • Custom deserializers
    • Using Avro deserialization with Kafka consumer
  • Standalone Consumer: Why and How to Use a Consumer Without a Group
  • Older Consumer APIs
  • Summary
• 5. Kafka Internals
  • Cluster Membership
  • The Controller
  • Replication
  • Request Processing
    • Produce Requests
    • Fetch Requests
    • Other Requests
  • Physical Storage
    • Partition Allocation
    • File Management
    • File Format
    • Indexes
    • Compaction
    • How Compaction Works
    • Deleted Events
    • When Are Topics Compacted?
  • Summary
• 6. Reliable Data Delivery
  • Reliability Guarantees
  • Replication
  • Broker Configuration
    • Replication Factor
    • Unclean Leader Election
    • Minimum In-Sync Replicas
  • Using Producers in a Reliable System
    • Send Acknowledgments
    • Configuring Producer Retries
    • Additional Error Handling
  • Using Consumers in a Reliable System
    • Important Consumer Configuration Properties for Reliable Processing
    • Explicitly Committing Offsets in Consumers
      • Always commit offsets after events were processed
      • Commit frequency is a trade-off between performance and number of duplicates in the event of a crash
      • Make sure you know exactly what offsets you are committing
      • Rebalances
      • Consumers may need to retry
      • Consumers may need to maintain state
      • Handling long processing times
      • Exactly-once delivery
  • Validating System Reliability
    • Validating Configuration
    • Validating Applications
    • Monitoring Reliability in Production
  • Summary
• 7. Building Data Pipelines
  • Considerations When Building Data Pipelines
    • Timeliness
    • Reliability
    • High and Varying Throughput
    • Data Formats
    • Transformations
    • Security
    • Failure Handling
    • Coupling and Agility
  • When to Use Kafka Connect Versus Producer and Consumer
  • Kafka Connect
    • Running Connect
    • Connector Example: File Source and File Sink
    • Connector Example: MySQL to Elasticsearch
    • A Deeper Look at Connect
      • Connectors and tasks
      • Workers
      • Converters and Connects data model
      • Offset management
  • Alternatives to Kafka Connect
    • Ingest Frameworks for Other Datastores
    • GUI-Based ETL Tools
    • Stream-Processing Frameworks
  • Summary
• 8. Cross-Cluster Data Mirroring
  • Use Cases of Cross-Cluster Mirroring
  • Multicluster Architectures
    • Some Realities of Cross-Datacenter Communication
    • Hub-and-Spokes Architecture
    • Active-Active Architecture
    • Active-Standby Architecture
      • Data loss and inconsistencies in unplanned failover
      • Start offset for applications after failover
      • After the failover
      • A few words on cluster discovery
    • Stretch Clusters
  • Apache Kafkas MirrorMaker
    • How to Configure
    • Deploying MirrorMaker in Production
    • Tuning MirrorMaker
  • Other Cross-Cluster Mirroring Solutions
    • Uber uReplicator
    • Confluents Replicator
  • Summary
• 9. Administering Kafka
  • Topic Operations
    • Creating a New Topic
    • Adding Partitions
    • Deleting a Topic
    • Listing All Topics in a Cluster
    • Describing Topic Details
  • Consumer Groups
    • List and Describe Groups
    • Delete Group
    • Offset Management
      • Export Offsets
      • Import Offsets
  • Dynamic Configuration Changes
    • Overriding Topic Configuration Defaults
    • Overriding Client Configuration Defaults
    • Describing Configuration Overrides
    • Removing Configuration Overrides
  • Partition Management
    • Preferred Replica Election
    • Changing a Partitions Replicas
    • Changing Replication Factor
    • Dumping Log Segments
    • Replica Verification
  • Consuming and Producing
    • Console Consumer
      • Message formatter options
      • Consuming the offsets topics
    • Console Producer
      • Line-Reader Options
  • Client ACLs
  • Unsafe Operations
    • Moving the Cluster Controller
    • Killing a Partition Move
    • Removing Topics to Be Deleted
    • Deleting Topics Manually
  • Summary
• 10. Monitoring Kafka
  • Metric Basics
    • Where Are the Metrics?
    • Internal or External Measurements
    • Application Health Checks
    • Metric Coverage
  • Kafka Broker Metrics
    • Under-Replicated Partitions
      • Cluster-level problems
      • Host-level problems
    • Broker Metrics
      • Active controller count
      • Request handler idle ratio
      • All topics bytes in
      • All topics bytes out
      • All topics messages in
      • Partition count
      • Leader count
      • Offline partitions
      • Request metrics
    • Topic and Partition Metrics
      • Per-topic metrics
      • Per-partition metrics
    • JVM Monitoring
      • Garbage collection
      • Java OS monitoring
    • OS Monitoring
    • Logging
  • Client Monitoring
    • Producer Metrics
      • Overall producer metrics
      • Per-broker and per-topic metrics
    • Consumer Metrics
      • Fetch manager metrics
      • Per-broker and per-topic metrics
      • Consumer coordinator metrics
    • Quotas
  • Lag Monitoring
  • End-to-End Monitoring
  • Summary
• 11. Stream Processing
  • What Is Stream Processing?
  • Stream-Processing Concepts
    • Time
    • State
    • Stream-Table Duality
    • Time Windows
  • Stream-Processing Design Patterns
    • Single-Event Processing
    • Processing with Local State
    • Multiphase Processing/Repartitioning
    • Processing with External Lookup: Stream-Table Join
    • Streaming Join
    • Out-of-Sequence Events
    • Reprocessing
  • Kafka Streams by Example
    • Word Count
    • Stock Market Statistics
    • Click Stream Enrichment
  • Kafka Streams: Architecture Overview
    • Building a Topology
    • Scaling the Topology
    • Surviving Failures
  • Stream Processing Use Cases
  • How to Choose a Stream-Processing Framework
  • Summary
• A. Installing Kafka on Other Operating Systems
  • Installing on Windows
    • Using Windows Subsystem for Linux
    • Using Native Java
  • Installing on MacOS
    • Using Homebrew
    • Installing Manually
• Index

pokaż cały spis treści

ukryj recenzje