Hadoop Application Architectures
- Autorzy:
- Mark Grover, Ted Malaska, Jonathan Seidman
- Ocena:
- Bądź pierwszym, który oceni tę książkę
- Stron:
- 400
- Dostępne formaty:
-
ePubMobi
Opis ebooka: Hadoop Application Architectures
Get expert guidance on architecting end-to-end data management solutions with Apache Hadoop. While many sources explain how to use various components in the Hadoop ecosystem, this practical book takes you through architectural considerations necessary to tie those components together into a complete tailored application, based on your particular use case.
To reinforce those lessons, the book’s second section provides detailed examples of architectures used in some of the most commonly found Hadoop applications. Whether you’re designing a new Hadoop application, or planning to integrate Hadoop into your existing data infrastructure, Hadoop Application Architectures will skillfully guide you through the process.
This book covers:
- Factors to consider when using Hadoop to store and model data
- Best practices for moving data in and out of the system
- Data processing frameworks, including MapReduce, Spark, and Hive
- Common Hadoop processing patterns, such as removing duplicate records and using windowing analytics
- Giraph, GraphX, and other tools for large graph processing on Hadoop
- Using workflow orchestration and scheduling tools such as Apache Oozie
- Near-real-time stream processing with Apache Storm, Apache Spark Streaming, and Apache Flume
- Architecture examples for clickstream analysis, fraud detection, and data warehousing
Wybrane bestsellery
-
While many companies ponder implementation details such as distributed processing engines and algorithms for data analysis, this practical book takes a much wider view of big data development, starting with initial planning and moving diligently toward execution. Authors Ted Malaska and Jonathan ...
Foundations for Architecting Data Solutions. Managing Successful Data Projects Foundations for Architecting Data Solutions. Managing Successful Data Projects
(152.15 zł najniższa cena z 30 dni)156.62 zł
179.00 zł(-13%) -
This book helps you create efficient databases, covering data modeling, query optimization, and more. You'll be equipped with the skills needed to design robust databases using PostgreSQL and MySQL for modern data-driven applications.
Database Design and Modeling with PostgreSQL and MySQL. Build efficient and scalable databases for modern applications using open source databases Database Design and Modeling with PostgreSQL and MySQL. Build efficient and scalable databases for modern applications using open source databases
(78.48 zł najniższa cena z 30 dni) -
This book provides a highly focused view of real business outcomes powered by data governance, that resonate with non-data executives such as CFOs and CEOs. You’ll also find useful insights into how to implement data governance initiatives.
Data Governance Handbook. A practical approach to building trust in data Data Governance Handbook. A practical approach to building trust in data
-
This book shows you how to use Apache Spark, Delta Lake, and Databricks to build data pipelines, manage and transform data, optimize performance, and more. Additionally, you’ll implement DataOps and DevOps practices, and orchestrate data workflows.
Data Engineering with Databricks Cookbook. Build effective data and AI solutions using Apache Spark, Databricks, and Delta Lake Data Engineering with Databricks Cookbook. Build effective data and AI solutions using Apache Spark, Databricks, and Delta Lake
-
This book will guide you through the fundamental and advanced features of the Snowpark framework in Python. You’ll learn how to use Snowpark for implementing workloads in the fields of data engineering, data science, and data applications.
The Ultimate Guide to Snowpark. Design and deploy Snowpark with Python for efficient data workloads The Ultimate Guide to Snowpark. Design and deploy Snowpark with Python for efficient data workloads
-
This report highlights the vital role of data quality in your data strategy, offering actionable steps to make it the foundation of your data culture, unlocking greater value and informed decisions in the evolving landscape of AI.
Data Quality in the Age of AI. Building a foundation for AI strategy and data culture Data Quality in the Age of AI. Building a foundation for AI strategy and data culture
-
Ця книжка познайомить вас з особливостями Jav...
Head First. Програмування на JavaScript. Head First. Програмування на JavaScript Head First. Програмування на JavaScript. Head First. Програмування на JavaScript
(84.16 zł najniższa cena z 30 dni)84.16 zł
103.90 zł(-19%) -
«Патерни проєктування» 2014 ваша книжка, якщо C...(84.16 zł najniższa cena z 30 dni)
84.16 zł
103.90 zł(-19%) -
This practical guide to implementing DeFi in your projects guides you through building full-stack DeFi solutions with popular tools and teaches you how to leverage blockchain technologies to manage crypto assets.
Building Full Stack DeFi Applications. A practical guide to creating your own decentralized finance projects on blockchain Building Full Stack DeFi Applications. A practical guide to creating your own decentralized finance projects on blockchain
Ebooka "Hadoop Application Architectures" przeczytasz na:
-
czytnikach Inkbook, Kindle, Pocketbook, Onyx Boox i innych
-
systemach Windows, MacOS i innych
-
systemach Windows, Android, iOS, HarmonyOS
-
na dowolnych urządzeniach i aplikacjach obsługujących formaty: PDF, EPub, Mobi
Masz pytania? Zajrzyj do zakładki Pomoc »
Audiobooka "Hadoop Application Architectures" posłuchasz:
-
w aplikacji Ebookpoint na Android, iOS, HarmonyOs
-
na systemach Windows, MacOS i innych
-
na dowolnych urządzeniach i aplikacjach obsługujących format MP3 (pliki spakowane w ZIP)
Masz pytania? Zajrzyj do zakładki Pomoc »
Kurs Video "Hadoop Application Architectures" zobaczysz:
-
w aplikacjach Ebookpoint i Videopoint na Android, iOS, HarmonyOs
-
na systemach Windows, MacOS i innych z dostępem do najnowszej wersji Twojej przeglądarki internetowej
Szczegóły ebooka
- ISBN Ebooka:
- 978-14-919-0005-5, 9781491900055
- Data wydania ebooka:
- 2015-06-30 Data wydania ebooka często jest dniem wprowadzenia tytułu do sprzedaży i może nie być równoznaczna z datą wydania książki papierowej. Dodatkowe informacje możesz znaleźć w darmowym fragmencie. Jeśli masz wątpliwości skontaktuj się z nami sklep@ebookpoint.pl.
- Język publikacji:
- angielski
- Rozmiar pliku ePub:
- 6.0MB
- Rozmiar pliku Mobi:
- 6.0MB
Spis treści ebooka
- Foreword
- Preface
- A Note About the Code Examples
- Who Should Read This Book
- Why We Wrote This Book
- Navigating This Book
- Conventions Used in This Book
- Using Code Examples
- Safari Books Online
- How to Contact Us
- Acknowledgments
- Mark Grovers Acknowledgements
- Ted Malaskas Acknowledgements
- Jonathan Seidmans Acknowledgements
- Gwen Shapiras Acknowledgements
- I. Architectural Considerations for Hadoop Applications
- 1. Data Modeling in Hadoop
- Data Storage Options
- Standard File Formats
- Text data
- Structured text data
- Binary data
- Standard File Formats
- Hadoop File Types
- File-based data structures
- Data Storage Options
- Serialization Formats
- Thrift
- Protocol Buffers
- Avro
- Columnar Formats
- RCFile
- ORC
- Parquet
- Avro and Parquet
- Compression
- Snappy
- LZO
- Gzip
- bzip2
- Compression recommendations
- HDFS Schema Design
- Location of HDFS Files
- Advanced HDFS Schema Design
- Partitioning
- Bucketing
- Denormalizing
- HDFS Schema Design Summary
- HBase Schema Design
- Row Key
- Record retrieval
- Distribution
- Block cache
- Ability to scan
- Size
- Readability
- Uniqueness
- Row Key
- Timestamp
- Hops
- Tables and Regions
- Put performance
- Compaction time
- Using Columns
- Using Column Families
- Time-to-Live
- Managing Metadata
- What Is Metadata?
- Why Care About Metadata?
- Where to Store Metadata?
- Examples of Managing Metadata
- Limitations of the Hive Metastore and HCatalog
- Other Ways of Storing Metadata
- Embedding metadata in file paths and names
- Storing the metadata in HDFS
- Conclusion
- 2. Data Movement
- Data Ingestion Considerations
- Timeliness of Data Ingestion
- Incremental Updates
- Access Patterns
- Original Source System and Data Structure
- Read speed of the devices on source systems
- Original file type
- Compression
- Relational database management systems
- Streaming data
- Logfiles
- Transformations
- Interceptors
- Selectors
- Data Ingestion Considerations
- Network Bottlenecks
- Network Security
- Push or Pull
- Sqoop
- Flume
- Failure Handling
- Level of Complexity
- Data Ingestion Options
- File Transfers
- HDFS client commands
- Mountable HDFS
- File Transfers
- Considerations for File Transfers versus Other Ingest Methods
- Sqoop: Batch Transfer Between Hadoop and Relational Databases
- Choosing a split-by column
- Using database-specific connectors whenever available
- Using the Goldilocks method of Sqoop performance tuning
- Loading many tables in parallel with fair scheduler throttling
- Diagnosing bottlenecks
- Keeping Hadoop updated
- Flume: Event-Based Data Collection and Processing
- Flume architecture
- Flume patterns
- File formats
- Recommendations
- Flume sources
- Flume sinks
- Flume interceptors
- Flume memory channels
- Flume file channels
- Sizing Channels
- Finding Flume bottlenecks
- Kafka
- Kafka fault tolerance
- Kafka and Hadoop
- Data Extraction
- Conclusion
- 3. Processing Data in Hadoop
- MapReduce
- MapReduce Overview
- Map phase
- InputFormat
- RecordReader
- Mapper.setup()
- Mapper.map
- Partitioner
- Mapper.cleanup()
- Combiner
- Map phase
- Reducer
- Shuffle
- Reducer.setup()
- Reducer.reduce()
- Reducer.cleanup()
- OutputFormat
- MapReduce Overview
- MapReduce
- Example for MapReduce
- When to Use MapReduce
- Spark
- Spark Overview
- DAG Model
- Spark Overview
- Overview of Spark Components
- Basic Spark Concepts
- Resilient Distributed Datasets
- Shared variables
- SparkContext
- Transformations
- Action
- Benefits of Using Spark
- Simplicity
- Versatility
- Reduced disk I/O
- Storage
- Multilanguage
- Resource manager independence
- Interactive shell (REPL)
- Spark Example
- When to Use Spark
- Abstractions
- Pig
- Pig Example
- When to Use Pig
- Crunch
- Crunch Example
- When to Use Crunch
- Cascading
- Cascading Example
- When to Use Cascading
- Hive
- Hive Overview
- Example of Hive Code
- When to Use Hive
- Impala
- Impala Overview
- Speed-Oriented Design
- Efficient use of memory
- Long running daemons
- Efficient execution engine
- Use of LLVM
- Impala Example
- When to Use Impala
- Conclusion
- 4. Common Hadoop Processing Patterns
- Pattern: Removing Duplicate Records by Primary Key
- Data Generation for Deduplication Example
- Code Example: Spark Deduplication in Scala
- Code Example: Deduplication in SQL
- Pattern: Removing Duplicate Records by Primary Key
- Pattern: Windowing Analysis
- Data Generation for Windowing Analysis Example
- Code Example: Peaks and Valleys in Spark
- Code Example: Peaks and Valleys in SQL
- Pattern: Time Series Modifications
- Use HBase and Versioning
- Use HBase with a RowKey of RecordKey and StartTime
- Use HDFS and Rewrite the Whole Table
- Use Partitions on HDFS for Current and Historical Records
- Data Generation for Time Series Example
- Code Example: Time Series in Spark
- Code Example: Time Series in SQL
- Conclusion
- 5. Graph Processing on Hadoop
- What Is a Graph?
- What Is Graph Processing?
- How Do You Process a Graph in a Distributed System?
- The Bulk Synchronous Parallel Model
- BSP by Example
- Giraph
- Read and Partition the Data
- Batch Process the Graph with BSP
- Write the Graph Back to Disk
- Putting It All Together
- When Should You Use Giraph?
- GraphX
- Just Another RDD
- GraphX Pregel Interface
- vprog()
- sendMessage()
- mergeMessage()
- Which Tool to Use?
- Conclusion
- 6. Orchestration
- Why We Need Workflow Orchestration
- The Limits of Scripting
- The Enterprise Job Scheduler and Hadoop
- Orchestration Frameworks in the Hadoop Ecosystem
- Oozie Terminology
- Oozie Overview
- Oozie Workflow
- Workflow Patterns
- Point-to-Point Workflow
- Fan-Out Workflow
- Capture-and-Decide Workflow
- Parameterizing Workflows
- Classpath Definition
- Scheduling Patterns
- Frequency Scheduling
- Time and Data Triggers
- Executing Workflows
- Conclusion
- 7. Near-Real-Time Processing with Hadoop
- Stream Processing
- Apache Storm
- Storm High-Level Architecture
- Storm Topologies
- Tuples and Streams
- Spouts and Bolts
- Stream Groupings
- Reliability of Storm Applications
- Exactly-Once Processing
- Fault Tolerance
- Integrating Storm with HDFS
- Integrating Storm with HBase
- Storm Example: Simple Moving Average
- Evaluating Storm
- Support for aggregation and windowing
- Enrichment and alerting
- Lamdba Architecture
- Trident
- Trident Example: Simple Moving Average
- Evaluating Trident
- Support for counting and windowing
- Enrichment and alerting
- Lamdba Architecture
- Spark Streaming
- Overview of Spark Streaming
- Spark Streaming Example: Simple Count
- Spark Streaming Example: Multiple Inputs
- Spark Streaming Example: Maintaining State
- Spark Streaming Example: Windowing
- Spark Streaming Example: Streaming versus ETL Code
- Evaluating Spark Streaming
- Support for counting and windowing
- Enrichment and alerting
- Lambda Architecture
- Flume Interceptors
- Which Tool to Use?
- Low-Latency Enrichment, Validation, Alerting, and Ingestion
- Solution One: Flume
- Solution Two: Kafka and Storm
- Low-Latency Enrichment, Validation, Alerting, and Ingestion
- NRT Counting, Rolling Averages, and Iterative Processing
- Complex Data Pipelines
- Conclusion
- II. Case Studies
- 8. Clickstream Analysis
- Defining the Use Case
- Using Hadoop for Clickstream Analysis
- Design Overview
- Storage
- Ingestion
- The Client Tier
- The Collector Tier
- Processing
- Data Deduplication
- Deduplication in Hive
- Deduplication in Pig
- Data Deduplication
- Sessionization
- Sessionization in Spark
- Sessionization in MapReduce
- Sessionization in Pig
- Sessionization in Hive
- Analyzing
- Orchestration
- Conclusion
- 9. Fraud Detection
- Continuous Improvement
- Taking Action
- Architectural Requirements of Fraud Detection Systems
- Introducing Our Use Case
- High-Level Design
- Client Architecture
- Profile Storage and Retrieval
- Caching
- Distributed memory caching
- HBase with BlockCache
- Caching
- HBase Data Definition
- Columns (combined or atomic)
- Event counting using HBase increment or put
- Event history using HBase put
- Delivering Transaction Status: Approved or Denied?
- Ingest
- Path Between the Client and Flume
- Client push
- Logfile pull
- Message queue or Kafka in the middle
- Path Between the Client and Flume
- Near-Real-Time and Exploratory Analytics
- Near-Real-Time Processing
- Exploratory Analytics
- What About Other Architectures?
- Flume Interceptors
- Kafka to Storm or Spark Streaming
- External Business Rules Engine
- Conclusion
- 10. Data Warehouse
- Using Hadoop for Data Warehousing
- Defining the Use Case
- OLTP Schema
- Data Warehouse: Introduction and Terminology
- Data Warehousing with Hadoop
- High-Level Design
- Data Modeling and Storage
- Choosing a storage engine
- Denormalizing
- Tracking updates in Hadoop
- Selecting storage format and compression
- Partitioning
- Data Modeling and Storage
- Ingestion
- Data Processing and Access
- Partitioning
- Merge/update
- Aggregations
- Data Export
- Orchestration
- Conclusion
- A. Joins in Impala
- Broadcast Joins
- Partitioned Hash Join
- Index
O'Reilly Media - inne książki
-
Keeping up with the Python ecosystem can be daunting. Its developer tooling doesn't provide the out-of-the-box experience native to languages like Rust and Go. When it comes to long-term project maintenance or collaborating with others, every Python project faces the same problem: how to build re...(203.15 zł najniższa cena z 30 dni)
208.19 zł
239.00 zł(-13%) -
Bringing a deep-learning project into production at scale is quite challenging. To successfully scale your project, a foundational understanding of full stack deep learning, including the knowledge that lies at the intersection of hardware, software, data, and algorithms, is required.This book il...(237.15 zł najniższa cena z 30 dni)
250.05 zł
289.00 zł(-13%) -
Frontend developers have to consider many things: browser compatibility, usability, performance, scalability, SEO, and other best practices. But the most fundamental aspect of creating websites is one that often falls short: accessibility. Accessibility is the cornerstone of any website, and if a...(194.65 zł najniższa cena z 30 dni)
207.45 zł
239.00 zł(-13%) -
In this insightful and comprehensive guide, Addy Osmani shares more than a decade of experience working on the Chrome team at Google, uncovering secrets to engineering effectiveness, efficiency, and team success. Engineers and engineering leaders looking to scale their effectiveness and drive tra...(118.15 zł najniższa cena z 30 dni)
121.44 zł
149.00 zł(-18%) -
Data modeling is the single most overlooked feature in Power BI Desktop, yet it's what sets Power BI apart from other tools on the market. This practical book serves as your fast-forward button for data modeling with Power BI, Analysis Services tabular, and SQL databases. It serves as a starting ...(194.65 zł najniższa cena z 30 dni)
207.00 zł
239.00 zł(-13%) -
C# is undeniably one of the most versatile programming languages available to engineers today. With this comprehensive guide, you'll learn just how powerful the combination of C# and .NET can be. Author Ian Griffiths guides you through C# 12.0 and .NET 8 fundamentals and techniques for building c...(228.65 zł najniższa cena z 30 dni)
250.44 zł
289.00 zł(-13%) -
Learn how to get started with Futures Thinking. With this practical guide, Phil Balagtas, founder of the Design Futures Initiative and the global Speculative Futures network, shows you how designers and futurists have made futures work at companies such as Atari, IBM, Apple, Disney, Autodesk, Luf...(152.15 zł najniższa cena z 30 dni)
155.45 zł
189.00 zł(-18%) -
Augmented Analytics isn't just another book on data and analytics; it's a holistic resource for reimagining the way your entire organization interacts with information to become insight-driven.Moving beyond traditional, limited ways of making sense of data, Augmented Analytics provides a dynamic,...(177.65 zł najniższa cena z 30 dni)
181.35 zł
219.00 zł(-17%) -
Learn how to prepare for—and pass—the Kubernetes and Cloud Native Associate (KCNA) certification exam. This practical guide serves as both a study guide and point of entry for practitioners looking to explore and adopt cloud native technologies. Adrián González Sánchez ...
Kubernetes and Cloud Native Associate (KCNA) Study Guide Kubernetes and Cloud Native Associate (KCNA) Study Guide
(169.14 zł najniższa cena z 30 dni)177.65 zł
209.00 zł(-15%) -
Python is an excellent way to get started in programming, and this clear, concise guide walks you through Python a step at a time—beginning with basic programming concepts before moving on to functions, data structures, and object-oriented design. This revised third edition reflects the gro...(148.82 zł najniższa cena z 30 dni)
148.72 zł
179.00 zł(-17%)
Dzieki opcji "Druk na żądanie" do sprzedaży wracają tytuły Grupy Helion, które cieszyły sie dużym zainteresowaniem, a których nakład został wyprzedany.
Dla naszych Czytelników wydrukowaliśmy dodatkową pulę egzemplarzy w technice druku cyfrowego.
Co powinieneś wiedzieć o usłudze "Druk na żądanie":
- usługa obejmuje tylko widoczną poniżej listę tytułów, którą na bieżąco aktualizujemy;
- cena książki może być wyższa od początkowej ceny detalicznej, co jest spowodowane kosztami druku cyfrowego (wyższymi niż koszty tradycyjnego druku offsetowego). Obowiązująca cena jest zawsze podawana na stronie WWW książki;
- zawartość książki wraz z dodatkami (płyta CD, DVD) odpowiada jej pierwotnemu wydaniu i jest w pełni komplementarna;
- usługa nie obejmuje książek w kolorze.
Masz pytanie o konkretny tytuł? Napisz do nas: sklep[at]helion.pl.
Książka, którą chcesz zamówić pochodzi z końcówki nakładu. Oznacza to, że mogą się pojawić drobne defekty (otarcia, rysy, zagięcia).
Co powinieneś wiedzieć o usłudze "Końcówka nakładu":
- usługa obejmuje tylko książki oznaczone tagiem "Końcówka nakładu";
- wady o których mowa powyżej nie podlegają reklamacji;
Masz pytanie o konkretny tytuł? Napisz do nas: sklep[at]helion.pl.
Książka drukowana
Oceny i opinie klientów: Hadoop Application Architectures Mark Grover, Ted Malaska, Jonathan Seidman (0) Weryfikacja opinii następuję na podstawie historii zamówień na koncie Użytkownika umieszczającego opinię. Użytkownik mógł otrzymać punkty za opublikowanie opinii uprawniające do uzyskania rabatu w ramach Programu Punktowego.