BigData HDFC, Hbase and Hive

HDFS (Hadoop Distributed File System)

• Purpose: HDFS is a distributed file system that provides high-throughput access to large datasets. It is the foundational storage layer for many Hadoop ecosystem components.

• Nature: HDFS stores files but does not inherently provide a table structure. It stores data as blocks across multiple nodes in a cluster.

• Usage: You do not create tables in HDFS directly. Instead, HDFS stores the raw data files (e.g., text files, CSV files, Parquet files), which can then be processed or queried using other tools like Hive, Spark, or Impala.

Hive

• Purpose: Hive is a data warehousing tool built on top of Hadoop. It allows you to query and manage large datasets residing in distributed storage using a SQL-like language called HiveQL.

• Nature: Hive provides a table structure and allows for schema-on-read, meaning it applies a schema to data when the data is read. Hive tables can be stored in HDFS.

Usage: You can create and manage tables using Hive, which abstracts the underlying HDFS files and provides a familiar SQL interface for querying data.

CREATE TABLE my_table (

    id INT,

    name STRING,

    age INT

)

ROW FORMAT DELIMITED

FIELDS TERMINATED BY ','

STORED AS TEXTFILE;

HBase

• Purpose: HBase is a NoSQL database that runs on top of HDFS. It is designed for real-time read/write access to large datasets.

• Nature: HBase provides a table-like structure with rows and columns, but it is schema-less for the columns, meaning you can add columns on the fly. It is suitable for applications requiring random, real-time read/write access to Big Data.

Usage: You create tables in HBase using its shell or API, defining column families rather than individual columns.
sh

create 'my_table', 'cf'

Differences

1. Storage Level:

   • HDFS: Stores raw data files.

   • Hive: Tables are metadata over HDFS files, enabling SQL-like queries.

   • HBase: Tables store data directly in HBase, optimized for quick reads and writes.

2. Schema Management:

   • HDFS: No inherent schema, just files.

   • Hive: Schema-on-read, defined when querying.

   • HBase: Schema-less for columns, but requires defining column families.

3. Use Cases:

   • HDFS: General storage for large datasets, batch processing.

   • Hive: Data warehousing, analytical queries using SQL.

   • HBase: Real-time data processing, low-latency access, and updates.

4. Data Access:

   • HDFS: Accessed through various tools (e.g., MapReduce, Spark, Hive).

   • Hive: Accessed via HiveQL or integrated with other query engines like Impala.

   • HBase: Accessed programmatically (Java, Python, REST API) or via the HBase shell.

Summary

In the Cloudera Big Data ecosystem, HDFS serves as the underlying storage system, Hive provides a data warehousing layer on top of HDFS, and HBase offers a NoSQL database for real-time data operations. Each serves a distinct purpose and fits different use cases, but together they form a powerful toolkit for managing and analyzing Big Data.

4o