BigTable, Key/Value, Distributed, Petabyte scale, fault tolerance

Sorted, schema less, graph support

On top of Hadoop, Zookeeper, Thrift

BTrees, bloom filters, compression

Uses Zookeeper, MR, have CLI

Clustered. Linear scalability. Failover master

NSA

Per cell ACL + table ACL

Built in user DB and plaintext on wire

Iterators can be inserted in read path and table maintenance code

Advanced UI

On top of HDFS

BigTable, Key/Value, Distributed, Petabyte scale, fault tolerance

Clustered. Linear scalability. Failover master

Sorted, RESTful

FB, eBay, Flurry, Adobe, Mozilla, TrendMicro

BTrees, bloom filters, compression

Uses Zookeeper, MR, have CLI

Column family ACL

Advanced security and integration with Hadoop

Triggers, stored procedures, cluster

management hooks

HBase is 4 5 times slower than HDFS in batch

context. HBase is for random access

Stores key/value pairs in columnar fashion

(columns are clubbed together as column families).

Provides low latency access to small amounts of

data from within a large data set.

Provides flexible data model.

HBase is not optimized for classic transactional

applications or even relational analytics.

HBase is CPU and Memory intensive with

sporadic large sequential I/O access

HBase supports massively parallelized processing via

MapReduce for using HBase as both source and sink

Hadoop part

Optimized for streaming access of large files.

Follows write once read many ideology.

Doesn't support random read/write.

Good for MapReduce jobs are primarily I/O bound

with fixed memory and sporadic CPU.

Memory centric

Hadoop/Spark compatible wo changes

High performance by leveraging lineage

information and using memory aggressively

40 contributors from over 15 institutions,

including Yahoo, Intel, and Redhat

High throughput writes

Read 200x and write 300x faster than HDFS

Latency 17x times less than HDFS

Uses memory (instead of disk) and recomputation (instead of replication) to

produce a distributed, fault tolerant, and high throughput file system.

Data sets are in the gigabytes or terabytes

Low latency

JSON like internal data model to

represent and process data

Dynamic schema discovery. Can query self describing

data formats as JSON, NoSQL, AVRO etc.

SQL queries against a petabyte or more of data distributed

across 10,000 plus servers

Drillbit instead of MapReduce

Full ANSI SQL:2003

Low latency distributed execution engine

Unknown schema on HBase,

Cassandra, MongoDB

Fault tolerant

Memory intensive

INPUT: RDBMS, NoSQL, HBase, Hive tables, JSON, Avro, Parquet, FS etc.

Shell, WebUI, ODBC, JDBC, CPP API

Not mature enough?

Low latency

HiveQL and SQL92

INPUT: Can use HDFS or HBase wo move/transform. Can use Hive tables as metastore

Integrated with Hadoop, queries Hadoop

Requires specific format for performance

Memory intensive (if join two tables, one of them need to be in cluster memory)

Claimed as x30 faster than Hive for queries with multiple MR jobs, but it's before

Stinger

No fault tolerance

Reads Hadoop file formats, including text, LZO, SequenceFile, Avro, RCFile, and Parquet

Fine grained, role based authorization with Sentry. Supports Hadoop security

ANSI SQL

Real time

Petabytes of data

INPUT: Can use Hive, HDFS, HBase, Cassandra, Scribe

Similar to Impala and Hive/Stinger

Own engine

Declared as CPU more efficient than Hive/MapReduce

Just opened, not so big community

Not fail tolerant

Read only

No UDF

Requires specific format for performance

A lot of MR micro jobs

HQL

Can use HBase

SQL, HQL or Scala to run on Spark

Can use Hive data also

JDBC/ODBC

Spark part

Replaced with Spark SQL

Long jobs

HQL

Based on Hadoop and MR

High latency

Schedule Hadoop jobs

Combines multiple jobs sequentially

into unit of work

Integrated with Hadoop stack

Supports jobs for MR, Pig, Hive and

Sqoop + system app, Java, shell

UI for Hadoop and satellites (HDFS, MR,

Hive, Oozie, Pig, Impala, Solr etc.)

Webpanel

Upload files to HDFS, send Hive queries etc.

High level scripting language

Can invoke from code on Java, Ruby etc.

Can get data from files, streams or other sources

Output to HDFS

Pig scripts translated to series of MR jobs

Transferring bulk data between Apache Hadoop

and structured datastores such as relational

databases.

Two way replication with both

snapshots and incremental updates.

Import between external datastores,

HDSF, Hive, HBase etc.

Works with relational databases such as:

Teradata, Netezza, Oracle, MySQL, Postgres,

and HSQLDB

INPUT: Can access data in Hadoop via Hive, Impala, Spark

SQL, Drill, Presto or any ODBC in Hortonworks, Cloudera,

DataStax, MapR distributions

OUTPUT: reports, UI web, UI client

Clustered. Nearly linear scalability

Can access traditional DB

Can explore and visualize data

SQL

Provides single point of authentication

and access to services in Hadoop

cluster

Spark part

API for graphs and graph parallel computation