Tags

, , ,

I recently spoke at Big mountain data 2014 conference. Here are the slides and steps for the demo.

Link to the slides : http://www.slideshare.net/abhishek376/real-time-analytics-using-hadoop-and-elasticsearch

Hadoop 2.4.0
Hive 0.13.0
Elasticsearch 1.3.4 (http://www.elasticsearch.org/download/)
HDP 2.1 (http://hortonworks.com/products/hortonworks-sandbox/)

HDP doesn’t come with elasticsearch installed.

INSTALLING ELASTICSEARCH :
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.4.noarch.rpm
rpm -Uvh elasticsearch-1.3.4.noarch.rpm
chkconfig –add elasticsearch
service elasticsearch start
cd /usr/share/elasticsearch/

INSTALLING HEAD AND DESK PLUGINS:
bin/plugin –install mobz/elasticsearch-head
bin/plugin –install lukas-vlcek/bigdesk

INSERTING DATA IN TO MYSQL:
drop database bigmountaindata;
create database bigmountaindata;

use bigmountaindata;
create table gender(id int, height int, gender varchar(50));
insert into gender values(1,6,’Male’);
insert into gender values(2,5,’Male’);
insert into gender values(3,6,’Female’);

select * from gender;

USING SQOOP TO IMPORT DATA IN TO HIVE:
Hive:
create database bigmountaindata;

SQOOP :
sqoop import –connect ‘jdbc:mysql://127.0.0.1:3306/bigmountaindata’ –username=root –table gender -m 1 –class-name bigmountaindata.gender –outdir /tmp/src/generated –target-dir /tmp/sqoop-import/bigmountaindata.gender –hive-import –hive-drop-import-delims –hive-table bigmountaindata.gender

USING HIVE EXTERNAL TABLES TO PUSH DATA IN TO ELASTICSEARCH:
add jar /usr/lib/hive/lib/aux/elasticsearch-hadoop-hive-2.1.0.jar;
use bigmountaindata;
CREATE EXTERNAL TABLE if not exists es_gender (
id INT,
height INT,
gender STRING)
STORED BY ‘org.elasticsearch.hadoop.hive.EsStorageHandler’
TBLPROPERTIES(‘es.resource’ = ‘bigmountaindata/gender’);
insert overwrite table es_gender select id, height, gender from the;

ELASTICSEARCH :

Number of male and Female
http://hortonworks:9200/bigmountaindata/gender/_search?search_type=count
{
“aggs”: {
“genders”: {
“terms”: {
“field”: “gender”
}
}
}
}

Average height of males and females
http://hortonworks:9200/bigmountaindata/gender/_search?search_type=count
{
“aggs”: {
“genders”: {
“terms”: {
“field”: “gender”
},
“aggs”: {
“avg_height”: {
“avg”: {
“field”: “height”
}
}
}
}
}
}

Feel free to reach me if you have any question @abhishek376/ abhishek376@gmail.com

Advertisements