Hadoop on Dataprd.Com

Data Governance in the Big Data and Hadoop world - Whitepaper

Fri, 25 Aug 2017 16:11:15 +0000

My full whitepaper — Download PDF

Amidst vast data lakes and a high velocity of incoming data, enterprises are finding that despite the procedures, policies, and systems that are already in place, their data governance frameworks still aren’t delivering the insights needed to drive smart business decisions. Pain points such as privacy and security, appropriate permissions, and labor-intensive data mining processes are challenging general data governance frameworks that do not allow for flexibility or agility. Hadoop can be an excellent framework to support a large data storage repository with immense processing power. The key to success is understanding how to unleash Hadoop’s powerful storage capabilities. This white paper reveals:

Evaluation of Apache Kylin 1.5.4.1 with HDP 2.5, performance comparison w Hive

Mon, 09 Jan 2017 20:24:45 +0000

Apache Kylin is a data cube solution on top of Hadoop providing an ODBC interface for BI tools. OLAP cubes boost performance for analytics via using a subset of data, enriched with pre-calculations on specific dimensions of interest. It enables loading dimensions from a Hive data source, therefore accelerating BI tool access via pre-calculating data and adding it to HBase. In our example we have a large dataset of flight information:

Top 7 challenges of building a data lake

Sat, 07 Jan 2017 18:26:42 +0000

While from the technical perspective, deployment, management and provisioning tools are available to quickly set up a Hadoop cluster, introducing it to the organization is a tough task. Here are the biggest challenges of building a data lake 1. Understand the purpose and limitations of the technology

Hadoop environment provides a vendor independent tool-chain for data storage, management and analytics in a scalable fashion - it can manage infinite amount of data and with add-ons makes real-time and data science use-cases available for the enterprise
Many organizations do not have Big Data. They just load their RDBMS content and get surprised that a 20 node enterprise Hadoop setup’s performance is subpar compared with a single PostgreSQL instance. Sure, because the purpose and capabilities are different - we should compare apples with apples after all
In 95% of the cases stakeholders think that this new technology will substitute their painful proprietary data environment of RDBMSs. It is a highly distributed environment so the case is that it is never capable to substitute proprietary data systems. But it can live well beneath those providing extreme value. The purpose of the data lake is different. Ingest data, all of your data, and deal with portions of it without costly ETLs to move it from here to there. Refine your data system, be it processing, reporting or prediction models by slowly adding more datasets.

2. Security

Performance test of Pig vs Hive with code examples

Fri, 08 Aug 2014 17:40:07 +0000

Performance testing high level Hadoop query languages with example scripts. Analysis of NOAA weather data: Western-European weather stations from 1980 to 2014, daily dataset of temperature (tmin and tmax) and precipitation data (prcp). Dataset is a structured table, non-existent measurement cells are filled with -9999. Example: STATION,STATION_NAME,DATE,PRCP,TMAX,TMIN GHCND:NLE00109300,STAVENISSE NL,19800101,53,-9999,-9999 GHCND:NLE00109300,STAVENISSE NL,19800102,21,-9999,-9999 GHCND:NLE00109300,STAVENISSE NL,19800103,133,-9999,-9999 … GHCND:NLE00109202,MARUM NL,20080602,0,-9999,-9999 GHCND:NLE00109202,MARUM NL,20080603,36,-9999,-9999 GHCND:NLE00109202,MARUM NL,20080604,4,-9999,-9999 … Data size: 1 Gb / 4 Gb / 8 Gb (the same 333 Mb data file replicated 3 / 12 / 24 times) HDFS block size: 128 Mb Platform:

Create a Hadoop Cluster easily by using PXE boot, Kickstart, Puppet and Ambari to auto-deploy nodes

Fri, 08 Aug 2014 17:12:52 +0000

This tutorial is to showcase unattended and automatic install of multiple CentOS 6.5 x86_64 Hadoop nodes pre-configured with Ambari-agents and an Ambari-server host. After configuring automatic install of bare metal (No OS pre-installed) nodes, deploying a Hadoop cluster will be a matter of clicks. The setup uses:

PXE boot (for automatic OS install)
TFTP server (for PXE network install image)
Apache server (to serve the kickstart file for unattended install)
DHCP server (for assigning IP addresses for the nodes)
DNS server (for internal domain name resolution)
Puppet-master (for automatic configuration management of all hosts in the network, Ambari install included in Puppet manifests)
Ambari-master and agents (for managing Hadoop ecosystem deployment)

The setup assumes that the nodes are on the 192.168.0.0/255.255.255.0 network, the master is on 192.168.0.1 and its hostname is bigdata1.hdp The domain for the network server by the configuration is hdp and the clients are named as bigdata[1-254].hdp Download all files here (configuration files, PXEBoot Linux image, Kickstart file and custom script for adding a node on the master).

Setting up a firewall to secure a Hadoop cluster's network with Shorewall

Fri, 14 Mar 2014 13:40:59 +0000

Shorewall is a tool to configure Linux inbuilt IPTables in an easy and understandable way. Assume we have a basic setup: Lan | Firewall with Proxy server | Internet A secure setup is to:

ACCEPT HTTP(80) and HTTPS(443) from LAN to NET
ACCEPT special services ports from specific LAN to NET (like e-banking)
ACCEPT only the needed FW services from LAN to FW - SSH(22) and MAIL(25,443,993,…) to FW (if mail server is on FW)
ACCEPT only the needed FW services from NET to FW - SSH(22) with IP restriction (and SSH key) and MAIL(25,443,993,…) to FW (if mail server is on FW)
- SSH port can be changed
LOC to LOC connections are not possible to be governed by FW, therefore all allowed
REJECT any incoming connection (other than above) to LAN or FW

Shorewall configuration - with some additional examples on the syntax and rules:

Security of a Hadoop cluster

Fri, 14 Mar 2014 13:30:30 +0000

Hadoop is not only a data processing but a data warehouse solution. When we are talking about high amount of data we are talking about business value (the basis of the services) for a company. As Hadoop uses many nodes, network devices all the equipment must be included in a security plan - no weakest links are allowed. To secure our assets the following tools and approaches can be used.

Weather data analysis and visualization - Big data tutorial Part 1/9 - Fundamentals

Thu, 13 Mar 2014 13:29:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 1/9

Analysis fundamentals

This experiment analyses the weather changes in the Carpathian Basin from 1900 to 2014 using a dataset of daily measurements of weather stations nearby our point of experiment. High emphasis is put on interactive visualization, as it is inevitable to make the comprehension of information easy. Big data analytics is gaining a lot of attention. This tutorial is to provide an overview on some open-source tools capable of supporting distributed analysis on huge datasets. Nonetheless it analyses the weather trends of the Carpathian Basin of Central Europe and provides an easy to use visualization on the changes.

Weather data analysis and visualization - Big data tutorial Part 2/9 - Dataset

Thu, 13 Mar 2014 13:28:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 2/9

Preparation - Dataset

Weather data from NOAA — National Climatic Center — is accessible using a great toolset that lets you select your area of interest on a map interactively. Custom dataset is downloadable via the NOAA map tool.

The detailed experiment dataset used in this tutorial is downloadable here: Weather.zip.

Weather data analysis and visualization - Big data tutorial Part 3/9 - Environment

Thu, 13 Mar 2014 13:27:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 3/9

Preparation - Analysis Environment

As the analyzed data is relatively small to get it processed on a single machine, I have spared some time to set up a new Hadoop cluster – I have administrative access to a smaller cluster of regular PCs chained into a Hadoop cluster but this one was reserved the time I made the experiment. Anyway, analyzing a small dataset with some big data tools is resulting in the same development efforts as analyzing Petabytes of data on a cluster of thousands of machines – only it takes less CPU time – one can still learn the basics on small datasets.

Weather data analysis and visualization - Big data tutorial Part 4/9 - Hadoop & Pig

Thu, 13 Mar 2014 13:26:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 4/9

Analysis with Hadoop and Pig

Igniting Hortonworks Hadoop environment

Starting the virtual machine
Logging in from the host machine’s browser: 127.0.0.1:8888
- Log in with credentials supplied on login screen: hue / 1111
A straightforward menu is given to manage stored files (HDFS), Pig or Hive analysis scripts and to check the jobs that are running

Weather data analysis and visualization - Big data tutorial Part 5/9 - Visualizing: GIS & map

Thu, 13 Mar 2014 13:25:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 5/9

Data visualization - GIS, map based using Kartograph - Subsection 1/2

For the showcase purposes, I have used this Kartograph example to show bubbles with different colors and radius based on the yearly temperature and precipitation data (two datasets are in two different directories / diagrams). Kartograph is an open-source Python based map generator and JavaScript based web map illustrator. A stand-alone package to generate maps without using 3rd party applications like Google Maps. If you generate your map once you could avoid any service level or API changes regarding your map.

Weather data analysis and visualization - Big data tutorial Part 6/9 - SED example

Thu, 13 Mar 2014 13:24:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 6/9

Manipulating output data with the Linux SED command - SED example

This Kartograph tutorial uses JSON as data format, so I needed the same format for my own data that the tutorial uses - example:

[{"Weather": "ARAD RO", "ll": [21.35, 46.1331], "1882": 742.0 … {"Weather": "MURSKA SOBOTA RAKICAN SI", "ll": [16.2, 46.7], … "1962": 931.0}]

And the resulting data from PIG is not compatible with it, as it has different markup, presented here:

Weather data analysis and visualization - Big data tutorial Part 7/9 - Visualizing: GIS & map part 2

Thu, 13 Mar 2014 13:23:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 7/9

Data visualization - GIS, map based using Kartograph - Subsection 2/2

The two examples showcased in the previous part:

I’ve combined the two together:

Using the files from the animated symbols of the first one and
Added tooltip data by altering the addSymbol function in the JS section of index.html
Tooltips containing PRCP data for each decade

symbols = map.addSymbols({
type: $K.Bubble,
data: weatherStations,
location: function(d) { return d.ll },
attrs: symbolAttrs,
tooltip: function(d) { return $.trim(d.Weather)+<br>Rainfall 1900: ' +d['1900']+' mm'+ '<br>Rainfall 1910: ' +d['1910']+' mm'+ '<br>Rainfall 1920: ' +d['1920']+' mm'+ '<br>Rainfall 1930: ' +d['1930']+' mm'+ '<br>Rainfall 1940: ' +d['1940']+' mm'+ '<br>Rainfall 1950: ' +d['1950']+' mm'+ '<br>Rainfall 1960: ' +d['1960']+' mm'+ '<br>Rainfall 1970: ' +d['1970']+' mm'+ '<br>Rainfall 1980: ' +d['1980']+' mm'+ '<br>Rainfall 1990: ' +d['1990']+' mm'+  '<br>Rainfall 2000: ' +d['2000']+' mm'+ '<br>;Rainfall 2010: ' +d['2010']+' mm'+ '<br>Rainfall 2013: ' +d['2013']+' mm'; }
});

Looking at the map, some tooltip refinement was needed as those were out of the page, due to their large size: As I’ve defined a big tooltip to show I’ve changed the positioning by altering kartograph.min.js Changing this line kartograph.min.js

Weather data analysis and visualization - Big data tutorial Part 8/9 - Visualizing: HTML charts

Thu, 13 Mar 2014 13:22:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 8/9

Data visualization - Interactive HTML5 charts with Flot JS Library

The two best tools for HTML5 charting I have checked out are

JQplot
Flot

I’ve chosen Flot’s interactive chart with switchable diagrams and came to an idea that it would be nice to also have trend lines for the charts and used it as an example. Both tools use proprietary data sets, instead of manipulating their example parsers, I’ve chosen to alter my data to comply with the provided examples. After altering the files with SED, so as above, I have come to a problem — Hadoop is a parallel system, all processes are analyzing portions of the dataset, resulting dataset is unordered and if it is complex, it can’t be ordered with big. So I had data like:

Weather data analysis and visualization - Big data tutorial Part 9/9 - Results

Thu, 13 Mar 2014 13:21:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 9/9

Results of the analysis

The two resulting artifacts from the analysis are presented here.

The same methods can be used to mine and visualize temperature change information too.

A multi-tiered Big Data warehouse & processing facility

Wed, 12 Mar 2014 18:17:16 +0000

The article details an exemplary setup for a multi-tiered data warehouse and processing facility using Hadoop and batch type of data analysis. Requirements in brief:

High peaks on data acquisition (e.g. financial transactions of yearly festivals, R&D facility with images or videos flowing in only at the time of experimenting)
- Velocity is high at peak times, Gigabytes / second for short time (max. few hours / day)
- Data to be quickly accessed for data processing for some weeks after acquisition
High volume of data to be stored for long-time
- Data to be accessed some years after acquisition, access can be delayed by days after request

The below setup details 3 tiers:

A Big Data Course

Wed, 12 Mar 2014 09:26:36 +0000

The published diagram is to detail a setup of a Big Data Course.

Fundamentals on databases (SQL and NoSQL), statistics (the R framework) and graph databases
The focus is on the Hadoop eco-system and its programming paradigm, MapReduce
MapReduce is available to be used with easier to master high level query languages like Pig and Hive
While Hadoop is for batch processing there are other usage areas:
- Real-time data access by HBase NoSQL daemon
- Fast but lower data volume processor, Spark
- Machine learning framework that can be run on top of Hadoop: Mahout
To be able to use these tools a well built, secure cluster is to be planned and developed, then operated securely
After the data analysis is done, final steps of visualization are detailed - to make an impact by using the achieved analytic results

A derivative of the course is held at the University of Szeged.

Environment setup for big data analytics

Tue, 04 Mar 2014 15:18:50 +0000

This article covers basic tools and technologies to use when conducting the first steps on big data analysis.

Linux as the base OS
- Debian
- Ubuntu
- RedHat
- CentOS
For basic data processing:
- Bash shell: environment for running multiple command-line Linux tools for data manipulation
  - Comes pre-installed with Linux; check Reference manual for usage
  - Bash might be not the default Linux shell, see how to switch to it
  - Learn Bash by examples
  - Most important Linux Commands and phenomena to master for data manipulation:
    - cat - output/concatenate a stream / file
    - head, tail - show the head or the tail of a stream / file
    - grep - to filter streams / lines of a file
    - sed - to manipulate streams / lines of a file
    - Regular expressions - used in a wide set of Linux tools
      - Explanation and tutorials
- AWK - simple data reformatter with compact coding features
  - The GNU Awk User’s Guide
  - AWK Tutorials
- Python - easy to learn, effective programming language with a huge amount of libraries available for various tasks. Great for data manipulation used from the command line.
  - Python Download
  - Python Documentation
  - Python Beginners Guide
  - Python Tutorials
And the big data analysis framework chosen based on the type of data analyzed. For the first step tutorials our suggestion would be:
- Hadoop, single cluster setup (can be downloaded pre-installed to a virtual appliance)
- Java based MapReduce programs
- Pig MapReduce query language