Cluster Based Data Analytics Tutorials on Dataprd.Com

Configure Apache Kylin with ODBC to work with MS PowerBI

Mon, 09 Jan 2017 21:13:16 +0000

PowerBI and Kylin - reporting from Hadoop via ODBC

This article discusses how to set up an ODBC interface for Kylin to work with Microsoft PowerBI. See previous article on the detailed dataset, environment setup, on what Kylin is and how to create a cube in Kylin. For the tutorial’s purposes, we will analyze the previously loaded flight delay data with Hadoop, Hive, HBase, Kylin, Kylin ODBC connector and MS PowerBI as an interface. With PowerBI, Microsoft provides a capable and simple BI tool for free (desktop version) - as competition heats up, this is a great strategy to gain some market share.

Evaluation of Apache Kylin 1.5.4.1 with HDP 2.5, performance comparison w Hive

Mon, 09 Jan 2017 20:24:45 +0000

Apache Kylin is a data cube solution on top of Hadoop providing an ODBC interface for BI tools. OLAP cubes boost performance for analytics via using a subset of data, enriched with pre-calculations on specific dimensions of interest. It enables loading dimensions from a Hive data source, therefore accelerating BI tool access via pre-calculating data and adding it to HBase. In our example we have a large dataset of flight information:

Top 7 challenges of building a data lake

Sat, 07 Jan 2017 18:26:42 +0000

While from the technical perspective, deployment, management and provisioning tools are available to quickly set up a Hadoop cluster, introducing it to the organization is a tough task. Here are the biggest challenges of building a data lake 1. Understand the purpose and limitations of the technology

Hadoop environment provides a vendor independent tool-chain for data storage, management and analytics in a scalable fashion - it can manage infinite amount of data and with add-ons makes real-time and data science use-cases available for the enterprise
Many organizations do not have Big Data. They just load their RDBMS content and get surprised that a 20 node enterprise Hadoop setup’s performance is subpar compared with a single PostgreSQL instance. Sure, because the purpose and capabilities are different - we should compare apples with apples after all
In 95% of the cases stakeholders think that this new technology will substitute their painful proprietary data environment of RDBMSs. It is a highly distributed environment so the case is that it is never capable to substitute proprietary data systems. But it can live well beneath those providing extreme value. The purpose of the data lake is different. Ingest data, all of your data, and deal with portions of it without costly ETLs to move it from here to there. Refine your data system, be it processing, reporting or prediction models by slowly adding more datasets.

2. Security

Performance test of Pig vs Hive with code examples

Fri, 08 Aug 2014 17:40:07 +0000

Performance testing high level Hadoop query languages with example scripts. Analysis of NOAA weather data: Western-European weather stations from 1980 to 2014, daily dataset of temperature (tmin and tmax) and precipitation data (prcp). Dataset is a structured table, non-existent measurement cells are filled with -9999. Example: STATION,STATION_NAME,DATE,PRCP,TMAX,TMIN GHCND:NLE00109300,STAVENISSE NL,19800101,53,-9999,-9999 GHCND:NLE00109300,STAVENISSE NL,19800102,21,-9999,-9999 GHCND:NLE00109300,STAVENISSE NL,19800103,133,-9999,-9999 … GHCND:NLE00109202,MARUM NL,20080602,0,-9999,-9999 GHCND:NLE00109202,MARUM NL,20080603,36,-9999,-9999 GHCND:NLE00109202,MARUM NL,20080604,4,-9999,-9999 … Data size: 1 Gb / 4 Gb / 8 Gb (the same 333 Mb data file replicated 3 / 12 / 24 times) HDFS block size: 128 Mb Platform:

Create a Hadoop Cluster easily by using PXE boot, Kickstart, Puppet and Ambari to auto-deploy nodes

Fri, 08 Aug 2014 17:12:52 +0000

This tutorial is to showcase unattended and automatic install of multiple CentOS 6.5 x86_64 Hadoop nodes pre-configured with Ambari-agents and an Ambari-server host. After configuring automatic install of bare metal (No OS pre-installed) nodes, deploying a Hadoop cluster will be a matter of clicks. The setup uses:

PXE boot (for automatic OS install)
TFTP server (for PXE network install image)
Apache server (to serve the kickstart file for unattended install)
DHCP server (for assigning IP addresses for the nodes)
DNS server (for internal domain name resolution)
Puppet-master (for automatic configuration management of all hosts in the network, Ambari install included in Puppet manifests)
Ambari-master and agents (for managing Hadoop ecosystem deployment)

The setup assumes that the nodes are on the 192.168.0.0/255.255.255.0 network, the master is on 192.168.0.1 and its hostname is bigdata1.hdp The domain for the network server by the configuration is hdp and the clients are named as bigdata[1-254].hdp Download all files here (configuration files, PXEBoot Linux image, Kickstart file and custom script for adding a node on the master).

Analysis tutorial with Tableau Desktop

Sun, 11 May 2014 17:09:31 +0000

Tableau Desktop supports visual analysis and data discovery, converts the raw information to easy to understand graphical format with interactive charts. No coding is required to create rich visualization. Tableau Business Intelligence toolset has a Desktop, Server and Cloud version (none open-source products but as good as worth a post on the open-bigdata blog). In this post I check its Desktop evaluation version that lets us connect to many data sources (including Hadoop, MySQL, Excel, Text, …). I will use the same Weather.Csv as in the Hadoop analysis tutorial.

Data mining from webpages with Python Mechanize Browser Automation - a Big data tutorial

Thu, 13 Mar 2014 13:31:28 +0000

Data mining from the web with Python Mechanize Browser Automation - Big data tutorial

In this example the Python Mechanize package is used for browser automation - Selenium is much more feature rich (and is also a bit more difficult to use) and is to be used when feature-rich Javascript and Ajax website data mining or automated test case setup is to be built. A Selenium browser automation example can be found here. This rich commented Python Mechanize Browser Automation example does the following:

Data mining from webpages with Selenium Python WebDriver Browser Automation - a Big data tutorial

Thu, 13 Mar 2014 13:30:28 +0000

Data mining from the web with Selenium Python WebDriver Browser Automation - Big data tutorial

In this example the Selenium web test automation framework uses Firefox for browser automation - Selenium is much more feature rich (and is also a bit more difficult to use) than Python Mechanize - having an example here. It uses a full suite of a web browser, so as an advantage Javascript and AJAX rich webpage parsing can be automated - in such cases it has to be used instead of Python Mechanize for data mining and auto testing purposes. This rich commented Selenium Python WebDriver Browser Automation example does the following:

Send to Kindle via email - using browser automation

Thu, 13 Mar 2014 13:29:27 +0000

I often read web articles on the Kindle e-reader for less eye strain. In order to achieve that, I use the Send to Kindle browser extension, maintained by Amazon. In the past few years many Kindle publishing services just came and went, so the official Send to Kindle was always the most stable one to use. However, one important feature is not supported, sending articles from mobile or tablet - by just sending the URL to an email address, not an article’s content attached, processable by the @free.kindle.com email addresses. Here, my script is presented, which could run on any machines using Python to check an email box with unread emails containing URLs and send the articles’ content to the Kindle e-reader.

Weather data analysis and visualization - Big data tutorial Part 1/9 - Fundamentals

Thu, 13 Mar 2014 13:29:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 1/9

Analysis fundamentals

This experiment analyses the weather changes in the Carpathian Basin from 1900 to 2014 using a dataset of daily measurements of weather stations nearby our point of experiment. High emphasis is put on interactive visualization, as it is inevitable to make the comprehension of information easy. Big data analytics is gaining a lot of attention. This tutorial is to provide an overview on some open-source tools capable of supporting distributed analysis on huge datasets. Nonetheless it analyses the weather trends of the Carpathian Basin of Central Europe and provides an easy to use visualization on the changes.

Weather data analysis and visualization - Big data tutorial Part 2/9 - Dataset

Thu, 13 Mar 2014 13:28:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 2/9

Preparation - Dataset

Weather data from NOAA — National Climatic Center — is accessible using a great toolset that lets you select your area of interest on a map interactively. Custom dataset is downloadable via the NOAA map tool.

The detailed experiment dataset used in this tutorial is downloadable here: Weather.zip.

Weather data analysis and visualization - Big data tutorial Part 3/9 - Environment

Thu, 13 Mar 2014 13:27:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 3/9

Preparation - Analysis Environment

As the analyzed data is relatively small to get it processed on a single machine, I have spared some time to set up a new Hadoop cluster – I have administrative access to a smaller cluster of regular PCs chained into a Hadoop cluster but this one was reserved the time I made the experiment. Anyway, analyzing a small dataset with some big data tools is resulting in the same development efforts as analyzing Petabytes of data on a cluster of thousands of machines – only it takes less CPU time – one can still learn the basics on small datasets.

Weather data analysis and visualization - Big data tutorial Part 4/9 - Hadoop & Pig

Thu, 13 Mar 2014 13:26:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 4/9

Analysis with Hadoop and Pig

Igniting Hortonworks Hadoop environment

Starting the virtual machine
Logging in from the host machine’s browser: 127.0.0.1:8888
- Log in with credentials supplied on login screen: hue / 1111
A straightforward menu is given to manage stored files (HDFS), Pig or Hive analysis scripts and to check the jobs that are running

Weather data analysis and visualization - Big data tutorial Part 5/9 - Visualizing: GIS & map

Thu, 13 Mar 2014 13:25:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 5/9

Data visualization - GIS, map based using Kartograph - Subsection 1/2

For the showcase purposes, I have used this Kartograph example to show bubbles with different colors and radius based on the yearly temperature and precipitation data (two datasets are in two different directories / diagrams). Kartograph is an open-source Python based map generator and JavaScript based web map illustrator. A stand-alone package to generate maps without using 3rd party applications like Google Maps. If you generate your map once you could avoid any service level or API changes regarding your map.

Weather data analysis and visualization - Big data tutorial Part 6/9 - SED example

Thu, 13 Mar 2014 13:24:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 6/9

Manipulating output data with the Linux SED command - SED example

This Kartograph tutorial uses JSON as data format, so I needed the same format for my own data that the tutorial uses - example:

[{"Weather": "ARAD RO", "ll": [21.35, 46.1331], "1882": 742.0 … {"Weather": "MURSKA SOBOTA RAKICAN SI", "ll": [16.2, 46.7], … "1962": 931.0}]

And the resulting data from PIG is not compatible with it, as it has different markup, presented here:

Weather data analysis and visualization - Big data tutorial Part 7/9 - Visualizing: GIS & map part 2

Thu, 13 Mar 2014 13:23:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 7/9

Data visualization - GIS, map based using Kartograph - Subsection 2/2

The two examples showcased in the previous part:

I’ve combined the two together:

Using the files from the animated symbols of the first one and
Added tooltip data by altering the addSymbol function in the JS section of index.html
Tooltips containing PRCP data for each decade

symbols = map.addSymbols({
type: $K.Bubble,
data: weatherStations,
location: function(d) { return d.ll },
attrs: symbolAttrs,
tooltip: function(d) { return $.trim(d.Weather)+<br>Rainfall 1900: ' +d['1900']+' mm'+ '<br>Rainfall 1910: ' +d['1910']+' mm'+ '<br>Rainfall 1920: ' +d['1920']+' mm'+ '<br>Rainfall 1930: ' +d['1930']+' mm'+ '<br>Rainfall 1940: ' +d['1940']+' mm'+ '<br>Rainfall 1950: ' +d['1950']+' mm'+ '<br>Rainfall 1960: ' +d['1960']+' mm'+ '<br>Rainfall 1970: ' +d['1970']+' mm'+ '<br>Rainfall 1980: ' +d['1980']+' mm'+ '<br>Rainfall 1990: ' +d['1990']+' mm'+  '<br>Rainfall 2000: ' +d['2000']+' mm'+ '<br>;Rainfall 2010: ' +d['2010']+' mm'+ '<br>Rainfall 2013: ' +d['2013']+' mm'; }
});

Looking at the map, some tooltip refinement was needed as those were out of the page, due to their large size: As I’ve defined a big tooltip to show I’ve changed the positioning by altering kartograph.min.js Changing this line kartograph.min.js

Weather data analysis and visualization - Big data tutorial Part 8/9 - Visualizing: HTML charts

Thu, 13 Mar 2014 13:22:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 8/9

Data visualization - Interactive HTML5 charts with Flot JS Library

The two best tools for HTML5 charting I have checked out are

JQplot
Flot

I’ve chosen Flot’s interactive chart with switchable diagrams and came to an idea that it would be nice to also have trend lines for the charts and used it as an example. Both tools use proprietary data sets, instead of manipulating their example parsers, I’ve chosen to alter my data to comply with the provided examples. After altering the files with SED, so as above, I have come to a problem — Hadoop is a parallel system, all processes are analyzing portions of the dataset, resulting dataset is unordered and if it is complex, it can’t be ordered with big. So I had data like:

Weather data analysis and visualization - Big data tutorial Part 9/9 - Results

Thu, 13 Mar 2014 13:21:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 9/9

Results of the analysis

The two resulting artifacts from the analysis are presented here.

The same methods can be used to mine and visualize temperature change information too.

A multi-tiered Big Data warehouse & processing facility

Wed, 12 Mar 2014 18:17:16 +0000

The article details an exemplary setup for a multi-tiered data warehouse and processing facility using Hadoop and batch type of data analysis. Requirements in brief:

High peaks on data acquisition (e.g. financial transactions of yearly festivals, R&D facility with images or videos flowing in only at the time of experimenting)
- Velocity is high at peak times, Gigabytes / second for short time (max. few hours / day)
- Data to be quickly accessed for data processing for some weeks after acquisition
High volume of data to be stored for long-time
- Data to be accessed some years after acquisition, access can be delayed by days after request

The below setup details 3 tiers:

A Big Data Course

Wed, 12 Mar 2014 09:26:36 +0000

The published diagram is to detail a setup of a Big Data Course.

Fundamentals on databases (SQL and NoSQL), statistics (the R framework) and graph databases
The focus is on the Hadoop eco-system and its programming paradigm, MapReduce
MapReduce is available to be used with easier to master high level query languages like Pig and Hive
While Hadoop is for batch processing there are other usage areas:
- Real-time data access by HBase NoSQL daemon
- Fast but lower data volume processor, Spark
- Machine learning framework that can be run on top of Hadoop: Mahout
To be able to use these tools a well built, secure cluster is to be planned and developed, then operated securely
After the data analysis is done, final steps of visualization are detailed - to make an impact by using the achieved analytic results

A derivative of the course is held at the University of Szeged.

Environment setup for big data analytics

Tue, 04 Mar 2014 15:18:50 +0000

This article covers basic tools and technologies to use when conducting the first steps on big data analysis.

Linux as the base OS
- Debian
- Ubuntu
- RedHat
- CentOS
For basic data processing:
- Bash shell: environment for running multiple command-line Linux tools for data manipulation
  - Comes pre-installed with Linux; check Reference manual for usage
  - Bash might be not the default Linux shell, see how to switch to it
  - Learn Bash by examples
  - Most important Linux Commands and phenomena to master for data manipulation:
    - cat - output/concatenate a stream / file
    - head, tail - show the head or the tail of a stream / file
    - grep - to filter streams / lines of a file
    - sed - to manipulate streams / lines of a file
    - Regular expressions - used in a wide set of Linux tools
      - Explanation and tutorials
- AWK - simple data reformatter with compact coding features
  - The GNU Awk User’s Guide
  - AWK Tutorials
- Python - easy to learn, effective programming language with a huge amount of libraries available for various tasks. Great for data manipulation used from the command line.
  - Python Download
  - Python Documentation
  - Python Beginners Guide
  - Python Tutorials
And the big data analysis framework chosen based on the type of data analyzed. For the first step tutorials our suggestion would be:
- Hadoop, single cluster setup (can be downloaded pre-installed to a virtual appliance)
- Java based MapReduce programs
- Pig MapReduce query language