Tutorial on Dataprd.Com

Performance test of Pig vs Hive with code examples

Fri, 08 Aug 2014 17:40:07 +0000

Performance testing high level Hadoop query languages with example scripts. Analysis of NOAA weather data: Western-European weather stations from 1980 to 2014, daily dataset of temperature (tmin and tmax) and precipitation data (prcp). Dataset is a structured table, non-existent measurement cells are filled with -9999. Example: STATION,STATION_NAME,DATE,PRCP,TMAX,TMIN GHCND:NLE00109300,STAVENISSE NL,19800101,53,-9999,-9999 GHCND:NLE00109300,STAVENISSE NL,19800102,21,-9999,-9999 GHCND:NLE00109300,STAVENISSE NL,19800103,133,-9999,-9999 … GHCND:NLE00109202,MARUM NL,20080602,0,-9999,-9999 GHCND:NLE00109202,MARUM NL,20080603,36,-9999,-9999 GHCND:NLE00109202,MARUM NL,20080604,4,-9999,-9999 … Data size: 1 Gb / 4 Gb / 8 Gb (the same 333 Mb data file replicated 3 / 12 / 24 times) HDFS block size: 128 Mb Platform:

Create a Hadoop Cluster easily by using PXE boot, Kickstart, Puppet and Ambari to auto-deploy nodes

Fri, 08 Aug 2014 17:12:52 +0000

This tutorial is to showcase unattended and automatic install of multiple CentOS 6.5 x86_64 Hadoop nodes pre-configured with Ambari-agents and an Ambari-server host. After configuring automatic install of bare metal (No OS pre-installed) nodes, deploying a Hadoop cluster will be a matter of clicks. The setup uses:

PXE boot (for automatic OS install)
TFTP server (for PXE network install image)
Apache server (to serve the kickstart file for unattended install)
DHCP server (for assigning IP addresses for the nodes)
DNS server (for internal domain name resolution)
Puppet-master (for automatic configuration management of all hosts in the network, Ambari install included in Puppet manifests)
Ambari-master and agents (for managing Hadoop ecosystem deployment)

The setup assumes that the nodes are on the 192.168.0.0/255.255.255.0 network, the master is on 192.168.0.1 and its hostname is bigdata1.hdp The domain for the network server by the configuration is hdp and the clients are named as bigdata[1-254].hdp Download all files here (configuration files, PXEBoot Linux image, Kickstart file and custom script for adding a node on the master).

Analysis tutorial with Tableau Desktop

Sun, 11 May 2014 17:09:31 +0000

Tableau Desktop supports visual analysis and data discovery, converts the raw information to easy to understand graphical format with interactive charts. No coding is required to create rich visualization. Tableau Business Intelligence toolset has a Desktop, Server and Cloud version (none open-source products but as good as worth a post on the open-bigdata blog). In this post I check its Desktop evaluation version that lets us connect to many data sources (including Hadoop, MySQL, Excel, Text, …). I will use the same Weather.Csv as in the Hadoop analysis tutorial.

Data mining from webpages with Python Mechanize Browser Automation - a Big data tutorial

Thu, 13 Mar 2014 13:31:28 +0000

Data mining from the web with Python Mechanize Browser Automation - Big data tutorial

In this example the Python Mechanize package is used for browser automation - Selenium is much more feature rich (and is also a bit more difficult to use) and is to be used when feature-rich Javascript and Ajax website data mining or automated test case setup is to be built. A Selenium browser automation example can be found here. This rich commented Python Mechanize Browser Automation example does the following:

Data mining from webpages with Selenium Python WebDriver Browser Automation - a Big data tutorial

Thu, 13 Mar 2014 13:30:28 +0000

Data mining from the web with Selenium Python WebDriver Browser Automation - Big data tutorial

In this example the Selenium web test automation framework uses Firefox for browser automation - Selenium is much more feature rich (and is also a bit more difficult to use) than Python Mechanize - having an example here. It uses a full suite of a web browser, so as an advantage Javascript and AJAX rich webpage parsing can be automated - in such cases it has to be used instead of Python Mechanize for data mining and auto testing purposes. This rich commented Selenium Python WebDriver Browser Automation example does the following: