Visualizing on Dataprd.Com

Air quality monitoring IoT – Arduino & sensors connected to a Raspberry Pi

Sat, 26 Aug 2017 16:42:00 +0000

I was very interested in monitoring the surrounding air’s quality, so I built a box that monitors Particulate Matter (PM), gas concentrations, temperature and humidity. The historical data is persisted and accessible via the network – translating to a typical IoT use case. The box built is based on Arduino with its sensors and a RaspberryPi. Arduino is a great platform for electronics prototyping with a convenient development IDE – it allows to connect sensors to the board and share the data via multiple interfaces (such as serial port, attached SD card, network, etc.). The networking and data persistence are solved via a Raspberry Pi Zero W, providing SD-card based data plotting, having a host and guest Wifi network for data access and data sharing with cloud providers.

Configure Apache Kylin with ODBC to work with MS PowerBI

Mon, 09 Jan 2017 21:13:16 +0000

PowerBI and Kylin - reporting from Hadoop via ODBC

This article discusses how to set up an ODBC interface for Kylin to work with Microsoft PowerBI. See previous article on the detailed dataset, environment setup, on what Kylin is and how to create a cube in Kylin. For the tutorial’s purposes, we will analyze the previously loaded flight delay data with Hadoop, Hive, HBase, Kylin, Kylin ODBC connector and MS PowerBI as an interface. With PowerBI, Microsoft provides a capable and simple BI tool for free (desktop version) - as competition heats up, this is a great strategy to gain some market share.

Analysis tutorial with Tableau Desktop

Sun, 11 May 2014 17:09:31 +0000

Tableau Desktop supports visual analysis and data discovery, converts the raw information to easy to understand graphical format with interactive charts. No coding is required to create rich visualization. Tableau Business Intelligence toolset has a Desktop, Server and Cloud version (none open-source products but as good as worth a post on the open-bigdata blog). In this post I check its Desktop evaluation version that lets us connect to many data sources (including Hadoop, MySQL, Excel, Text, …). I will use the same Weather.Csv as in the Hadoop analysis tutorial.

Weather data analysis and visualization - Big data tutorial Part 1/9 - Fundamentals

Thu, 13 Mar 2014 13:29:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 1/9

Analysis fundamentals

This experiment analyses the weather changes in the Carpathian Basin from 1900 to 2014 using a dataset of daily measurements of weather stations nearby our point of experiment. High emphasis is put on interactive visualization, as it is inevitable to make the comprehension of information easy. Big data analytics is gaining a lot of attention. This tutorial is to provide an overview on some open-source tools capable of supporting distributed analysis on huge datasets. Nonetheless it analyses the weather trends of the Carpathian Basin of Central Europe and provides an easy to use visualization on the changes.

Weather data analysis and visualization - Big data tutorial Part 2/9 - Dataset

Thu, 13 Mar 2014 13:28:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 2/9

Preparation - Dataset

Weather data from NOAA — National Climatic Center — is accessible using a great toolset that lets you select your area of interest on a map interactively. Custom dataset is downloadable via the NOAA map tool.

The detailed experiment dataset used in this tutorial is downloadable here: Weather.zip.

Weather data analysis and visualization - Big data tutorial Part 3/9 - Environment

Thu, 13 Mar 2014 13:27:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 3/9

Preparation - Analysis Environment

As the analyzed data is relatively small to get it processed on a single machine, I have spared some time to set up a new Hadoop cluster – I have administrative access to a smaller cluster of regular PCs chained into a Hadoop cluster but this one was reserved the time I made the experiment. Anyway, analyzing a small dataset with some big data tools is resulting in the same development efforts as analyzing Petabytes of data on a cluster of thousands of machines – only it takes less CPU time – one can still learn the basics on small datasets.

Weather data analysis and visualization - Big data tutorial Part 4/9 - Hadoop & Pig

Thu, 13 Mar 2014 13:26:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 4/9

Analysis with Hadoop and Pig

Igniting Hortonworks Hadoop environment

Starting the virtual machine
Logging in from the host machine’s browser: 127.0.0.1:8888
- Log in with credentials supplied on login screen: hue / 1111
A straightforward menu is given to manage stored files (HDFS), Pig or Hive analysis scripts and to check the jobs that are running

Weather data analysis and visualization - Big data tutorial Part 5/9 - Visualizing: GIS & map

Thu, 13 Mar 2014 13:25:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 5/9

Data visualization - GIS, map based using Kartograph - Subsection 1/2

For the showcase purposes, I have used this Kartograph example to show bubbles with different colors and radius based on the yearly temperature and precipitation data (two datasets are in two different directories / diagrams). Kartograph is an open-source Python based map generator and JavaScript based web map illustrator. A stand-alone package to generate maps without using 3rd party applications like Google Maps. If you generate your map once you could avoid any service level or API changes regarding your map.

Weather data analysis and visualization - Big data tutorial Part 6/9 - SED example

Thu, 13 Mar 2014 13:24:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 6/9

Manipulating output data with the Linux SED command - SED example

This Kartograph tutorial uses JSON as data format, so I needed the same format for my own data that the tutorial uses - example:

[{"Weather": "ARAD RO", "ll": [21.35, 46.1331], "1882": 742.0 … {"Weather": "MURSKA SOBOTA RAKICAN SI", "ll": [16.2, 46.7], … "1962": 931.0}]

And the resulting data from PIG is not compatible with it, as it has different markup, presented here:

Weather data analysis and visualization - Big data tutorial Part 7/9 - Visualizing: GIS & map part 2

Thu, 13 Mar 2014 13:23:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 7/9

Data visualization - GIS, map based using Kartograph - Subsection 2/2

The two examples showcased in the previous part:

I’ve combined the two together:

Using the files from the animated symbols of the first one and
Added tooltip data by altering the addSymbol function in the JS section of index.html
Tooltips containing PRCP data for each decade

symbols = map.addSymbols({
type: $K.Bubble,
data: weatherStations,
location: function(d) { return d.ll },
attrs: symbolAttrs,
tooltip: function(d) { return $.trim(d.Weather)+<br>Rainfall 1900: ' +d['1900']+' mm'+ '<br>Rainfall 1910: ' +d['1910']+' mm'+ '<br>Rainfall 1920: ' +d['1920']+' mm'+ '<br>Rainfall 1930: ' +d['1930']+' mm'+ '<br>Rainfall 1940: ' +d['1940']+' mm'+ '<br>Rainfall 1950: ' +d['1950']+' mm'+ '<br>Rainfall 1960: ' +d['1960']+' mm'+ '<br>Rainfall 1970: ' +d['1970']+' mm'+ '<br>Rainfall 1980: ' +d['1980']+' mm'+ '<br>Rainfall 1990: ' +d['1990']+' mm'+  '<br>Rainfall 2000: ' +d['2000']+' mm'+ '<br>;Rainfall 2010: ' +d['2010']+' mm'+ '<br>Rainfall 2013: ' +d['2013']+' mm'; }
});

Looking at the map, some tooltip refinement was needed as those were out of the page, due to their large size: As I’ve defined a big tooltip to show I’ve changed the positioning by altering kartograph.min.js Changing this line kartograph.min.js

Weather data analysis and visualization - Big data tutorial Part 8/9 - Visualizing: HTML charts

Thu, 13 Mar 2014 13:22:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 8/9

Data visualization - Interactive HTML5 charts with Flot JS Library

The two best tools for HTML5 charting I have checked out are

JQplot
Flot

I’ve chosen Flot’s interactive chart with switchable diagrams and came to an idea that it would be nice to also have trend lines for the charts and used it as an example. Both tools use proprietary data sets, instead of manipulating their example parsers, I’ve chosen to alter my data to comply with the provided examples. After altering the files with SED, so as above, I have come to a problem — Hadoop is a parallel system, all processes are analyzing portions of the dataset, resulting dataset is unordered and if it is complex, it can’t be ordered with big. So I had data like:

Weather data analysis and visualization - Big data tutorial Part 9/9 - Results

Thu, 13 Mar 2014 13:21:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 9/9

Results of the analysis

The two resulting artifacts from the analysis are presented here.

The same methods can be used to mine and visualize temperature change information too.