Weather data analysis and visualization - Big data tutorial Part 6/9 - SED example

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 6/9

Manipulating output data with the Linux SED command - SED example

This Kartograph tutorial uses JSON as data format, so I needed the same format for my own data that the tutorial uses - example:

[{"Weather": "ARAD RO", "ll": [21.35, 46.1331], "1882": 742.0 … {"Weather": "MURSKA SOBOTA RAKICAN SI", "ll": [16.2, 46.7], … "1962": 931.0}]

And the resulting data from PIG is not compatible with it, as it has different markup, presented here:

((ARAD RO,46.1331,21.35),{((1882,742)),((1883,680)),((1884,656)),((1885,656)),((1886,770)),((1887,718)),((1888,467)),((1889,893)),((1890,570)),



?



92))})



((DEVA RO,45.8667,22.9),?

So I’ve used the SED Linux command to alter the resulting dataset from Pig

SED example

Sed command 1

sed 's/(([A-Z ]*),([0-9.]*),([0-9.]*))/[{"Weather": "1", "ll: [3, 2], /g' rain_orig.csv > rain_new.csv && cat rain_new.csv

This command checks for “(any number of capital characters)”, followed by “,” and “(any number of 0-9 numbers or “.”)” blocks of two separated by a “,” so is valid for "((ARAD RO,46.1331,21.35)" and changes this string to [{“Weather”: “ARAD RO”, “ll”: [21.35, 46.1331] The syntax:

sed 's/  /  /g' input > output

for each string of the input find:

regular expression / change it to something.

1, 2 in SED Command 1 is a link to the regular expressions defined in the first part, in our case, 1 refers to ARAD RO, which is encapsulated in " " marks I parse rain_orig.csv , the output of PIG and output it to rain_new.csv Sed command 2

sed 's/{((([0-9]*),/"1": /g' rain_new.csv > rain_new2.csv && cat rain_new2.csv

The following are set up as the above, parsing for {((number, and changing it to “number” - valid for years Sed command 3

sed 's/)),((([0-9]*),/, "1":/g' rain_new2.csv > rain_new3.csv && cat rain_new3.csv

While this one parses for )),((numbers), and changes it to numbers So after running all the 3 SED commands we have the needed results in the format of

[{"Weather": "ARAD RO", "ll": [21.35, 46.1331], "1882": 742.0, "1883": 680.0, …

One manual task left is to properly close the encapsulation at the end of the JSON file by changing } , to }] The JSON for the Map is available now, and I’ve saved it to the directory of the new map visualization example.

VisualizingHadoopInteractive Visualization