Dataprd.Com

Turning Natural Language Pricing into Structured Data for Automated Pricing

Sat, 15 Aug 2026 09:00:00 +0000

Every accommodation booking system eventually runs into the same wall. Owners describe their prices in prose, and software needs rows in a table.

What arrives looks like this, give or take a language. The colours show which sentence becomes which kind of row:

Owner's text

High season from 15 June to 22 August, 35,000 per night for up to 6 guests, extra person 5,000 per night.

Cleaning 15,000, free from 7 nights.

Christmas package 24 to 28 December, 4 nights 180,000.

Dogs 2,000 per night, maximum two, up to 15 kg.

Booking of 7 nights gets 1 night free.

Saturday to Saturday in high season.

Row	Captures
season	Date range, nightly rate, guests included, extra guest fee, cleaning fee and when it is waived
holiday	Fixed night count and a package total
pet fee	Per-animal rate, maximum animals, maximum weight
discount	Type `night_bonus`: 7 nights required, 1 night free
rule	Arrival and departure day, minimum stay

Six sentences become five rows, with every ambiguity resolved on the way. The model returns this:

Migrating from WordPress to Hugo: Speed, Security, and GitHub Pages

Wed, 03 Jun 2026 14:00:00 +0000

The WordPress setup behind this blog kept causing trouble: security issues, breaches, and a constant stream of updates for core, theme and plugins. For a site of articles that almost never change, that was too much upkeep and too much risk.

So I moved it to Hugo and now serve it as flat files from GitHub Pages, for free. This post covers why, how the SQL dump became Markdown, and the deployment setup.

Building a Self-Hosted Personal Dashboard with Glance

Wed, 03 Jun 2026 13:00:00 +0000

I used to start the day by going through the same handful of browser tabs: weather, calendar, news, markets, YouTube. I wanted all of it on one screen instead.

Glance is an open source, self-hosted dashboard. It ships as a single binary with no database, and the whole page (widgets, data sources, refresh intervals, layout) is declared in one YAML file, glance.yml.

My config has grown to about 730 lines, plus roughly 2,000 lines of PHP in the proxy layer that feeds the parts Glance cannot fetch on its own. This post covers what is on the screen, where the data comes from, and why the PHP layer exists at all.

A 7.5-inch E-ink Weather & Calendar Dashboard Powered by ESP32

Wed, 03 Jun 2026 12:00:00 +0000

A dedicated weather and calendar display for the wall: a Waveshare 7.5" three-colour e-ink panel driven by an ESP32. It holds its image with no power, has no backlight, and reads the same PHP proxies as the Glance dashboard. It refreshes every two hours.

Renderer output before it is packed into a bitmap. The real panel shows Hungarian; labels here are translated and the calendar entries are placeholders.

Air quality monitoring IoT – Arduino & sensors connected to a Raspberry Pi

Sat, 26 Aug 2017 16:42:00 +0000

I was very interested in monitoring the surrounding air’s quality, so I built a box that monitors Particulate Matter (PM), gas concentrations, temperature and humidity. The historical data is persisted and accessible via the network – translating to a typical IoT use case. The box built is based on Arduino with its sensors and a RaspberryPi. Arduino is a great platform for electronics prototyping with a convenient development IDE – it allows to connect sensors to the board and share the data via multiple interfaces (such as serial port, attached SD card, network, etc.). The networking and data persistence are solved via a Raspberry Pi Zero W, providing SD-card based data plotting, having a host and guest Wifi network for data access and data sharing with cloud providers.

IT support for project management processes

Sat, 26 Aug 2017 13:07:00 +0000

I co-authored lecture notes on IT support for project management processes e-learning book.

For a full-fledged overview check the lecture notes of the IT support for project management processes e-learning course authored by Lajos Jenő Fülöp Dr. and Peter Kortvelyesi. The course is published at the pages of the University of Szeged.

The objective of the course is to provide an overview on the IT tool and system support of modern project management. As for complex projects it is inevitable to have a collaborative tracking system, the authors list the main tools and their usage. The aim is to build on previous project management experience and given knowledge by providing an overview on widespread project management methodologies including:

Data Governance in the Big Data and Hadoop world - Whitepaper

Fri, 25 Aug 2017 16:11:15 +0000

My full whitepaper — Download PDF

Amidst vast data lakes and a high velocity of incoming data, enterprises are finding that despite the procedures, policies, and systems that are already in place, their data governance frameworks still aren’t delivering the insights needed to drive smart business decisions. Pain points such as privacy and security, appropriate permissions, and labor-intensive data mining processes are challenging general data governance frameworks that do not allow for flexibility or agility. Hadoop can be an excellent framework to support a large data storage repository with immense processing power. The key to success is understanding how to unleash Hadoop’s powerful storage capabilities. This white paper reveals:

Configure Apache Kylin with ODBC to work with MS PowerBI

Mon, 09 Jan 2017 21:13:16 +0000

PowerBI and Kylin - reporting from Hadoop via ODBC

This article discusses how to set up an ODBC interface for Kylin to work with Microsoft PowerBI. See previous article on the detailed dataset, environment setup, on what Kylin is and how to create a cube in Kylin. For the tutorial’s purposes, we will analyze the previously loaded flight delay data with Hadoop, Hive, HBase, Kylin, Kylin ODBC connector and MS PowerBI as an interface. With PowerBI, Microsoft provides a capable and simple BI tool for free (desktop version) - as competition heats up, this is a great strategy to gain some market share.

Evaluation of Apache Kylin 1.5.4.1 with HDP 2.5, performance comparison w Hive

Mon, 09 Jan 2017 20:24:45 +0000

Apache Kylin is a data cube solution on top of Hadoop providing an ODBC interface for BI tools. OLAP cubes boost performance for analytics via using a subset of data, enriched with pre-calculations on specific dimensions of interest. It enables loading dimensions from a Hive data source, therefore accelerating BI tool access via pre-calculating data and adding it to HBase. In our example we have a large dataset of flight information:

Top 7 challenges of building a data lake

Sat, 07 Jan 2017 18:26:42 +0000

While from the technical perspective, deployment, management and provisioning tools are available to quickly set up a Hadoop cluster, introducing it to the organization is a tough task. Here are the biggest challenges of building a data lake 1. Understand the purpose and limitations of the technology

Hadoop environment provides a vendor independent tool-chain for data storage, management and analytics in a scalable fashion - it can manage infinite amount of data and with add-ons makes real-time and data science use-cases available for the enterprise
Many organizations do not have Big Data. They just load their RDBMS content and get surprised that a 20 node enterprise Hadoop setup’s performance is subpar compared with a single PostgreSQL instance. Sure, because the purpose and capabilities are different - we should compare apples with apples after all
In 95% of the cases stakeholders think that this new technology will substitute their painful proprietary data environment of RDBMSs. It is a highly distributed environment so the case is that it is never capable to substitute proprietary data systems. But it can live well beneath those providing extreme value. The purpose of the data lake is different. Ingest data, all of your data, and deal with portions of it without costly ETLs to move it from here to there. Refine your data system, be it processing, reporting or prediction models by slowly adding more datasets.

2. Security

Performance test of Pig vs Hive with code examples

Fri, 08 Aug 2014 17:40:07 +0000

Performance testing high level Hadoop query languages with example scripts. Analysis of NOAA weather data: Western-European weather stations from 1980 to 2014, daily dataset of temperature (tmin and tmax) and precipitation data (prcp). Dataset is a structured table, non-existent measurement cells are filled with -9999. Example: STATION,STATION_NAME,DATE,PRCP,TMAX,TMIN GHCND:NLE00109300,STAVENISSE NL,19800101,53,-9999,-9999 GHCND:NLE00109300,STAVENISSE NL,19800102,21,-9999,-9999 GHCND:NLE00109300,STAVENISSE NL,19800103,133,-9999,-9999 … GHCND:NLE00109202,MARUM NL,20080602,0,-9999,-9999 GHCND:NLE00109202,MARUM NL,20080603,36,-9999,-9999 GHCND:NLE00109202,MARUM NL,20080604,4,-9999,-9999 … Data size: 1 Gb / 4 Gb / 8 Gb (the same 333 Mb data file replicated 3 / 12 / 24 times) HDFS block size: 128 Mb Platform:

Create a Hadoop Cluster easily by using PXE boot, Kickstart, Puppet and Ambari to auto-deploy nodes

Fri, 08 Aug 2014 17:12:52 +0000

This tutorial is to showcase unattended and automatic install of multiple CentOS 6.5 x86_64 Hadoop nodes pre-configured with Ambari-agents and an Ambari-server host. After configuring automatic install of bare metal (No OS pre-installed) nodes, deploying a Hadoop cluster will be a matter of clicks. The setup uses:

PXE boot (for automatic OS install)
TFTP server (for PXE network install image)
Apache server (to serve the kickstart file for unattended install)
DHCP server (for assigning IP addresses for the nodes)
DNS server (for internal domain name resolution)
Puppet-master (for automatic configuration management of all hosts in the network, Ambari install included in Puppet manifests)
Ambari-master and agents (for managing Hadoop ecosystem deployment)

The setup assumes that the nodes are on the 192.168.0.0/255.255.255.0 network, the master is on 192.168.0.1 and its hostname is bigdata1.hdp The domain for the network server by the configuration is hdp and the clients are named as bigdata[1-254].hdp Download all files here (configuration files, PXEBoot Linux image, Kickstart file and custom script for adding a node on the master).

Analysis tutorial with Tableau Desktop

Sun, 11 May 2014 17:09:31 +0000

Tableau Desktop supports visual analysis and data discovery, converts the raw information to easy to understand graphical format with interactive charts. No coding is required to create rich visualization. Tableau Business Intelligence toolset has a Desktop, Server and Cloud version (none open-source products but as good as worth a post on the open-bigdata blog). In this post I check its Desktop evaluation version that lets us connect to many data sources (including Hadoop, MySQL, Excel, Text, …). I will use the same Weather.Csv as in the Hadoop analysis tutorial.

Setting up a firewall to secure a Hadoop cluster's network with Shorewall

Fri, 14 Mar 2014 13:40:59 +0000

Shorewall is a tool to configure Linux inbuilt IPTables in an easy and understandable way. Assume we have a basic setup: Lan | Firewall with Proxy server | Internet A secure setup is to:

ACCEPT HTTP(80) and HTTPS(443) from LAN to NET
ACCEPT special services ports from specific LAN to NET (like e-banking)
ACCEPT only the needed FW services from LAN to FW - SSH(22) and MAIL(25,443,993,…) to FW (if mail server is on FW)
ACCEPT only the needed FW services from NET to FW - SSH(22) with IP restriction (and SSH key) and MAIL(25,443,993,…) to FW (if mail server is on FW)
- SSH port can be changed
LOC to LOC connections are not possible to be governed by FW, therefore all allowed
REJECT any incoming connection (other than above) to LAN or FW

Shorewall configuration - with some additional examples on the syntax and rules:

Security of a Hadoop cluster

Fri, 14 Mar 2014 13:30:30 +0000

Hadoop is not only a data processing but a data warehouse solution. When we are talking about high amount of data we are talking about business value (the basis of the services) for a company. As Hadoop uses many nodes, network devices all the equipment must be included in a security plan - no weakest links are allowed. To secure our assets the following tools and approaches can be used.

Data mining from webpages with Python Mechanize Browser Automation - a Big data tutorial

Thu, 13 Mar 2014 13:31:28 +0000

Data mining from the web with Python Mechanize Browser Automation - Big data tutorial

In this example the Python Mechanize package is used for browser automation - Selenium is much more feature rich (and is also a bit more difficult to use) and is to be used when feature-rich Javascript and Ajax website data mining or automated test case setup is to be built. A Selenium browser automation example can be found here. This rich commented Python Mechanize Browser Automation example does the following:

Data mining from webpages with Selenium Python WebDriver Browser Automation - a Big data tutorial

Thu, 13 Mar 2014 13:30:28 +0000

Data mining from the web with Selenium Python WebDriver Browser Automation - Big data tutorial

In this example the Selenium web test automation framework uses Firefox for browser automation - Selenium is much more feature rich (and is also a bit more difficult to use) than Python Mechanize - having an example here. It uses a full suite of a web browser, so as an advantage Javascript and AJAX rich webpage parsing can be automated - in such cases it has to be used instead of Python Mechanize for data mining and auto testing purposes. This rich commented Selenium Python WebDriver Browser Automation example does the following:

Send to Kindle via email - using browser automation

Thu, 13 Mar 2014 13:29:27 +0000

I often read web articles on the Kindle e-reader for less eye strain. In order to achieve that, I use the Send to Kindle browser extension, maintained by Amazon. In the past few years many Kindle publishing services just came and went, so the official Send to Kindle was always the most stable one to use. However, one important feature is not supported, sending articles from mobile or tablet - by just sending the URL to an email address, not an article’s content attached, processable by the @free.kindle.com email addresses. Here, my script is presented, which could run on any machines using Python to check an email box with unread emails containing URLs and send the articles’ content to the Kindle e-reader.

Weather data analysis and visualization - Big data tutorial Part 1/9 - Fundamentals

Thu, 13 Mar 2014 13:29:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 1/9

Analysis fundamentals

This experiment analyses the weather changes in the Carpathian Basin from 1900 to 2014 using a dataset of daily measurements of weather stations nearby our point of experiment. High emphasis is put on interactive visualization, as it is inevitable to make the comprehension of information easy. Big data analytics is gaining a lot of attention. This tutorial is to provide an overview on some open-source tools capable of supporting distributed analysis on huge datasets. Nonetheless it analyses the weather trends of the Carpathian Basin of Central Europe and provides an easy to use visualization on the changes.

Weather data analysis and visualization - Big data tutorial Part 2/9 - Dataset

Thu, 13 Mar 2014 13:28:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 2/9

Preparation - Dataset

Weather data from NOAA — National Climatic Center — is accessible using a great toolset that lets you select your area of interest on a map interactively. Custom dataset is downloadable via the NOAA map tool.

The detailed experiment dataset used in this tutorial is downloadable here: Weather.zip.

Weather data analysis and visualization - Big data tutorial Part 3/9 - Environment

Thu, 13 Mar 2014 13:27:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 3/9

Preparation - Analysis Environment

As the analyzed data is relatively small to get it processed on a single machine, I have spared some time to set up a new Hadoop cluster – I have administrative access to a smaller cluster of regular PCs chained into a Hadoop cluster but this one was reserved the time I made the experiment. Anyway, analyzing a small dataset with some big data tools is resulting in the same development efforts as analyzing Petabytes of data on a cluster of thousands of machines – only it takes less CPU time – one can still learn the basics on small datasets.

Weather data analysis and visualization - Big data tutorial Part 4/9 - Hadoop & Pig

Thu, 13 Mar 2014 13:26:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 4/9

Analysis with Hadoop and Pig

Igniting Hortonworks Hadoop environment

Starting the virtual machine
Logging in from the host machine’s browser: 127.0.0.1:8888
- Log in with credentials supplied on login screen: hue / 1111
A straightforward menu is given to manage stored files (HDFS), Pig or Hive analysis scripts and to check the jobs that are running

Weather data analysis and visualization - Big data tutorial Part 5/9 - Visualizing: GIS & map

Thu, 13 Mar 2014 13:25:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 5/9

Data visualization - GIS, map based using Kartograph - Subsection 1/2

For the showcase purposes, I have used this Kartograph example to show bubbles with different colors and radius based on the yearly temperature and precipitation data (two datasets are in two different directories / diagrams). Kartograph is an open-source Python based map generator and JavaScript based web map illustrator. A stand-alone package to generate maps without using 3rd party applications like Google Maps. If you generate your map once you could avoid any service level or API changes regarding your map.

Weather data analysis and visualization - Big data tutorial Part 6/9 - SED example

Thu, 13 Mar 2014 13:24:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 6/9

Manipulating output data with the Linux SED command - SED example

This Kartograph tutorial uses JSON as data format, so I needed the same format for my own data that the tutorial uses - example:

[{"Weather": "ARAD RO", "ll": [21.35, 46.1331], "1882": 742.0 … {"Weather": "MURSKA SOBOTA RAKICAN SI", "ll": [16.2, 46.7], … "1962": 931.0}]

And the resulting data from PIG is not compatible with it, as it has different markup, presented here:

Weather data analysis and visualization - Big data tutorial Part 7/9 - Visualizing: GIS & map part 2

Thu, 13 Mar 2014 13:23:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 7/9

Data visualization - GIS, map based using Kartograph - Subsection 2/2

The two examples showcased in the previous part:

I’ve combined the two together:

Using the files from the animated symbols of the first one and
Added tooltip data by altering the addSymbol function in the JS section of index.html
Tooltips containing PRCP data for each decade

symbols = map.addSymbols({
type: $K.Bubble,
data: weatherStations,
location: function(d) { return d.ll },
attrs: symbolAttrs,
tooltip: function(d) { return $.trim(d.Weather)+<br>Rainfall 1900: ' +d['1900']+' mm'+ '<br>Rainfall 1910: ' +d['1910']+' mm'+ '<br>Rainfall 1920: ' +d['1920']+' mm'+ '<br>Rainfall 1930: ' +d['1930']+' mm'+ '<br>Rainfall 1940: ' +d['1940']+' mm'+ '<br>Rainfall 1950: ' +d['1950']+' mm'+ '<br>Rainfall 1960: ' +d['1960']+' mm'+ '<br>Rainfall 1970: ' +d['1970']+' mm'+ '<br>Rainfall 1980: ' +d['1980']+' mm'+ '<br>Rainfall 1990: ' +d['1990']+' mm'+  '<br>Rainfall 2000: ' +d['2000']+' mm'+ '<br>;Rainfall 2010: ' +d['2010']+' mm'+ '<br>Rainfall 2013: ' +d['2013']+' mm'; }
});

Looking at the map, some tooltip refinement was needed as those were out of the page, due to their large size: As I’ve defined a big tooltip to show I’ve changed the positioning by altering kartograph.min.js Changing this line kartograph.min.js

Weather data analysis and visualization - Big data tutorial Part 8/9 - Visualizing: HTML charts

Thu, 13 Mar 2014 13:22:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 8/9

Data visualization - Interactive HTML5 charts with Flot JS Library

The two best tools for HTML5 charting I have checked out are

JQplot
Flot

I’ve chosen Flot’s interactive chart with switchable diagrams and came to an idea that it would be nice to also have trend lines for the charts and used it as an example. Both tools use proprietary data sets, instead of manipulating their example parsers, I’ve chosen to alter my data to comply with the provided examples. After altering the files with SED, so as above, I have come to a problem — Hadoop is a parallel system, all processes are analyzing portions of the dataset, resulting dataset is unordered and if it is complex, it can’t be ordered with big. So I had data like:

Weather data analysis and visualization - Big data tutorial Part 9/9 - Results

Thu, 13 Mar 2014 13:21:16 +0000

Tutorial big data analysis: Weather changes in the Carpathian-Basin from 1900 to 2014 - Part 9/9

Results of the analysis

The two resulting artifacts from the analysis are presented here.

The same methods can be used to mine and visualize temperature change information too.

Data security on Android

Thu, 13 Mar 2014 13:13:23 +0000

The biggest data generator is in our pockets - so securing it to protect privacy is the very first step to do before starting to use a device. Steps to secure Android:

Remove all non-needed apps
- Acquire root permissions on your phone (phone specific, use XDA forum)
- Use Titanium Backup
  - Backup all your data
  - Freeze or uninstall all unnecessary programs
    - Remove all which is vendor specific (Samsung, Sony, MIUI fork, …)
    - Remove all Google packages
- Disable all unused or privacy wise dangerous services with Disable Services
  - If some vital functions are not working you can re-enable it
  - Regularly, apps with disabled services still work (e.g. disabling LocationAccess for Facebook app)

Secure Linux based environment - communication for SMEs

Thu, 13 Mar 2014 13:11:16 +0000

Data governance procedures include the management of communication assets (e-mail, instant messaging, phone calls), which are considered as business secrets. While small enterprises should make calculations on using the cloud as the price benefit might be worth the negative impact on data management and ownership, a medium/large sized company is suggested to use own solutions with own security and data backup policies. Hereby - in brief - I present an exemplary cost-effective, secure setup for small to medium enterprises own infrastructure based on encrypted communication and messaging storage with data security and backup in mind. This is a high-level overview: data governance practices, backup policies, disaster recovery, update procedures, security testing, physical security, etc. are to be defined. Linux based servers: CentOS or Ubuntu Office 1: ~100-200 users

Set up Owncloud for personal data management and Android sync

Thu, 13 Mar 2014 13:09:25 +0000

If privacy is a concern and you’d be feeling better to host your own CalDAV and CardDAV (calendar and contact) data but still would like to access it from your smartphone and web, Owncloud is a straightforward open-source tool to use. Remark: hosting your email would only be a possibility if high availability server hardware and internet connection is available for use.

Register a cheap domain with Namecheap and configure DynamicDNS
Add this script to CRON to regularly update Namecheap on your IP address
- Do this on the Internet interface of your host or OpenWRT router
- Use this script

urllib2.urlopen("http://dynamicdns.park-your-domain.com/update?host=%s&domain=%s&password=%s" % ("subdomain", "yourdomain.com", "hash.by.namecheap..82d9365e23be655"))

Set it in crontab

crontab -e

#Add this to run IP update script every 0:05

        5 0 * * * /path/to/script

Use an Ubuntu Server (14.04 64)

#Install Owncloud from Repository
cd ~
wget http://download.opensuse.org/repositories/isv:ownCloud:community/xUbuntu_14.04/Release.key
sudo sh -c "echo 'deb http://download.opensuse.org/repositories/isv:/ownCloud:/community/xUbuntu_14.04/ /' >> /etc/apt/sources.list.d/owncloud.list"
sudo apt-get update
sudo apt-get install owncloud
sudo apt-get install mysql-server
#Define MySQL password

#Create MySQL table
sudo mysql -u root -p #DefinedPwHere
	CREATE DATABASE owncloud;
	GRANT ALL ON owncloud.* to 'owncloud'@'localhost' IDENTIFIED BY 'database_password';
	exit

#Set file permissions
sudo chown -R www-data:www-data /var/www/owncloud
find /var/www/owncloud -type d -exec chmod 750 {} ;
find /var/www/owncloud -type f -exec chmod 640 {} ;

#Enable Apache SSL
sudo a2enmod ssl
sudo a2ensite default-ssl
sudo service apache2 reload
sudo a2enmod rewrite

#Generate SSL certs
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/apache2/ssl/apache.key -out /etc/apache2/ssl/apache.crt

#Modify Apache configuration
sudo nano /etc/apache2/sites-available/default-ssl

#Add this to the file being edited with Nano
<IfModule mod_ssl.c>
    <VirtualHost _default_:443>
        ServerName YourServerName
        ServerAdmin webmaster@localhost
        DocumentRoot /var/www
        <Directory />
            Options FollowSymLinks
            AllowOverride None
        </Directory>
        <Directory /var/www/>
            Options Indexes FollowSymLinks MultiViews
            AllowOverride None
            Order allow,deny
            allow from all
        </Directory>
        ErrorLog ${APACHE_LOG_DIR}/error.log
        LogLevel warn
        CustomLog ${APACHE_LOG_DIR}/ssl_access.log combined
        SSLEngine on
        SSLCertificateFile /etc/apache2/ssl/apache.crt
        SSLCertificateKeyFile /etc/apache2/ssl/apache.key
        <FilesMatch ".(cgi|shtml|phtml|php)$">
            SSLOptions +StdEnvVars
        </FilesMatch>
        <Directory /usr/lib/cgi-bin>
            SSLOptions +StdEnvVars
        </Directory>
        <Directory /var/www/owncloud>
            Options Indexes FollowSymLinks MultiViews
            AllowOverride All
            Order allow,deny
            Allow from all
            # add any possibly required additional directives here
            # e.g. the Satisfy directive (see below for details):
            Satisfy Any
        </Directory>
    </VirtualHost>
</IfModule>

sudo service apache2 restart

Access your Owncloud at http://host/owncloud
- Follow install steps, use MySQL, owncloud as database and use your database password as root or owncloud user
Use your Owncloud
Configure your router to forward requests of yourdomain:443 to yourowncloudhost:443 to access Owncloud from the Internet
Configure mobile device sync
- Use DAVdroid for your Android phone
  - CalDAV: https://yourowncloudhost/remote.php/caldav/
  - CardDAV: https://yourowncloudhost/remote.php/carddav/
- As most likely a self-signed certificate is used, set it up with CADroid accessible from FDroid market

E-learning framework, on-line tests

Thu, 13 Mar 2014 13:08:56 +0000

I’ve been involved in lecturing courses on IT support of Project Management and Big Data at institutions during the last years. As there have been many participants I’ve been looking into solutions to facilitate the homework hand-in and evaluation process and the automated assessment of final tests. The main idea was to support me and not me supporting the system - so an out-of-the-box working tool was preferred. After some tries on automated test creation with Limesurvey - which is a great tool for survey creation - I found Moodle which supports not only a lot easier test creation but the whole e-learning process. The tool is open-source, has great documentation and great functionality - which of only a few pieces I use regularly. So to share news with a specified group, assign homework with deadlines and create automated, shuffled tests - the use-cases I need - it does all. Let’s install it on an Ubuntu 14.04 Server. In terminal:

A multi-tiered Big Data warehouse & processing facility

Wed, 12 Mar 2014 18:17:16 +0000

The article details an exemplary setup for a multi-tiered data warehouse and processing facility using Hadoop and batch type of data analysis. Requirements in brief:

High peaks on data acquisition (e.g. financial transactions of yearly festivals, R&D facility with images or videos flowing in only at the time of experimenting)
- Velocity is high at peak times, Gigabytes / second for short time (max. few hours / day)
- Data to be quickly accessed for data processing for some weeks after acquisition
High volume of data to be stored for long-time
- Data to be accessed some years after acquisition, access can be delayed by days after request

The below setup details 3 tiers:

A Big Data Course

Wed, 12 Mar 2014 09:26:36 +0000

The published diagram is to detail a setup of a Big Data Course.

Fundamentals on databases (SQL and NoSQL), statistics (the R framework) and graph databases
The focus is on the Hadoop eco-system and its programming paradigm, MapReduce
MapReduce is available to be used with easier to master high level query languages like Pig and Hive
While Hadoop is for batch processing there are other usage areas:
- Real-time data access by HBase NoSQL daemon
- Fast but lower data volume processor, Spark
- Machine learning framework that can be run on top of Hadoop: Mahout
To be able to use these tools a well built, secure cluster is to be planned and developed, then operated securely
After the data analysis is done, final steps of visualization are detailed - to make an impact by using the achieved analytic results

A derivative of the course is held at the University of Szeged.

Environment setup for big data analytics

Tue, 04 Mar 2014 15:18:50 +0000

This article covers basic tools and technologies to use when conducting the first steps on big data analysis.

Linux as the base OS
- Debian
- Ubuntu
- RedHat
- CentOS
For basic data processing:
- Bash shell: environment for running multiple command-line Linux tools for data manipulation
  - Comes pre-installed with Linux; check Reference manual for usage
  - Bash might be not the default Linux shell, see how to switch to it
  - Learn Bash by examples
  - Most important Linux Commands and phenomena to master for data manipulation:
    - cat - output/concatenate a stream / file
    - head, tail - show the head or the tail of a stream / file
    - grep - to filter streams / lines of a file
    - sed - to manipulate streams / lines of a file
    - Regular expressions - used in a wide set of Linux tools
      - Explanation and tutorials
- AWK - simple data reformatter with compact coding features
  - The GNU Awk User’s Guide
  - AWK Tutorials
- Python - easy to learn, effective programming language with a huge amount of libraries available for various tasks. Great for data manipulation used from the command line.
  - Python Download
  - Python Documentation
  - Python Beginners Guide
  - Python Tutorials
And the big data analysis framework chosen based on the type of data analyzed. For the first step tutorials our suggestion would be:
- Hadoop, single cluster setup (can be downloaded pre-installed to a virtual appliance)
- Java based MapReduce programs
- Pig MapReduce query language