Edit: 29 June 2014: This guide is slightly out of date. There is now a combined method. However this How-To is still relevant if you would like to run the scripts separately/or just one or two. (for instance, I am mostly interested in the database logging). If you want to run the combined method on Ubuntu, please follow the new instructions in the ReadMe file on GitHub
This guide below should get the multiple scripts that do not need configuration by Markus Gebhard up and running on you server. Pieces are pulled from his readme files on github and others are specific to a clean installation of ubuntu-14.04-server. You should still read all those files and forum topics because there is a lot of explanation and information in them. It may look like i know what I'm doing but don't let that fool you.
This guide is written with the intent that you can just copy and paste as you read through. From personal experience I know this is an easy way to get in over your head. I said above that this is based on a clean install, and what I mean is that once you have ubuntu installed and you are looking at a command prompt this is the first thing you do. If you have other services listening at port 1080 or 8080 these scripts will not work. (you can of course modify them to run on whatever port you desire)
With the hopes that maybe this guide will help get this project installed on more devices and more folks hacking on it.
Environment:
Set up the environment/Get all the pieces in place
Update-
sudo apt-get updatesudo apt-get upgrade
Git-
sudo apt-get install git
mySQL-
sudo apt-get install mysql-serverYou will be prompted to set a root password
Tell MySQL to generate the directory structure it needs to store its databases and information
sudo mysql_install_db
Run a simple security script. You can change your root password for mySQL here or say no to the first question and answer the rest of the questions yes.
sudo mysql_secure_installation
Set up the mySQL db to have all the tables and settings that the scripts will expect
mysql -u root -pcreate database flm;use flm;create user 'pi'@'localhost' identified by 'raspberry';grant all privileges on flm.* to 'pi'@'localhost';flush privileges;set password for 'pi'@'localhost' = password('raspberry');quit
node.js-
You may need to sudo apt-get install software-properties-common if the add-apt-repository below fails.
sudo add-apt-repository ppa:chris-lea/node.jssudo apt-get updatesudo apt-get install nodejs build-essential libavahi-compat-libdnssd-dev
The scripts-
Install the scripts from Markus Gebhard using git below or download them as a zip file directly from github
cd ~git clone git://github.com/gebhardm/flmdisplay/
Panel:
cd ~/flmdisplay/panelnpm install mqtt socket.io mdns./panel.sh
Persist:
cd ~/flmdisplay/persistnpm install mqtt mysql mdns./persist.sh
Chart:
cd ~/flmdisplay/chartnpm install socket.io mysql./chart.sh
If you have gotten through without errors then everything should be up and running. If you want to double check try pgrep -a node This will list the process ID of each node instance and what the script name is at that pid. (useful if you want to kill -9 a specific process.)
You will want to know what the local IP address of your machine is. Try this if you are using wired ethernet (wlan0 if wireless)ifconfig eth0 | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1}' or just type ifconfig and read it yourself.
On another computer you should then go to the local address of your server to access the chart and panel.
Panel:
http://"local IP address":1080
Chart:
http://"locall IP address":8080
Remember the persist node is dumping data every second into the database. (even in the case of 0 data) Lots of info in there to manipulate. There are plenty of ways to get the data out if you want to do some presentation. Tab separated files can be dumped directly from mySQL. PHPmyAdmin can pop the data out in a spreadsheet readable file... etc some assembly required...
Size:
With 3 FLM and therefor 9 sensors recording data every second I am seeing the database become 97MB in about a day. I'd love some feed back regarding your experience with the speed of drive space consumption in your set up. (I plan on making more precise measurements) But just a heads up, you should plan accordingly, whether it be large enough drives or clean up scripts...
feedback and corrections appreciated.
haha! Well, looks like I hit save instead of preview...
Oh well, hope I caught all my typos. ;-)
Pietro, thank you for this nice tutorial.
I will look at it as soon a have some spare time to test it on a Ubuntu desktop, not a server.
Later perhaps a RaspberryPi should be fine as a low consumption server.
It Works! Thank you Pietro! Thank you Markus!
@Fluc
Should work fine on desktop.
you can check ahead of time to see if you ever installed anything that is listening on the ports we need with the command
netstat -tulpn(lists everything)Or you can filter out the noise and look specifically for 8080 and 1080
netstat -tulpn | grep :1080netstat -tulpn | grep :8080(should return nothing)
It works! Thanks for making it possible.
I used fresh 14.04 server on VM. I had to
sudo apt-get install software-properties-common
in addition to above instructions.
Very awesome post
Any ideas if this same principle could be used to do this on a Synology NAS? - I believe it runs unix and can run mysql/php easily :)
@adamcowin
The other question is, can you create a node.js environment on the NAS? If you can, then all you would need to run from here is the persist script and either write the php to pull the info off the database or find some one who has done that. However if you can get node.js up and running on the NAS then why not use the chart tool for your visualizations...
There is some relevant reference here but i suspect a generic search for using php to graph info that is in a database might be fruitful as well.
https://www.flukso.net/content/flukso-and-phpmysl-graphing-synology-nas
https://www.flukso.net/content/json
Since this site doesn't have built in search it is recommended to use the method
site:flukso.net [search terms]@Adam Cowin - I tried it with php first (database writing as well as graph display) but must admit that it is much more cumbersome and not straight forward in using websockets as it is with node.js... Therefore I still recommend to spend the 30Eur plus 8GB SDcard plus a small case plus power plug (total around 45Eur - or buy a kit for ~60Eur) in a RasPi model B and make yourself life much easier - but feel free to try it on a NAS (I won't do it)
Thanks a lot for the wonderfull tool, detailed documentation (on github), and clear walktrough (here)!
I tried to get it working on a RasPi B, with raspbian weezy OS (jan 2014).
The install procedure largely worked for me, but I had to include 3 additions (see A),
and one important roadbump (B) still remains: The persist script doesnt work as planned yet.
A) Install procedure additions:
I had to add following 3 commands:
sudo apt-get install software-properties-common python-software-propertiessudo npm config set strict-ssl falsesudo npm install mdns(in ./persist/ and ./chart directories)Reasons:
1) the raspberry didnt accept apt-add-repository, even after "sudo apt-get install software-properties-common". With python software properties, it did work.
2) The certificate used for npmjs.org has changed and it is not recognized by the bundled CA: Workaround is to not verify ssl certificate: "sudo npm config set strict-ssl false".
3) I got some errors with require('mdns'): "Model version mismatch, Expected 11, got 1").
It seemed that node-gyp couldnt access the dev dir "/root/.node-gyp/0.10.29". A sudo reinstall made a "local ./node-gyp temporary dir" and solved the error.
B) Critical roadbump remaining:
The "persist.sh"-script fails to fetch and store the data correctly (while the panel does show the 1-second data correctly, and the chart is accessible). The persist script can access the db and make the table, but does not populate it with readings.
Can someone give me a clue as what could cause (B) ? Or give pointers to tests/debugging options, that might point me in the right direction? Thanks! The install procedure and code digging was very educational already (which is the main goal of RasbPie, right?), but I assume I made some newbie errors along the way, so feel free to guide me if you know where I went wrong! FYI: I have 3 sensors attached to the FLM: One electrical, one water (pulse) and one gas (pulse) sensor.
The ./persist.sh output is O.K. and shows that there is a connect on the database and the FLM; have you installed the user 'pi' in the database (as the script relies on this - I bet, you did, otherwise I'd expect an error thrown)?
For debugging purposes you may add a "console.log" into the script, that will fill your screen on getting mqtt payloads:
As
would insist on not writing to the database, how do you look into the db in the first place? Use from the RasPi prompt
This should give you a hint if the table to write into exists and if it is filled. The script as such is just straight forwards in filling the db...
Regards, and thanks for using
Markus
With respect to
npm install ...- you actually should not do it per subfolder, but once in the home directory which is in the path of node to find its modules; I think it is rather cumbersome to keep modules up to date when they are redundantly deployed.Thanks for the pointers. I cannot access the device atm,
but you can find some feedback below already.
>> have you installed the user 'pi' in the database
yes. and since the script can make the table (and no errors occur), I assume that db connectivity is ok.
>>
I tried adding console.log('script in case gauge ') in that location yesterday, but it didnt show anything.
I assume the script doesnt reach that part of the code (if thats possible)?
I'll check the payload later today.
>> how do you look into the db in the first place?
Yes, like that. the table "flmdata" exist (was created by the scirpt), but it is empty.
>> "nmp install"
thanks, will do that next time :-)
Well, then the script seems to get no mqtt messages at all; you may check with "any" message subscribing to # and place a console.log into the corresponding section in analogy to using mosquitto_sub - the code is actually the same in the panel, so it "should" work the same (I experienced no issues with a fresh installed RasPi with the same wheezy version you use)...
Note: If there is any kind of error in a javascript routine, then my experience with conditions like "nothing happens" is, that the code is just neglected from the point where the error occurred... So, if you altered the code why so ever, try with a
git stashto go back to the original version taken from github and start over with the script execution - computer science is no science, so expect no absolute results ;-)I did some more tests now, and as assumed:
The script didn't get beyond the "mqtt CreateClient" subroutine.
In the mean while, I localised the real culprit :
a posterior node-gyp installation... *shame on me*
I assume that installing node-gyp AFTER "nmp installing the node modules (such as mqtt and mdns)
has caused the issue. And the obvious solution: a renewed sudo npm install mqtt,
Now the persist script works like a charm! (count >> 0)
Also the panel & Graph views work flawlessly now!
@Gebhard: Thanks a lot for the quick & in-depth support! You've earned a free beer, next time we meet! :-)
Hurra! :-) that may be an elektro:camp; as far as I understood Bart there should be one in Berlin in November (I never thought how difficult it is to get a seminar room when you are outside an institution that has such, but doesn't publically offer it - practically impossible; for a lot of money of course you'll get one)
Please note: I added a combined version of all scripts, so panel, chart and persistence, into one single instance; so alternatively to ./persist.sh, ./chart.sh, and ./panel.sh you may simply use ./flmdata.sh in the combined/ directory of https://github.com/gebhardm/flmdisplay
Hi All,
sorry for the delayed reply
@Gebhardm + @PIETRO SPINA
I had a quick look to see if node.js is able to be installed on the synology nas and it seems that it is indeed possible :D
Stumbled across this: http://forum.synology.com/enu/viewtopic.php?f=3&t=80868
but im quite a linux newb and most of this doesn't mean anything to me yet :(
I'll have a look as I also own a Synology drive; issue here is (for me) that I run it with time schedule, so not always on (writing db would mean also hd always on, what I regard as total inefficiency - option would be to put the db on an external USB stick, but that makes the installation even less simple)
@AdamCowin - if you like to try it, see also http://www.gridshore.nl/2011/04/04/installing-node-js-on-my-new-synology/ for getting node on a Synology NAS; I fear that even with node.js installed, the next hurdle will be installing the required modules; I am still not convinced if it is worth the effort (especially with respect to forcing the NAS into always-non-idle)
Attention: Due to an issue raised by "Ryton" I stumbled across Javascript oddities, especially on time and timezone handling. Javascript implementations in different browsers and even different versions of the same browser differ! The provided scripts run on "current" versions of Safari, Chrome, IE and Firefox (checked on PC Win8.1 IE/FF, Mac OS10.9.3 Safari/FF, Nexus 7 Android 4.4.4 Chrome and Firefox).
The chart does not work anymore on an old iPad 1 with iOS 5.1.1 as date.parse behaves differently. There is no common solution except for implementing a variant for each oddity (please don't expect that from me!) - you may alter the corresponding frontend code (.js part or file on time emit: date.parse w/ 'Z' and offset) so that especially timezones are handled correctly in your special case...
An updated version is available that allows the de-/selection of sensors in the chart as well as in a graph display that "may" replace this site's minute dash. See https://github.com/gebhardm/flmdisplay/tree/master/panel to get an impression ("markdown is cool") - the "panel" is a version using "just" the MQTT realtime information; the combined/ folder contains a variant that integrates also handling of persisted data from a database in a corresponding chart view.
i'm very interested in this way of displaying and storing data from the flukso. Is there someone who has experience with installing this on a debian nslu2 device ? Mine is still on debian squeeze running owfs to a RRD database for displaying 1wire data. I fear destroying my perfectly working environment.
Someone experience with nslu2 ?
Kind Regards,
Bart
The NSLU2's data looks not too promising performance-wise to run node.js plus "normal" database... You may try to just test the panel/MQTT visitor by installing node.js and its required modules via the squeeze package manager. If you use a "round robin db" you may adapt the persistence logic as it also just pushes data into the db - the chart then would require another type of SELECT (and logon, of course). Just try it out and post your findings (backup beforehand ;-)), the code is not really highly sophisticated...
Hi,
I'm already using the nslu2 with a 1wire sensor network which puts data into a RRD database by means of the OWFS filesystem. The following page describes the process : http://owfs.org/?page=garden
Just found following which I guess does the trick :
http://www.vanheusden.com/misc/blog/2009-10-11-flukso.php
thanks
We're talking MQTT here, not REST ;-)