Making your Flukso log to a database

I wondering if this tutorial, made in 2009 for the first version of the FLM, is still valuable with the today’s version 02a or 02b ?
Here is the link of this tutorial i found:
http://www.vanheusden.com/misc/blog/2009-10-11-flukso.php
It seems interesting but perhaps not useful for non Linux users ...

gebhardm's picture

With the REST and MQTT protocol interface of the FLM v2 you don't need any adaptations on the FLM anymore; the tutorial describes on how to send the syslog entries to a remote destination, how to grep via a perl script the received messages and write the determined values to a database - this is "legacy", but should still work "through the breast in the eye" still.
It's much easier, also in "non-Linux" environments via MQTT - see https://www.flukso.net/content/mqtt-mq-telemetry-transport - install yourself a MQTT-client and receive your data; then go on with a database:
Using Python and MySQL should be possible also in M$; you may have a look at https://github.com/gebhardm/energyhacks/tree/master/RaspberryPi - this is made for a RasPi, but with Python and mySQL installed this should be invariant to the operating system used; a bit of transfer knowledge should be available in any case...

gebhardm's picture

BTW: Using Python and mySQL should be possible also on "any better" NAS drive... (at least for Synology drives it works also)

Fluc's picture

I only use Ubuntu and derivates of it, so i think one day i give it a try to see real time readings more nicely then in the Terminal where i already use Mosquito once and a while. But i am not a programmer :-( ...
Also, thanks for this reply

gebhardm's picture

https://www.flukso.net/content/mqtt-nicely-designed-gauges is the starter to provide a convenient local graphical solution right from the FLM; the FLM's resources are sufficient, at the moment just time is limited (I wanted to get a little further on this during winter vacation; sadly my newly bought books on Node.js and Javascript are still untouched and cross-compiling openWRT applications is still lacking the proper variable "displacement") - so, hopefully you don't have to be a programmer to get a modular solution (one fine day :-)) - I should think about a sabbatical.
Maybe one comment to the above: Running a PC to log the FLM's data I regard cannons on sparrows; with a RasPi you get a convenient environment to easily log and store data down to a second level... For this you actually have "just to configure mySQL, install Python and the corresponding MQTT client and run the flm_mqtt_to_db.py with sudo nohub python flm_mqtt_to_db.py" - I'll add a small "how to" hopefully soon...

Fluc's picture

Yes, that would be wonderful :D

icarus75's picture

Markus, that sabbatical would be very useful indeed. :-)

Fluc's picture

Thanks Markus, i shall take some time off, to try this, +1

Fluc's picture

Ok, thank you for letting us know.
One day perhaps, the RaspberryPI shall appear on my wish-list.

gebhardm's picture

With respect to the RasPi - I use it "just" as a very efficient and low consumption computing environment; the scripts should thus run on any linux machine as well if the corresponding requirements on database and scripting environment are met. So, please take my reference to the RasPi just as that, a reference...

hob's picture

Does anyone know if it is possible to obtain counter values via the REST API interface described in the Flukso manual, in addition to the gauge values? I know that both are published via MQTT and I am able to pick them up, but I am finding my client to be unreliable (RasPi running openhab) and so I'd like to be able to pull counter values from the REST API as a fall back.

I'll also try to resolve the reliability problems with openhab but I think that is an openhab/Raspi issue as the Flukso just keeps sending out the data seemingly very reliably!

gebhardm's picture

@hob - corresponding to the description of the valid units of the REST API in https://www.flukso.net/files/flm02/manual.pdf there is no transfer of the counter values. Maybe clone the fls-repo and do a search if you find any evidence on watthours (as we know that documentation is not the primary strength of this very product, no offense, and of course, code is the best kind of docu ;-))

hob's picture

@gebhardm thanks for the pointers! So is TMPO already installed on my Flukso? How can I get the data over to my Raspi?

gebhardm's picture

If you are on 247, yes. The data is stored in /usr/share/tmpo - SCP it to the RaspPi or do it the Python way, described by icarus75 in https://github.com/flukso/tmpo-py

hob's picture

Found it - and managed to gzip -d it to a human readable form. I'd like to get this in to a mysql database so I'll look at the python code.

In the meantime I also installed your flmlocal directly onto the Flukso - cool - thanks!

gebhardm's picture

Transferring the TMPO object back to its initial content is exemplary done here in Javascript: https://github.com/gebhardm/flmlocal/blob/master/www/scripts/controllers... - line 156 and following:

  • get time and counter start values from head, where also all configuration information is found
  • rebuild the time and counter values by taking the TMPO increments one after another
  • et voila, you have the original time series

Thus, fairly easy also without all the Python "hoo-ha" (SciPy, Pandas, etc.)...