{"id":2843,"date":"2017-08-31T00:19:17","date_gmt":"2017-08-30T22:19:17","guid":{"rendered":"https:\/\/blog.mi.hdm-stuttgart.de\/?p=2843"},"modified":"2023-06-08T15:43:08","modified_gmt":"2023-06-08T13:43:08","slug":"iot-with-the-raspberry-pi-node-red-part-2","status":"publish","type":"post","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2017\/08\/31\/iot-with-the-raspberry-pi-node-red-part-2\/","title":{"rendered":"IoT with the Raspberry Pi \u2013 Node RED – Part 2"},"content":{"rendered":"

As already stated in the introduction to our project, we decided to create a Cloud Foundry-Application in IBM Bluemix. We used the boilerplate called \u201cInternet of Things Platform Starter\u201d. Using this boilerplate Node Red is deployed initially.<\/span><\/p>\n

Node Red is a software tool for graphical dataflow programming. It was developed by IBM and is open source since 2016. Providing a browser-based flow editor it enables to wire together hardware devices, APIs and online services.<\/span>
\n
\nSince it was already deployed we decided to give it a try. Getting started by setting login credentials for your flow editor and following some beginner tutorials was quite easy. Most of those easy tutorials use the inject node to insert some sample data or the twitter node to receive tweets for a specified hashtag and use them as input data.<\/span><\/p>\n

Receive, display and save live data<\/span><\/h3>\n

Unfortunately we wanted to work with data from our Raspberry Pis instead of any tweets, which was not that intuitive anymore. After trying out some nodes using websockets seemed to be the easiest way to get started quick. WebSocket is a computer communications protocol, providing full-duplex communication channels over a single TCP connection. Since we only need to send data from our Raspberry Pi to the cloud application this protocol does not perfectly fit our needs and causes some overhead, but to get started it was ok. Configuring the websocket node like this: <\/span><\/p>\n

\"\"<\/p>\n

and sending the data from the Raspberry Pi to<\/span> https:\/\/use-your-data.mybluemix.net\/ws\/simple<\/span> using the <\/span>websocket client library<\/span><\/a>, worked out pretty good.<\/span><\/p>\n

The data comes in from the websocket as a String, which can easily be parsed to JSON by just passing it through the JSON node, as long as the String contains a valid JSON object. But debugging malformed JSON using error messages from the node red flow editor is not recommendable.<\/span><\/p>\n

As a next step we used the cloudant-out node to save our JSON data. Cloudant is an IBM software product based on the No-SQL CouchDB. It is integrated into the Internet of Things Platform Starter Boilerplate we used as default database that is already connected to the application and already contains a database called \u201cnodered\u201d. By configuring the cloudant-out node to use these defaults, saving our JSON data worked straightaway.<\/span><\/p>\n

\"\"<\/p>\n

So far our JSON data looked for example like this:<\/span><\/p>\n

{\n    \"timestamp\":\"2017-08-18 10:22:11.882596\",\n    \"device\":\"raspiMQ135\", \n    \"values\": {\n        \"CO2_NH4\": 3.85567e-05, \n        \"CO\": 1.80491e-12, \n        \"Alcohol\":0.00175228\n    }\n}<\/pre>\n
(It contains some strange values because the gas sensor was not yet calibrated correctly as described in Part 2 when the Node-RED application was built.)<\/span><\/p>\n
Since first of all we were only interested in the values from our sensor we needed to split our JSON object. The data passed through a node red flow is called a message (msg). By convention each message will have a msg.payload property containing the actual data. Other nodes may attach their own properties to the message, and they should be described in their documentation. A property that is quite common in some nodes is the msg.topic.<\/span><\/p>\n
We decided to use a function node that creates a new message for each key of our JSON object, writes the key to msg.topic (because we need this property later in our dashboard nodes) and adds the corresponding value to msg.payload. The function-node is like an all-round node because it allows any JavaScript code to be run against the messages that are passed in and then return zero or more messages to continue the flow. But therefore this node requires actual coding instead of just configuring a node.<\/span><\/p>\n
\"\"<\/p>\n
Now we have single messages containing timestamp, devicename or values. Since we are only interested in displaying our values we use a switch-node to differentiate between them.<\/span><\/p>\n
\"\"<\/p>\n
Looking at node\u2019s configuration makes it quite self-explaining: define a property to look at and define filtering rules that tell which way out of the node a matching message should take. Instead of equality it is also possible to filter based on regex and other criteria.<\/span><\/p>\n
Continuing only with our values we use again a function to split them and set the keys as msg.topic. The only difference to the function node before is that the node now has three outputs, one for each value, instead of sending all the messages through the same output.<\/span><\/p>\n
Finally we send our values to dashboard-nodes. These additionally installed nodes create a website under \/ui. To display our data especially chart- and gauge-nodes are useful. We mainly kept the default settings of these nodes and instantly got some suitable charts.<\/span><\/p>\n
But unfortunately reloading them on every new message they received is not implemented very browser-friendly and made stuck everything.<\/span><\/p>\n
The probably best solution for this problem would have been to aggregate some messages in the flow before inserting them into dashboard-nodes. Luckily there is an aggregator-node ready to install to our node palette. Unluckily this node does not keep the value set to msg.topic. Instead it overwrites it with an empty string or a new value you can specify, but you cannot reference the old value to set as the new one again (at least not in any common way that we tried out). So we decided not to use the aggregation node and ended up with this flow to receive, display and save our live data:<\/span><\/p>\n
\"\"<\/p>\n
The upper part of the dashboard page this flow creates looks like this (remember that the sensor is still not calibrated correctly here).<\/span><\/p>\n
\"\"<\/p>\n
Query saved data from cloudant db<\/span><\/h2>\n
 After saving our data to CloudantDB was that easy of course we also wanted to query that data again to display legacy data. Setting up the cloudant-in-node to query all data saved to our database so far was not very complicated either. But of course we needed some more specific queries which turned out to be not that easy anymore.<\/span><\/p>\n
First of all an index is required for any specific query. To create one you need to access the dashboard of your cloudant service, which can be found in the Bluemix dashboard. To retrieve legacy data we needed an index on our timestamp field and after reading some slightly unorganized cloudant docs we ended up with a very basic index. Testing it inside the cloudant dashboard returned the expected result.<\/span><\/p>\n
Index <\/strong><\/p>\n
{\n    \"index\": {\n        \"fields\": [\"timestamp\"]\n    },\n    \"type\": \"json\"\n}<\/pre>\n
 <\/p>\n
Query<\/strong><\/p>\n
{\n    \"selector\": {\n        \"timestamp\": \"2017-06-22 21:16:43.880295\"\n    },\n    \"fields\": [\n        \"timestamp\",\"device\",\"values\"\n    ]\n}<\/pre>\n
 <\/p>\n
Result<\/strong><\/p>\n
{\n    \"timestamp\": \"2017-06-22 21:16:43.880295\",\n    \"device\": \"raspiMQ135\",\n    \"values\": {\n        \"CO2_NH4\": 0.00000511376,\n        \"CO\": 1.61913e-14,\n        \"Alcohol\": 0.000426041\n    }\n}<\/pre>\n
But using the cloudant-in-node in our node-red flow did not work at all using our index and the same query. Even after googling around and trying out a lot of different configurations of the node and different format variants of the input including the query this node did not return the desired result. In most cases it returned \u201ccouch returned 400\u201d or other error codes instead, which makes debugging quite cumbersome.<\/span><\/p>\n
Finally we gave up on using the cloudant-in-node and to use a different way to query our database from within our node-red flow. Since it is also possible to query a cloudant database by posting the query to the URL https:\/\/[cloudantusername]-bluemix.cloudant.com\/[nameofdatabase]\/_find we managed to use the http-request-node to get the desired results from our database. The endpoint is protected through basic authentication by default and adding credentials therefore is quite easy \u2013 as long as you know them. But your cloudant credentials are not the same as your Bluemix credentials although navigating from your Bluemix dashboard to your cloudant dashboard does not require an additional login. Searching for cloudant credentials finding the username is the easier one. You can find it for example included to the link in browser when you are on your cloudant dashboard. To get a password for your account you need to navigate through your Bluemix dashboard to the cloudant service and there create a new entry under \u201cService credentials\u201d where you can set a password.<\/span><\/p>\n
So we ended up with the following configuration for our http node.<\/span><\/p>\n
\"\"<\/p>\n
And at first inserted only a static query using an inject node that sets our query as msg.payload and msg.topic to \u201cquery\u201d.<\/span><\/p>\n
\"\"<\/p>\n
Afterwards we used the same function as in our displaying live data part to split the data object of our result and map its values to its labels and finally we used simple text-dashboard nodes to display them.<\/span><\/p>\n
Later we also added a form-node to our dashboard to insert a timestamp to query for. We also added some very basic input validation using a switch node which checks the input for the correct format using a regex and then either an error message or a DB query is created using function nodes.<\/span><\/p>\n
So our final flow to query our database looked like this:<\/span><\/p>\n
\"\"<\/p>\n
Which resulted like this on our dashboard page:<\/span><\/p>\n
\"\"<\/p>\n
\"\"<\/span><\/h2>\n
Conclusion: Pros and Cons of Node Red<\/span><\/h2>\n
Of course this query feature is quite poor so far because you need to know the exact timestamp of an existing entry in the database otherwise the query won\u2019t return anything. Also the dashboards that display the live data are still pretty bad. But at this time we reached the point in our project when we decided to turn away from node-red and implement our project in real code instead.<\/span><\/p>\n
Most of the problems we experienced using node-red that lead us to this decision were already mentioned above. The major problems in short again:<\/span><\/p>\n
    \n
  • Some nodes do not really fit to each other like the aggregation-node that is not able to pass on a value set to msg.topic with dashboard-nodes that require these values<\/span><\/li>\n
  • Documentation of some of the nodes is pretty poor<\/span><\/li>\n
  • Debugging can become quite difficult since all you can do is use debug-nodes or console logs inside of function-nodes but it is not possible to see what is going wrong inside of a node<\/span><\/li>\n
  • Some things to do still require you to write code in function-nodes or maybe even to write your own nodes<\/span><\/li>\n<\/ul>\n
    All in all node-red seemed to us still poorly conceived and not as intuitive as it might look like at first sight.<\/span><\/p>\n
    Anyway graphical dataflow programming of course also has its benefits. The clear view of a program you can achieve when doing it right (by keeping flows as simple as possible) makes initial orientation and maintenance easy. It also becomes possible to talk about a program with domain experts that do not have any software development knowledge.<\/span><\/p>\n
    But at the moment node-red is not sophisticated enough therefore. It might fit one\u2019s needs as a prototyping tool yet.<\/span><\/p>\n
     <\/p>\n","protected":false},"excerpt":{"rendered":"
    As already stated in the introduction to our project, we decided to create a Cloud Foundry-Application in IBM Bluemix. We used the boilerplate called \u201cInternet of Things Platform Starter\u201d. Using this boilerplate Node Red is deployed initially. Node Red is a software tool for graphical dataflow programming. It was developed by IBM and is open […]<\/p>\n","protected":false},"author":481,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1,654,650,22],"tags":[],"ppma_author":[732],"class_list":["post-2843","post","type-post","status-publish","format-standard","hentry","category-allgemein","category-internet-of-things","category-scalable-systems","category-student-projects"],"aioseo_notices":[],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":22530,"url":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2022\/02\/28\/discord-monitoring-system-with-amplify-and-ec2\/","url_meta":{"origin":2843,"position":0},"title":"Discord Monitoring System with Amplify and EC2","author":"mk322","date":"28. February 2022","format":false,"excerpt":"Abstract Discord was once just a tool for gamers to communicate and socialize with each other, but since the pandemic started, discord gained a lot of popularity and is now used by so many other people, me included, who don't necessarily have any interest in video gaming. So after exploring\u2026","rel":"","context":"In "Allgemein"","block_context":{"text":"Allgemein","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/category\/allgemein\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2022\/02\/data-workflow.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2022\/02\/data-workflow.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2022\/02\/data-workflow.png?resize=525%2C300&ssl=1 1.5x"},"classes":[]},{"id":13157,"url":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2021\/03\/17\/monitoring-with-the-elastic-stack\/","url_meta":{"origin":2843,"position":1},"title":"Monitoring with the Elastic Stack","author":"Joel Beiter","date":"17. March 2021","format":false,"excerpt":"Netflix, stackoverflow and Linkedin are just a few of the companies activly using the Elastic Stack. Use cases include performance and security monitoring or log analytics. [1] The Elastic Stack consists of different products with the most popular ones being Elasticsearch, Kibana, Logstash and Beats. This article will introduce the\u2026","rel":"","context":"In "Allgemein"","block_context":{"text":"Allgemein","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/category\/allgemein\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2021\/03\/shards.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2021\/03\/shards.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2021\/03\/shards.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2021\/03\/shards.png?resize=700%2C400&ssl=1 2x"},"classes":[]},{"id":2151,"url":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2017\/03\/08\/of-apache-spark-hadoop-vagrant-virtualbox-and-ibm-bluemix-services-part-2-apache-hadoop-ecosystem\/","url_meta":{"origin":2843,"position":2},"title":"Of Apache Spark, Hadoop, Vagrant, VirtualBox and IBM Bluemix Services – Part 2 – Apache Hadoop Ecosystem","author":"bh051, cz022, ds168","date":"8. March 2017","format":false,"excerpt":"In our project we primarily implemented Spark applications, but we used components of Apache Hadoop like the Hadoop distributed file system or the cluster manager Hadoop YARN. For our discussion in the last part of this blog article it is moreover necessary to understand Hadoop MapReduce for comparison to Apache\u2026","rel":"","context":"In "Student Projects"","block_context":{"text":"Student Projects","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/category\/student-projects\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":22485,"url":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2022\/03\/16\/optimizing-practical-byzantine-fault-tolerance-in-wireless-iot-blockchain-networks\/","url_meta":{"origin":2843,"position":3},"title":"Optimizing Practical Byzantine Fault Tolerance in Wireless IoT Blockchain Networks","author":"Judith Pfannenschmidt","date":"16. March 2022","format":false,"excerpt":"Don't we all appreciate a little help in our everyday lives? \"Alexa, turn on the lights in the living room.\"Voice assistants, smart lights and other smart home devices are all part of the Internet of Things (IoT). Those and other possible applications of IoT devices increase continuously and with it\u2026","rel":"","context":"In "Internet of Things"","block_context":{"text":"Internet of Things","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/category\/scalable-systems\/internet-of-things\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2022\/03\/PBFT_fig_6_smallSized.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2022\/03\/PBFT_fig_6_smallSized.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2022\/03\/PBFT_fig_6_smallSized.jpg?resize=525%2C300&ssl=1 1.5x"},"classes":[]},{"id":12550,"url":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2021\/02\/27\/migrating-from-heroku-to-hetzner-achieving-scalability-with-docker-kubernetes-and-rancher\/","url_meta":{"origin":2843,"position":4},"title":"Migrating from Heroku to Hetzner: Achieving Scalability with Docker, Kubernetes and Rancher","author":"Mario Koch","date":"27. February 2021","format":false,"excerpt":"Dockerizing an existing application and deploying it in a Kubernetes Cluster via Rancher to achieve better scalability and cost minimization. Load Testing with Artillery, Monitoring with Prometheus & Grafana and GitHub Actions for CI\/CD were used in the process.","rel":"","context":"In "Allgemein"","block_context":{"text":"Allgemein","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/category\/allgemein\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2021\/02\/vidar-nordli-mathisen-y8TMoCzw87E-unsplash-1-scaled.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2021\/02\/vidar-nordli-mathisen-y8TMoCzw87E-unsplash-1-scaled.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2021\/02\/vidar-nordli-mathisen-y8TMoCzw87E-unsplash-1-scaled.jpg?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2021\/02\/vidar-nordli-mathisen-y8TMoCzw87E-unsplash-1-scaled.jpg?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2021\/02\/vidar-nordli-mathisen-y8TMoCzw87E-unsplash-1-scaled.jpg?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2021\/02\/vidar-nordli-mathisen-y8TMoCzw87E-unsplash-1-scaled.jpg?resize=1400%2C800&ssl=1 4x"},"classes":[]},{"id":6493,"url":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2019\/03\/17\/consensus-protocols-a-key-to-cluster-management\/","url_meta":{"origin":2843,"position":5},"title":"Consensus protocols – A key to cluster management?","author":"Benjamin Kowatsch","date":"17. March 2019","format":false,"excerpt":"blogpost In these times, applications require increasing robustness and scalability, since otherwise they will collapse under the burden of the vast number of users. Cluster managers like kubernetes, Nomad or Apache Marathon are a central factor of this resilience and scalability. A closer look at the insides of cluster managers\u2026","rel":"","context":"In "Allgemein"","block_context":{"text":"Allgemein","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/category\/allgemein\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_sharing_enabled":true,"authors":[{"term_id":732,"user_id":481,"is_guest":0,"slug":"mr143","display_name":"mr143@hdm-stuttgart.de","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/2a169003f6b79b3c75d54001ae4c144bd4e03ff2bbcafda15f2a705800c27636?s=96&d=mm&r=g","0":null,"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/posts\/2843","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/users\/481"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/comments?post=2843"}],"version-history":[{"count":9,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/posts\/2843\/revisions"}],"predecessor-version":[{"id":24729,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/posts\/2843\/revisions\/24729"}],"wp:attachment":[{"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/media?parent=2843"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/categories?post=2843"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/tags?post=2843"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/ppma_author?post=2843"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}