{"id":2859,"date":"2017-08-31T00:20:10","date_gmt":"2017-08-30T22:20:10","guid":{"rendered":"https:\/\/blog.mi.hdm-stuttgart.de\/?p=2859"},"modified":"2023-06-08T15:43:32","modified_gmt":"2023-06-08T13:43:32","slug":"iot-with-the-raspberry-pi-final-application-part-3","status":"publish","type":"post","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2017\/08\/31\/iot-with-the-raspberry-pi-final-application-part-3\/","title":{"rendered":"IoT with the Raspberry Pi \u2013 Final application \u2013 Part 3"},"content":{"rendered":"

In our final application, we have put together a solution consisting of four different modules. First, we have again the Raspberry Pi which raises and sends the sensor data using the already presented Python script. We changed the transfer protocol in the final application to MQTT, which gives us more possibilities in different aspects, but more on that later.<\/span>
\n
\nThe main part of our application in the cloud is based on a Node.js application where various frameworks such as Express and Passport are used.<\/span><\/p>\n

To secure the data persistently, we use the NoSQL solution MongoDB, which is linked to our application via its manifest file as a Bluemix service.<\/span><\/p>\n

Last but not least, we also need a broker for the chosen MQTT transmission protocol. The approximate task of the broker is to take care of the transmission of data between Raspberry Pi and the cloud application. In order to realize this service at Bluemix, we have created a Docker image that includes the MQTT broker.<\/span><\/p>\n

Module 1 – Raspberry PI<\/span><\/h3>\n

To determine the sensor data, the Raspberry PI continues using the already presented Python script. In order to transmit the sensor data via the MQTT protocol, we integrated the Eclipse Paho MQTT client into the project.<\/span><\/p>\n

Module 2 \u2013 MQTT broker<\/span><\/h3>\n

MQTT (Message Queue Telemetry Transport)<\/h4>\n

MQTT is a lightweight transmission protocol for machine-to-machine communication (M2M). To realize the transmission of the data the protocol is based on the principle of a Publish \/ Subscribe solution. In this context, there are clients that take on different roles. On the one hand, there is the role as publisher, which provides and sends messages. On the other hand, there is the role as subscriber, which accepts the provided messages. Communication takes place via a topic with a unique ID. You can see the topic as a bulletin board with<\/span><\/p>\n

a unique inventory number. For example, there may be a publisher who sends its data to a particular topic and an undefined set of subscribers who have subscribed to this topic to receive the data.<\/span><\/p>\n

MQTT broker<\/h4>\n

A MQTT broker is the central component in the communication via MQTT. It manages the topics and the related messages. Also, it regulates access to these topics and takes care of data security as well as the Quality of Service levels. The Quality of Service can be defined in three different stages. The lowest level is declared by 0 and means that there is no guarantee that the message arrives at the receiver. This variant produces the least overhead during transmission, it follows the principle fire’n’forget. The next level, 1, says the message is recognized at least once in the topic queue. At level 2 it is ensured that the message arrives exactly once.<\/span><\/p>\n

\"\"<\/p>\n

Setting up our own MQTT-Broker<\/h4>\n

Instead of using the Bluemix-Services to register the devices, we wanted to use our own<\/span> solution<\/span> to be more flexible. For that step, we used<\/span> this<\/span><\/a> Docker image which includes the open source Eclipse Mosquitto MQTT broker. The Docker image builds on a Linux distribution called Alpine Linux which is described as small, fast and safe.  <\/span><\/p>\n

For more security and to disable everybody is sending us data an auth plugin for the MQTT broker is included into the Docker image. The auth plugin is written in c and uses a c-library for handling requests to the MongoDB. The Bluemix service for the MongoDB requires an authentication via a certificate. But the auth plugin doesn\u2019t support this so we had to include this step by ourselves and had to modify the Docker image for our needs.<\/span><\/p>\n

In order to be able to transfer data, users first have to register on our cloud application and create a sensor. With this user data a topic is generated to which the user is allowed to send his data. This topic is structured as follows:<\/span><\/p>\n

client\/username\/devicename<\/pre>\n
The auth plugin expects the password for authentication in the form of PBKDF2 (Password-Based Key Derivation Function 2). This is a standard function to derive a key from a password.<\/span><\/p>\n
PBKDF2$sha256$901$veyFV98cpOUiYpuI$pr+\/jGkvwCdPu6n\/SnTKxjOUtgfx34Qp<\/pre>\n
The various parts of this key are separated by the separator $. The first part is the start marker followed by the description of the hash function. The third part shows the number of iterations followed by the salt. The last part is the hashed password. We have adapted the registry of our application so that the user password is stored in the database in this form.<\/span><\/p>\n
For the cloud application, we have created a so-called superuser, who has the rights to subscribe to each topic. In the application, this user connects and subscribes to the topic “client\/#”. The diamond is a placeholder for all possible topics.<\/span><\/p>\n
To load the Docker image in Bluemix you have to install the Bluemix Container registry plug-in. This is done via the command-line interface.<\/span><\/p>\n
bx plugin install container-registry -r Bluemix<\/pre>\n
You now have two options: Either create the Docker image on the local computer and push it to Bluemix or you create the image, like us, right in the Bluemix cloud.<\/span><\/p>\n
bx ic build -t registry.eu-de.bluemix.net\/my_namespace\/my_image:v1 .<\/pre>\n
At Bluemix, we also tested the delivery pipeline for the Docker image. Unfortunately, compiling the image in the pipeline failed. Since we had to do this step only once, we did not deal with this problem any further. Then log in to Bluemix and create a new container. The created image should now be selectable. You have to unlock the required ports if not already done in the Dockerfile and request a fixed IP address.<\/span><\/p>\n
 <\/p>\n
Module 3 – MongoDB as Service<\/span><\/h3>\n
We used MongoDB in our main application in combination with Mongoose. <\/span>It was somewhat tiring to establish within the Node.js application the connection with the MongoDB via Mongoose. The supplied credentials by the MongoDB service are designed to automatically connect to the admin database. However, this database ideally shouldn\u2019t be used in productive use. Mongoose does not directly provide the possibility to select another one after connecting to a database. You have to submit the desired database name into the connection call with additional settings. Without this settings the connection call would be much shorter.<\/span><\/p>\n
\/\/ load local VCAP configuration and service credentials\nvar vcapLocal;\ntry {\n    vcapLocal = require('.\/vcap-local.json');\n} catch (e) { }\n\nconst appEnvOpts = vcapLocal ? { vcap: vcapLocal} : {}\nconst appEnv = cfenv.getAppEnv(appEnvOpts);\n\n\/\/ mongoose - mongoDB connection\nvar mongoDBUrl, mongoDBOptions = {};\nvar mongoDBCredentials = appEnv.services[\"compose-for-mongodb\"][0].credentials;\n\nif (mongoDBCredentials) {\n    var ca = [new Buffer(mongoDBCredentials.ca_certificate_base64, 'base64')];\n    mongoDBUrl = mongoDBCredentials.uri;\n    mongoDBOptions = {\n        auth: {\n            authSource: 'admin'\n        },\n        mongos: {\n            ssl: true,\n            sslValidate: true,\n            sslCA: ca,\n            poolSize: 1,\n            reconnectTries: 1,\n            promiseLibrary: global.Promise\n        }\n    };\n} else {\n    console.error(\"No MongoDB connection configured!\");\n}\n\n\/\/ connect to our database (iot)\nvar db = mongoose.createConnection(mongoDBUrl, \"iot\", mongoDBOptions);<\/pre>\n
While testing the application, we probably filled the 1 GB database completely because we couldn\u2019t save any new data anymore. The Bluemix backend has reported only 0.035 GB of used storage capacity. After clearing the database, the data was saved again.<\/span><\/p>\n
 <\/p>\n
Module 4 – Cloud application<\/span><\/h3>\n
We created and launched a Cloud Foundry App for Node.js at Bluemix for the main application. The application allows users to register, create a sensor and then download a configuration file for the Raspberry Pi script. The user must enter his or her correct password in the configuration file. If the sensor now successfully sends data to the application, these can be read out in a chart.<\/span><\/p>\n
\"\"<\/p>\n
The application relies on common JavaScript packages which are loaded via the package manager npm and managed in the package.json file.<\/span><\/p>\n
    \n
  • Express is a web framework for Node.js and performs important tasks such as routing and has a view system that supports a variety of template engines.<\/span><\/li>\n
  • Passport is an express compatible authentication middleware for Node.js. Through a plugin for Passport we implemented the user registration in our application.<\/span><\/li>\n
  • Instead of using Paho Eclipse as a MQTT client like at the Raspberry Pi, we use MQTT.js in the application. MQTT.js offers a wider range of functions, a simpler API and often receives updates.<\/span><\/li>\n<\/ul>\n
    Frontend<\/b><\/p>\n
    We use Pug as template engine in the frontend. It\u2019s a very simple view engine and recommended for small projects. If we would be going to expand the project further, we would exchange it for a view engine with a larger range of functions. Reading out nested arrays wasn’t always easy with Pug.<\/span><\/p>\n
    For the basic layout we used Bootstrap 4, so the application is also responsive. However, reading the diagrams on the smartphone is not ideal. Bootstrap 4 was still in beta status during the project. This status was already so far advanced that we were able to work productively with it. Currently Bootstrap 4 has gone into the alpha phase.<\/span><\/p>\n
    Last but not least, the graphic heart of our application. We use Chart.js, which offers a large range of configuration possibilities and a variety of different diagram types. Its declarative approach makes it easy to use and the huge community makes it easy to find answers to any additional questions. Via a modal we offer adjustment possibilities such as the limitation of the time period of the sensor data. These settings are currently stored via a cookie.<\/span><\/p>\n
    \"\"<\/p>\n
    Build tool<\/b><\/p>\n
    As build tool we used Webpack. For the development we additionally relied on the plugin webpack-dev-middleware. It provides a watch mode that recompiles the JavaScript and Sass files as soon as changes are made to them. The plugin does not write the files directly to the hard disk, but keeps them in memory, which increases the speed. Before the application is pushed into the cloud, the data must be compiled with Webpack.<\/span><\/p>\n
    Gitlab<\/b><\/p>\n
    As repository we used the Gitlab of the HdM. In Gitlab we have set up a delivery pipeline. Once a commit is made to the repository, the data from the repository is pushed into the cloud. In the application, a Yaml file with the name .gitlab-ci.yml is created in which the required commands are stored.<\/span><\/p>\n
    stages:\n  - test\n  - deploy\ntest:\n  image: node:6\n  cache:\n    paths:\n      - node_modules\/\n  stage: test\n  script:\n   - npm install\n   - npm test\nstaging:\n  image: ruby:2.3\n  type: deploy\n  script:\n    - apt-get update -yq\n    - apt-get install -y ruby-dev\n    - gem install dpl -v 1.8.39\n    - dpl --provider=bluemixcloudfoundry --username=$BLUEMIX_USER --password=$BLUEMIX_PASSWORD --organization=$BLUEMIX_ORG --space=$BLUEMIX_SPACE --api=https:\/\/api.ng.bluemix.net --skip-ssl-validation\n  only:\n  - master<\/pre>\n
    The variables are created in the backend of Gitlab.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"
    In our final application, we have put together a solution consisting of four different modules. First, we have again the Raspberry Pi which raises and sends the sensor data using the already presented Python script. We changed the transfer protocol in the final application to MQTT, which gives us more possibilities in different aspects, but […]<\/p>\n","protected":false},"author":481,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1,654,650,22],"tags":[],"ppma_author":[732],"class_list":["post-2859","post","type-post","status-publish","format-standard","hentry","category-allgemein","category-internet-of-things","category-scalable-systems","category-student-projects"],"aioseo_notices":[],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":2820,"url":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2017\/08\/31\/iot-with-the-raspberry-pi-part-1\/","url_meta":{"origin":2859,"position":0},"title":"IoT with the Raspberry Pi – Part 1","author":"mr143@hdm-stuttgart.de","date":"31. August 2017","format":false,"excerpt":"Introduction to the project As part of the lecture \u201cSoftware Development for Cloud Computing\u201d in summer term 2017 we primarily wanted to work on a project that has something to do with the Internet of Things. In more detail we decided to measure air quality using a Raspberry Pi with\u2026","rel":"","context":"In "Allgemein"","block_context":{"text":"Allgemein","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/category\/allgemein\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2017\/08\/raspberry_pi_gas_sensor-steckplatine-1024x520.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2017\/08\/raspberry_pi_gas_sensor-steckplatine-1024x520.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2017\/08\/raspberry_pi_gas_sensor-steckplatine-1024x520.png?resize=525%2C300&ssl=1 1.5x"},"classes":[]},{"id":12751,"url":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2021\/02\/28\/industry-4-0-real-time-data-visualization\/","url_meta":{"origin":2859,"position":1},"title":"Industry 4.0 – Real time data visualization","author":"Philip Betzler","date":"28. February 2021","format":false,"excerpt":"As part of the lecture \"System Engineering and Management\", we worked on a project in cooperation with IBM to visualize Industry 4.0 data in real time using a configuration-based approach. This project aims to avoid needing a web developer every time there is a modification in the data.","rel":"","context":"In "Allgemein"","block_context":{"text":"Allgemein","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/category\/allgemein\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2021\/03\/config_type_harting_mica.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2021\/03\/config_type_harting_mica.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2021\/03\/config_type_harting_mica.png?resize=525%2C300&ssl=1 1.5x"},"classes":[]},{"id":11460,"url":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2020\/09\/29\/get-car-location-using-raspberrypi-and-google-cloud-iot-core\/","url_meta":{"origin":2859,"position":2},"title":"Get car location using Raspberry Pi and Google Cloud IoT Core","author":"Simon L\u00f6bert","date":"29. September 2020","format":false,"excerpt":"Project idea Have you ever been in the situation, that you parked your car somewhere in the city and some hours later, you couldn't remember where you parked it? You may wish to have an application on your smartphone, which is able to locate your car. From this consideration, the\u2026","rel":"","context":"In "Allgemein"","block_context":{"text":"Allgemein","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/category\/allgemein\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2020\/09\/grafik.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":2843,"url":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2017\/08\/31\/iot-with-the-raspberry-pi-node-red-part-2\/","url_meta":{"origin":2859,"position":3},"title":"IoT with the Raspberry Pi \u2013 Node RED – Part 2","author":"mr143@hdm-stuttgart.de","date":"31. August 2017","format":false,"excerpt":"As already stated in the introduction to our project, we decided to create a Cloud Foundry-Application in IBM Bluemix. We used the boilerplate called \u201cInternet of Things Platform Starter\u201d. Using this boilerplate Node Red is deployed initially. Node Red is a software tool for graphical dataflow programming. It was developed\u2026","rel":"","context":"In "Allgemein"","block_context":{"text":"Allgemein","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/category\/allgemein\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2017\/08\/websocketnode_config-300x143.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":3513,"url":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2018\/03\/30\/ci-cd-with-gitlab-ci-for-a-web-application-part-3\/","url_meta":{"origin":2859,"position":4},"title":"CI\/CD with GitLab CI for a web application – Part 3","author":"Nina Schaaf","date":"30. March 2018","format":false,"excerpt":"Hosting your own GitLab server Some users might have concerns regarding security using GitLab for a variety of purposes, including commercial and business applications. That is, because GitLab is commonly used as a cloud-based service - on someone else's computer, so to speak. So setting it up for running it\u2026","rel":"","context":"In "DevOps"","block_context":{"text":"DevOps","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/category\/scalable-systems\/devops\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":24490,"url":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2023\/03\/06\/how-to-scale-an-iot-platform\/","url_meta":{"origin":2859,"position":5},"title":"How to scale an IoT-Platform","author":"Simon Janik","date":"6. March 2023","format":false,"excerpt":"Written by Marvin Blessing, Michael Partes, Frederik Omlor, Nikolai Thees, Jan Tille, Simon Janik, Daniel Heinemann - for System Engineering And Management IntroductionArchitectureScaling & Load TestEstimations and predicted bottlenecksMonitoringLoad TestScalingLessons LearnedMonitoring & Error searchDevelopment ProcessIdentifying and improving bottlenecks Introduction The aim of the project was to develop a system with\u2026","rel":"","context":"In "Allgemein"","block_context":{"text":"Allgemein","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/category\/allgemein\/"},"img":{"alt_text":"","src":"https:\/\/lh6.googleusercontent.com\/Gsjx91zLxbPn2Uer_-kXHZ68xbA2YmKIrqreIyWNPF9MwowChC5IHi1wx6G6Ctj2MKTRA1n-uZHlwfjxk-dYkjGrzGY10KbnOLN1UKQVbMNaO-RIvm3c7cBFVZQEy5lqi33i_F5TEbln0X7C3CZfL4k","width":350,"height":200},"classes":[]}],"jetpack_sharing_enabled":true,"authors":[{"term_id":732,"user_id":481,"is_guest":0,"slug":"mr143","display_name":"mr143@hdm-stuttgart.de","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/2a169003f6b79b3c75d54001ae4c144bd4e03ff2bbcafda15f2a705800c27636?s=96&d=mm&r=g","0":null,"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/posts\/2859","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/users\/481"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/comments?post=2859"}],"version-history":[{"count":15,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/posts\/2859\/revisions"}],"predecessor-version":[{"id":24730,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/posts\/2859\/revisions\/24730"}],"wp:attachment":[{"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/media?parent=2859"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/categories?post=2859"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/tags?post=2859"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/ppma_author?post=2859"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}