Fog computing offers a compromise between cloud and edge computing for real-time, scalable data analysis. Ideal for regional applications and IoT. However, authentication and privacy issues must be addressed.
Most Cloud Developers seem to agree that the best way to gain knowledge and optimize processes is to collect and store data in the cloud and to analyze the collected data after the fact or in almost-real-time. In a world where the number of users and devices is already almost uncountable, the name Big Data becomes more and more a euphemism for the vast landscape we call the internet. On the other hand, we can also execute data analysis tools on the devices locally in a network using edge computing. This allows developers to work on the data of single devices or single users-grade networks.
Apparently, we can either have all the data or only a little data, but either way is difficult to handle efficiently. This is where fog computing comes into play.
Architecture placing nodes in the fog between the data centers of the cloud and devices at the edge; from “Moving the Cloud to the Edge”; PubNub; 06.06.2017
What is the Fog?
When it comes to internet services like Internet of Things, web services, Infrastructure-as-a-service, or networking solutions, the limiting factors are latency, networking resources, data storage, and computing power. While most of these problems can be solved through more or faster hardware, there are problems that need analytical and clever solutions. More parallelization, e.g. load balancing multiple servers, and chaining of microservices, e.g. message queues, do not solve the logical problems of conflicting data or processes, e.g. database locking. It is simply physically impossible to manage all requests all the time.
In cloud computing, resources are distributed over the world and information is shared across that ominous, obscure cloud. This makes it possible to handle requests on devices that have availability, in order to handle requests with medium latency and high cumulative computing power. This makes it easy to scale the infrastructure, but makes the analysis of data very time and resource consuming and necessitates to store data permanently from time to time.
Whereas cloud computing handles all requests on a “synchronized” level, where we don’t know and don’t care about the location and type of device that handles a request, edge computing happens at the “edge to the internet”. That means that edge devices or edge networks process data either incoming or outgoing at or close to the user. This approach is often also referred to as “serverless” because no server is needed to handle any requests other than offering the code to create the interface (HTML, CSS etc.) and the code to be executed on edge devices.
With edge computing it is easy to pinpoint the physical and logical “location” and state and type of requests, data and devices. It is possible to handle those requests with extremely low latency, both network and compute, and a low need for computing power and resources. Although the need is low, scalability and access to data is very limited. This means that the calculations and processes done in edge computing are fast, but not always very rich or productive.
How does Fog Computing improve system architecture?
Fog computing places nodes between the edge, where the user with his private devices and networks resides, and the cloud, where nobody can identify any instance for sure, unless one asks the cloud providers themselves. The cloud stores and manages data in a somewhat manageable number of datacenters, and the edge completes processes on billions of devices. The fog on the other hand, consists of servers with numbers in the high thousands or few millions worldwide. This infrastructure allows developers and operators to make a compromise between cloud and edge computing. These servers are physically close to users’ networks.
Difference between Cloud, Fog and Edge Computing in IoT; Digiteum; 04.05.2022
The compromise has the following characteristics:
The shorter physical distance also implies a low network latency and low need for computing resources due to limited data amounts.
Data can be analyzed locally and in real time.
Nodes can be scaled flexibly in response to growing numbers of users in small networks and regions with a small priority of single nodes.
Nodes in the fog act as handlers between edge networks and the cloud. Urgent requests are forwarded to the cloud, and requests that can be processed locally are.
Connecting edge devices and the cloud through an independent actor allows for more and better control over the incoming and outgoing data. Information can be separated regionally and the usage of the cloud can be controlled in the fog. Forwarding requests is a concern because both the storage of sensitive information and the limitation of needed computing resources is detrimental to the overall quality of the cloud.
Data security and media rights management can be provided through those nodes on a regional basis. Sensitive information is stored locally on fog level and never forwarded to the cloud. The urgency of a request has to be determined on fog nodes. Under normal circumstances, networking bandwidth and computing resources are spared in the cloud by storing data locally whenever necessary and storing it in the cloud when the localization of data is no concern.
Smart Cities: An example for Fog Computing applications
One example for the use of fog nodes are smart cities, meaning independent traffic systems interacting to reduce traffic jams and incidents. Cars, traffic signals and road barriers are installed with sensors to collect data and send that data to a fog node. It is not necessary for a traffic light in New York to gain information about a car in Los Angeles, but the general gist of traffic optimization is the same. Fog computing allows real-time data analysis that enables traffic signals to rapidly change according to the traffic situation.
AI illustration of connected smart cities, Stabillity AI, 22.02.2023
To this end, it is more useful to connect every device in the city to a fog node that is only responsible for that city. That way, millions of fog nods are placed to control traffic in a city or wider city area with its own scaled infrastructure. For example, New York City will most likely need more resources per square mile than Austin, Texas.
This separation of fog nodes leads to a separation of private data, like location data, e.g. home addresses, from the cloud service providers, and between each fog node. Only essential data is forwarded to the cloud and used to generate predictions, update traffic models, and generate newer algorithms and usage analysis.
With this method, degrading performance due to high demand only affects one city. High loads can be forwarded to the cloud, where requests are distributed globally to free resources. This way, the averaged load of the cloud can be used to cushion the impact of high local demand.
Obstacles for the implementation of Fog Computing
Authentication and trust issues: Just like cloud service providers, fog service providers can be different parties with varying trust levels. One provider can pose as a trustworthy entity and manipulate the integrity of the fog for its connected end-users.
Privacy: Decentralizing data security and media rights management to the fog and outsourcing the responsibility to a third party instead of the cloud or edge devices endangers user’s privacy due to the number of fog nodes.
Security: Due to the number of devices connected to fog nodes, it is difficult to ensure the security of all gateways and the protection of personal information.
Node placement: The physical and logical location of fog nodes should be optimized to reach the maximum service quality. The data for the right placement has to be analyzed and chosen carefully.
Conclusion
Fog Computing places nodes logically between edge networks and devices and the cloud.
It provides 4 main advantages:
Network latency: Lower distance to the end-user and smaller data consumption leads to lower network delay and lower computing time.
Data analysis: Low data amounts allow for real-time data analysis and the limitation of cloud usage.
Security: Configuring fog nodes to the data protection needs ensures that cloud service providers only gain access to as much data as needed.
Cost reduction: The regional placement and subdivision can minimize the hardware and energy cost for the fog service providers.
Gaming is fun. Strategy games are fun. Multiplayer is fun. That’s the idea behind this project.
In the past I developed some games with the Unity engine – mainly 2D strategy games – and so I thought it is now time for an awesome 3D multiplayer game; or more like a prototype.
The focus of this blog post is not on the game development though, but rather on the multiplayer part with the help of Cloud Computing.
However where to start? There are many ways one can implement a multiplayer game. I chose a quite simple, yet most of the time very effective and cheap approach: Peer-to-Peer (P2P).
But first, let us dive in the gameplay of Admiral: WW2 (working title).
2 Game Demo
2.1 Gameplay
Admiral: WW2 is basically like the classic board game “Battleships”. You’ve got a fleet and the enemy player has got a fleet. Destroy the enemy’s fleet before your own fleet is sunk. The big difference is that Admiral: WW2 is a real-time strategy game. So the gameplay is more like a real-life simulation where you as the admiral can command your ships via direct orders:
Set speed of a ship (stop, slow ahead, full ahead, …)
Set course of a ship
Set the target of the ship (select a ship in the enemy fleet)
Currently there is only one ship class (the German cruiser Admiral Hipper), so the tactical options are limited. Other classes like battleships, destroyers or even aircraft carriers would greatly improve replayability; on the other hand they would need many other game mechanics to be implemented first.
Ships have multiple damage zones:
Hull (decreases the ship’s hitpoints or triggers a water ingress [water level of the ship increases and reduces the hitpoints based on the amount of water in the hull])
Turrets (disables the gun turrets)
Rudder (rudder cannot change direction anymore)
Engine/Propeller (ship cannot accelerate anymore)
If a ship loses all hitpoints the ship will sink and is not controllable.
2.2 The Lobby Menu
Before entering the gameplay action the player needs to connect to another player to play against. This is done via the lobby menu.
Here is the place where games are hosted and all available matches are listed.
On the right hand side is the host panel. To create a game the host must enter a unique name and a port. If the IP & Port combination of the host already exists, hosting is blocked.
After entering valid infos the public IP of the host is obtained via an external service (e.g. icanhazip.com). Then the match is registered on a server and the host waits for incoming connections from other players.
On the left hand side there is the join panel. The player must enter a port before viewing the match list. After clicking “Join”, a Peer-to-Peer connection to the host is established. Currently the game only supports two players, so after both peers (host and player) are connected the game will launch.
More on the connection process later.
3 Multiplayer Communication with Peer2Peer
3.1 Peer-to-Peer
P2P allows a direct connection between the peers with UDP packets – in this case the game host and player.
So in between no dedicated server handling all the game traffic data is needed, thus reducing hosting costs immensely.
Because most peers are behind a NAT and therefore connection requests between peers are blocked, one can make use of the NAT-Traversal method Hole-Punching.
3.1.1 P2P Connection with Hole-Punching
Given peer A and peer B. A direct connection between A and B is possible if:
A knows the public IP of B
A knows the UDP port B will open
B knows the public IP of A
B knows the UDP port A will open
A and B initiate the connection simultaneously
This works without port-forwarding, because each peer keeps the port open as if they would contact a simple web server and wait for the response.
To exchange the public IPs and ports of each peer a Rendezvous-Server behind no NAT is required.
3.1.2 Rendezvous-Server
The Rendezvous-Server needs to be hosted in the public web, so behind no NAT. Both peers now can send simple web requests as if the users would browse the internet.
If peer A tells the server he wants to host a game, the server saves the public IP and port of A.
If B now decides to join A’s game the server informs B of the IP and port of A.
A is informed of B’s public IP and port as well.
After this process A and B can now hole-punch through their NATs and establish a P2P connection to each other.
A Rendezvous-Server can be very cheap, because the workload is quite small.
But there are some cases where Hole-Punching does not succeed (“…we find that about 82% of the NATs tested support hole punching for UDP…”, https://bford.info/pub/net/p2pnat/).
In those cases a Relay-Server is needed.
3.1.3 Relay-Server
The Relay-Server is only used as a backup in case P2P fails. It has to be hosted in the public internet, so behind no NAT.
Its only task is the transfer of all game data from one origin peer to all other peers. So the game data just takes a little detour to the Relay-Server before continuing it’s usual way to the peers.
This comes at a price though. Since all of the game traffic is now travelling through this server the workload can be quite tough depending on the amount of information the game needs to exchange. Naturally the ping or RTT (Round Trip Time: the time it takes to send a packet from peer to peer) of a packet is increased resulting in lags. And finally multiple Relay-Servers would be required in each region (Europe, America, Asia, …). Otherwise players far away from the Relay-Server suffer heavy lags. All of these lead to high demands on the server hardware. To be clear: a proper Relay-Server architecture can be expensive in time and money.
Because of that in this project I ignored the worst-case and focused on the default Peer-to-Peer mechanism.
3.1.4 Peer2Peer Conclusion
The big advantage of this method: it’s mainly serverless, so the operation costs of the multiplayer is very low. Because of that, P2P is a very viable multiplayer solution for small projects and indie games. The only thing that is needed is a cheap Rendezvous-Server (of course only if no Relay-Server is used). P2P also does not require to port-forward, which can be a difficult and/or time consuming task depending on the player’s knowledge.
But there are disadvantages:
A home network bandwidth may not be enough to host larger games with much traffic; a server hosted at a server farm has much more bandwidth
The game stops if a P2P host leaves the game
No server authority
every player has a slightly different game state that needs to be synchronized often; a dedicated server has only one state and distributes it to the players; players only send inputs to the server
anti-cheat has to be performed by every peer and not just the server
random is handled better if only the server generates random values, otherwise seeds have to be used
game states may need to be interpolated between peers, which is not the case if only the server owns the game state
A dedicated server would solve these disadvantages but in return the hardware requirements are much higher making this approach more expensive. Also multiple servers would be needed in all regions of the world to reduce ping/RTT.
3.2 Game Connection Process
After starting the game the player sees the multiplayer games lobby. As described previously the player can host or join a game from the list.
3.2.1 Hosting a game
The host needs to input a unique game name and the port he will open for the connection. When the host button is clicked the following procedure is triggered:
Obtain public IP address
Originally this should be handled by the Rendezvous-Server, because it is hosted behind no NAT and can see the public IP of requests, but limitations of the chosen hosting service prevented this approach (more on that later)
Instead I used a web request to free services like icanhazip.com or bot.whatismyipaddress.com as a second backup in case the first service is down; these websites respond with a plain text containing the ipv6 or ipv4 of client/request
The Rendezvous-Server is notified of the new multiplayer game entry and saves the game with public IP and port, both sent to the server by the host
Host sends GET-Request to the server (web server) containing all the information needed /registermpgame?name=GameOne&hostIP=1.1.1.1&hostPort=4141
On success the game is registered and a token is returned to the host; the token is needed for further actions affecting the created multiplayer game
The host now waits for incoming connections from other players/peers
The host sends another GET-Request to the Rendezvous-Server /listenforjoin?token=XYZ123
This is a long-polling request (websocket alternative): the connection is held open by the server until a player joined the multiplayer game
If that is the case the GET-Request is resolved with the public IP and port of the joined player, so that hole-punching is possible
If no player joins until the timeout is reached (I’ve set the timeout to 15 seconds), the request is resolved with http status code 204 No content and no body
In that case the GET-Request has to be sent again and again until a player joins
On player join both peers init a connection and punch through NAT
If successful the game starts
(Otherwise a Relay-Server is needed; explained previously)
The host closes the game with another GET /startorremovempgame?token=XYZ123
3.2.2 Joining a game
The player first needs to input a valid port. After that he is presented with a list of multiplayer games by retrieving the information from the Rendezvous-Server with a GET-Request to the endpoint /mpgameslist. This returns a JSON list with game data objects containing the following infos:
name: multiplayer game name
hostIP: public IP of the host
hostPort: port the host will open for the connection
If the player clicks “Join” on a specific game list item the following process handles the connection with the host:
Obtain public IP address
Originally this should be handled by the Rendezvous-Server, because it is hosted behind no NAT and can see the public IP of requests, but limitations of the chosen hosting service prevented this approach (more on that later)
Instead I used a web request to free services like icanhazip.com or bot.whatismyipaddress.com as a second backup in case the first service is down; these websites respond with a plain text containing the ipv6 or ipv4 of the client/request
Inform the Rendezvous-Server of the join
Send a GET-Request with all the information needed /joinmpgame?name=GameOne&ownIP=2.2.2.2&hostPort=2222
Now the host is informed by the server if the host was listening
The server resolves the request with the public IP and port of the host
Now the player and the host try to establish a P2P connection with hole-punching
If successful the game starts
(Otherwise a Relay-Server is needed; explained previously)
3.3 Game Synchronization
Real-time synchronization of game states is a big challenge. Unlike turn-based games the game does not wait until all infos are received from the other players. The game always goes on with a desirably minimal amount of lag.
Of course the whole game state could be serialized and sent to all players, but this would have to happen very frequently and the package size would be very large. Thus resulting in very high bandwidth demand.
Another approach is to only send user inputs/orders, which yields far less network traffic. I used this lightweight idea, so when the player issues an order the order is immediately transmitted to the other player. There the order is executed as well.
The following game events are synchronized:
GameStart: After the game scene is loaded the game is paused and the peer sends this message to the other player periodically until he receives the same message from the other peer; then the game is started
RandomSeed: Per game a “random seed master” (the host) periodically generates a random seed and distributes that seed to the other player; this seed is then used for all random calculations
All 3 ship orders:
ShipCourse
ShipSpeed
ShipTarget
GameSync: All of the previous messages still led to diverging game states, so a complete game serialization and synchronization is scheduled to happen every 30 seconds
Projectile positions, rotations, velocities are synched
The whole ship state is synched
Both game states (the received one and the own one) are interpolated, because I don’t use an authoritative server model and so both game states are “valid”
The following game events should have a positive impact on game sync, but are not implemented yet:
ProjectileFire: Syncs projectiles being fired
Waves: Because the waves have a small impact on the position where projectiles are fired and hit the ship the waves should be in-sync as well
3.3.1 IDs
In game development you mostly work with references. So for example a ship has a reference to another ship as the firing target. In code this has the benefit of easy access to the target ship’s properties, fields and methods.
The problem is with networking these references do not work. Every machine has different references although it may represent the same ship. So if we want to transfer the order “Ship1 course 180” we cannot use the local reference value to Ship1.
Ship1 needs an unique ID that is exactly the same on all machines. Now we can send “ShipWithID1234 course 180” and every machine knows which ship to address.
In code this is a bit more tedious, because the received ID has to be resolved to the appropriate ship reference.
The most difficult part is finding unique IDs for all gameobjects.
Ships can obtain an ID easily on game start by the host. Projectiles are a bit more tricky, because they are spawned later on. I solved this by counting the shots fired by a gun turret and combining the gun turret’s ID with the shot number to generate a guaranteed unique ID, provided the gun turret ID is unique. Gun turret IDs are combined as well: Ship ID + gun turret location (sternA, sternB, bowA, bowB, …).
Of course with an authoritative server this gets easier as only the server generates IDs and distributes them to all clients.
3.3.2 Lockstep
Additionally there is an interesting and promising approach to discretize the continuous game time called Lockstep. It is used in prominent real-time strategy games like Age of Empires (https://www.gamasutra.com/view/feature/131503/1500_archers_on_a_288_network_.php). The basic idea is to split up the time in small time chunks, for example 200ms intervals. In this time frame every player can do exactly one action that gets transferred to all the other players. Of course this action can also be “no action”. The action is then executed in the next interval almost simultaneously for all players. This way the real-time game is transformed into a turn-based game. It is important to adjust the interval based on the connection speeds between the players, so that no player lags behind. For the players the small order input delay is usually unnoticed, if the interval is small enough.
An important requirement is that the game is deterministic and orders issued by players have the same outcome on all machines. Sure there are ways to handle random game actions, but because AdmiralWW:2 uses random for many important calculations and my development time frame was limited I unfortunately did not implement this technique.
4 Rendezvous-Server Hosting
There are almost unlimited hosting options on the internet. Usually the selection shrinks after a specific programming language is picked. But because I used NodeJS with Typescript, which transpiles the code to default Javascript, there were still plenty of hosting options. If I decided to write the server in C# and therefore run a .NET Core application like the game is written with (Unity uses C# or some exotic programmers use Javascript) many hosting providers drop out.
4.1 Alternatives
Of course there is the option of renting an own dedicated server: very expensive for a simple Rendezvous-Server and maintenance heavy, but powerful and flexible (.NET ok).
There’s the option of a managed server: little maintenance but very, very expensive.
We have VPS (Virtual Private Servers): dedicated servers that are used by many customers and the hardware is distributed among them, cheaper.
Then there are the big players like AWS, Google Cloud Platform, IBM Cloud and Microsoft Azure: they can get very expensive, but in return they offer vast opportunities and flexibility; it is easy to scale and monitor your whole infrastructure and a load-balancer can increase availability and efficiency of your server(s); on the other hand the learning-curve is steeper and setting up a project needs more time.
4.2 Heroku
Heroku is a cloud based Platform-as-a-service (PaaS) offering hosting of many common programming languages like Javascript/NodeJS (which I used), Python and Ruby. It does not offer as many possibilities as AWS and co, but it is way simpler to learn and set up.
Also it does have a completely free plan, which grants over 500 hours uptime per month. This is not enough to run the whole month with 30 * 24 = 720 hours, but the application sleeps after 1 hour with no actions and automatically wakes up again if needed. This is perfectly fine for a Rendezvous-Server, because it is not used all the time. The wake up time is not that bad as well (around 4-8 seconds).
Of course Heroku offers scaling so that the performance is massively increased and the app will never sleep, but this comes with a price tag.
In a paid plan Heroku also has a solid monitoring page with events, up- and downtimes, traffic and so on.
Server logs are easily accessible as well.
For setup you just need to create a “Procfile” in your project folder that defines what to execute after the build is completed: web: npm run start will run the npm script called start as a web service. The application is then publicly reachable on your-app-name.herokuapp.com. The NodeJS web server can then listen on the port that is provided by Heroku in the environment variable process.env.PORT.
Deployment is automated: just push to your github master branch (or the branch you specified in Heroku); after that a github webhook triggers the build of your app in Heroku.
But during development I discovered a big disadvantage: Heroku does not support ipv6.
This is a problem, because I wanted to use the Rendezvous-Server as a STUN-Server as well, which can determine and save the public IPs of client requests. But if a client like me only has Dual-Stack lite (unique ipv6 but the ipv4 address is shared among multiple customers) Peer2Peer is not possible with the shared ipv4.
As a workaround the clients obtain their public ipv4 or ipv6 via GET-Request from icanhazip.com or as a backup from bot.whatismyipaddress.com. These websites return a plain text body containing the public IP. After that the peers send their public IP to the Rendezvous-Server as explained previously.
5 Architecture Overview
Typescript usually is a very good choice for larger projects, simply because of the type-safety and development-time error checking. This guarantees no more searching for bugs like typos as it is the case in plain Javascript.
To realize the web server I used the very popular ExpressJS, which does not need any introduction and should be well-known by this time.
6 Conclusion
Real-time multiplayer games are tricky. The game states quickly diverge and much effort has to be done to counteract this. Game time differences and lag drastically compound this. But methods such as Lockstep can help to synchronize the time across multiple players.
While developing, try to keep the game as deterministic as possible, so that player actions yield the same result on every machine. Random is usually problematic, but can be handled via a dedicated game server or seeds.
Peer-to-Peer is a simple and great solution for smaller indie multiplayer games, but comes with some disadvantages. For larger projects dedicated/authoritative servers are favourable.
Heroku offers a fast and simple setup for hosting cloud applications and the free plan is great for smaller projects. If demand increases scaling is no problem and the deployment is automated. But be aware of the missing ipv6 support of Heroku.
All in all: Gaming is fun. Strategy games are fun. Multiplayer is fun – for the player and an exciting challenge for developers.
Kein anderes Getränk ist mit Weihnachtsmärkten so verbunden wie Glühwein. Und so trinkt sich der ausschweifende Weihnachtsmarktbesucher im Laufe der Adventszeit von Stand zu Stand bis er schließlich am Ende des Jahres seinen Lieblingsstand gefunden hat. Doch auch daheim kann der perfekte Glühwein gelingen.
Wir zeigen, wie man sich ein Glühweinthermoter mit Cloudanbindung selber baut, und so perfekten Glühwein und Komfort miteinander kombiniert. Und das ganz ohne gedrängte Weihnachtsmärkte und Mundschutz.
Prost!
Einleitung
Unser dreiköpfiges Team hatte für die Vorlesung Software Development for Cloud Computing das Ziel, die Grundlagen der Entwicklung in einer Cloud Umgebung zu lernen und dabei ein Projekt auf die Beine zu stellen, welches diese Grundlagen in der Praxis umsetzt. Ein interessanter Aspekt der Cloud war für uns dabei die Bereitstellung einer überall erreichbaren Umgebung, über welche wir verschiedene Geräte miteinander kommunizieren lassen können.
Daher kam uns die Idee, ein Thermometer zu bauen, welches mit einem Raspberry Pi verbunden ist und wir die Daten über die Cloud verarbeiten und an ein Smartphone weiterleiten. Darüber soll es möglich sein, die aktuelle Temperatur abzulesen und eine Prognose für die Dauer bis zum Erreichen einer einstellbaren Temperatur zu stellen.
Unser Projekt besteht aus drei logischen Schichten. Unser Sensor stellt ein Raspberry Pi mit angeschlossenem Thermometer dar. Der Sensor dient der Feststellung der Flüssigkeitstemperatur, die wir anschließend in der zweiten Schicht verarbeiten. Unsere zweite Schicht stellt dabei eine EC2 Instanz bei AWS dar. Diese erledigt die Berechnung der Zielzeit und stellt einen Webserver für die dritte Schicht, der Datenanzeige bereit. Die Anzeige stellt Informationen und bietet Möglichkeiten der Steuerung des Systems. Es bestehen also bidirektionale Verbindungen, damit der Benutzer Konfigurationen am System unternehmen kann.
Ablauf
Der grundlegende Ablauf in unserem Projekt sollte also folgendermaßen aussehen:
Im ersten Schritt scannt das Smartphone einen QR-Code auf dem Raspberry Pi, damit die richtige Zuordnung von Raspi und Smartphone in der Cloud später gewährleistet werden kann. Als nächstes beginnt der Raspberry Pi, die Temperatur über das Thermometer auszulesen und schickt diese an die Cloud weiter. Sobald der Nutzer nun seine Zieltemperatur eingegeben hat und die Abfrage gestartet hat, wird dies im vierten Schritt mit der ID des Raspberry Pis an die Cloud übermittelt. Nun kann diese die Daten des Raspis mit der passenden ID verarbeiten, die Zeit errechnen und das Ergebnis an die App weiterleiten.
Backend
In unseren ersten Schritten wollten wir uns mit der Cloud vertraut machen und erste Instanzen darauf laufen lassen. Dabei entschieden wir uns für die Cloud von Amazon Web Services (AWS), da es zu dieser eine gute Dokumentation gibt und sie alle für uns notwendigen Komponenten bereitstellt. Zwar kostet die AWS Cloud im Gegensatz zur IBM Cloud auch für Studenten etwas, aber dies stellte für uns kein Problem dar, da wir von der HdM genug Credits zur Verfügung gestellt bekommen haben.
Bei unserem ersten Versuch, eine EC2 Instanz zu starten, stießen wir aber bereits auf einige Probleme. Der Grund dafür war, dass die AWS Cloud relativ komplex ist und sehr viele Möglichkeiten bietet, die Instanzen zu individualisieren und zu optimieren. Dies ist besonders für Einsteiger zu Beginn relativ überfordernd. Am meisten Probleme hatten wir mit dem Einstellen der Security Groups. Diese sind notwendig, damit der Zugriff auf den Server von außerhalb möglich ist. Erst nachdem wir den Zugriff auch über die verschiedenen Protokolle wie TCP und UDP geöffnet haben, konnten wir auf den Server zugreifen.
Als nächstes mussten wir unseren Raspberry Pi so erweitern, dass er die Temperatur messen kann. Dafür haben wir ein Thermometer gekauft, welches wir mit dem Raspberry Pi verkabeln.
Um nun die Temperatur auch zu verarbeiten, benötigten wir ein Skript auf dem Raspberry Pi. Wir entschieden uns hierbei für Python, stellten aber im Nachhinein fest, dass eine Sprache, welche nativ auf dem Gerät läuft, sich hier besser geeignet hätte. Dies hat damit zu tun, dass der Raspberry Pi in unserem Fall ja nur als Testobjekt fungiert, auf welchem Linux installiert ist. Eigentlich sollte es auch möglich sein, die Aufgabe des Raspberry Pis auf ein embedded System zu übertragen, welches nicht die Möglichkeit hat, Python zu nutzen. Hätten wir dies im Vorhinein beachtet, wäre der Übergang vom Raspberry Pi zu embedded Systems einfacher.
Das Thermometer schreibt die ganze Zeit die aktuelle Temperatur in eine Datei auf dem Raspi. Diese lesen wir mit dem Skript jede Sekunde aus und schicken sie dann gemeinsam mit der ID des Raspis an den Server.
Im weiteren Verlauf des Projekts beschäftigten wir uns mehr mit der Serverseite in der Cloud. Wir entschieden uns für eine Node.js-Lösung, welche in der EC2 Instanz läuft, da Node.js mit get und post requests alle von uns benötigten Kommunikationsmittel zwischen dem Raspberry Pi und dem Smartphone bereitstellt. Auch im Nachhinein erwies sich Node.js als eine gute Wahl, da das Aufsetzen des Webservers keinerlei Probleme bereitet hat und die Kommunikation auch mit dem Python Skript auf dem Raspi einwandfrei geklappt hat.
Unser technischer Ablauf und die Kommunikation unter den Geräten sah nun folgendermaßen aus:
Frontend
Zu Beginn des Projekts bestand unser Frontend nur aus einer Webseite, welche die Temperatur des Thermometers anzeigen sollte. Später erweiterten wir diese mit einem Zeit-Temperatur Graph und einem Thermometer zur Darstellung der Temperatur. Dies ließ sich mit HTML, CSS und etwas JavaScript relativ simpel realisieren. Später wurde diese Ansicht durch eine Android App erweitert. Diese stellt eine mobile Möglichkeit dar, sich über den aktuellen Stand zu informieren. Technisch gesehen handelt es sich hierbei um eine WebView, die die Webseite mobil anzeigt.
Dieser Weg, erst eine leicht wartbare Webseite zu erstellen und diese anschließend per WebView auf dem Smartphone aufzurufen erwies sich als gute Idee. So konnten wir uns erst um die Funktionalität der Geschäftslogik konzentrieren und diese anschließend ohne viel Code auf dem Handy nutzen. Jedoch muss man die Optimierung für verschiedene Geräte dann nicht in der App selbst, sondern in der Website vornehmen, was etwas mühsamer ist als in Java für Android.
Zeitberechnung
Ein zentraler Wunsch war es, eine zeitliche Abschätzung zu erhalten, wann nach aktuellem Temperaturtrend die Zieltemperatur erreicht wird.
Dafür haben wir zuerst eine Beispielmessung eines Temperaturverlaufs durchgeführt. Eine Analyse verschiedener Trendlinien hat ergeben, dass sich eine quadratische Regression am Besten eignet. Bei der Auswahl haben wir ein besonderes Augenmerk auf die Genauigkeit der Zeitabschätzung nach kurzer Zeit gelegt, sodass wir bereits relativ früh eine gute Abschätzung bekommen.
Die eigentliche Berechnung erfolgte dann in 3 Schritten. Zuerst wurden die Mittelwerte der Messwerte ermittelt und mit diesen nach den Formeln der quadratischen Regression die Faktoren einer quadratischen Gleichung bestimmt. Anhand dieser konnten wir nun den Schnittpunkt mit der gewünschten Zieltemperatur berechnen. Vorteil dieser Variante ist es, dass wir auch negative Temperaturtrends, sowie andere Zieltemperaturen verarbeiten können. Bei der praktischen Anwendung stellte sich jedoch heraus, dass das System einige Schwachstellen aufweist. So können gleichbleibende Temperaturen, die vor allem in der Anfangsphase einer Erhitzung auftreten, die Berechnung sehr ins Schwanken bringen, sodass manchmal für längere Zeit keine Zielzeit berechnet werden kann. Auch kommt es zu teils starken Schwankungen im Verlauf einer Messung. Diese Probleme können jedoch durch eine Bereinigung der Daten im Voraus gelöst werden.
Fazit
Wir haben im Laufe des Projektes natürlich nicht nur viel Mathe gemacht, sondern auch sehr viel über Cloud Computing gelernt. Für einen Anfänger, der vorher noch nie mit AWS in Kontakt kam, ist der Einstieg ziemlich überfordernd. Es gibt deutlich einsteigerfreundliche IaaS-Anbieter wie z.B. die IBM-Cloud.
Was unseren Server angeht, sind wir auch recht zufrieden mit unserer Wahl von Node.js als Web-Backend. Node.js bietet den Vorteil, dass es sehr einfach ist, einen Webserver aufzusetzen, der auf Anfragen hört und gleichzeitig eine Webseite liefern kann. Braucht man mehr Performance und stellt viele parallele Anfragen an den Server, würde es sich lohnen einen Server in Go aufzusetzen. Dasselbe gilt für unseren Raspberry Pi. Das Python-Skript zu schreiben ging ziemlich schnell, aber auch hier könnte man auf eine performantere Lösung in C++ einsetzen.
Durch eine erprobte Zielberechnung hat unser Thermometer deutlich an Funktionalität gewonnen und kann für nun für verschiedene Temperaturen eingesetzt werden.
Unser Projekt war ganz klar auf Anfänger ausgerichtet. So wurden bereits genutzte und bekannte Technologien mit neuen Technologien der Cloud kombiniert. Dabei konnte der Funktionsumfang von AWS natürlich nicht vollständig ausgenutzt werden. Jedoch haben wir uns Schritt für Schritt an der Cloud bedient und so einen ersten Einblick in die Welt von IaaS erhalten.
Geschrieben von: Nikolai Thees, Michael Partes & Joshua Gertheiss
This article is about finding the right strategy if you want to empower your business with cloud computing. It’s written for those of you who are still new to this topic and want to get a first sight through the dense cloud mystery. After reading you should have gained an orientation on how to start your first cloud project and a basic knowledge of the most important buzzwords of the industry.
“The essence of strategy is choosing what not to do” – Prof. Dr. Michael E. Porter – Harvard Business School Cofounder of the strategic management
It’s stormy outside
It’s been over a decade now that Amazon was stirring up the market with their web services and cloud computing is now more mainstream than ever. What for many people started with file-sharing- and backup-services like Dropbox or company internal NAS-systems, quickly transformed into a trending global topic. Why should you always upgrade local hardware and storage to stay competitive if you can have the same or even greater performance with shared resources, reduced costs and accessibility from everywhere?
Some understood that earlier than others and can already harvest the fruits from their work – and some, hopefully not you – should seriously think about protecting their house because the storm is not over yet! With the introduction of cloud gaming services like GeForce Now and the start of projects from the major players in the market, it starts to be clear that cloud computing is not just a hype, quite the reverse, anything seems to be possible and any upcoming business transformations should consider the opportunities of cloud computing for their needs. And who knows, maybe one day you can even sell your computing power as a part of the worldwide cloud like it’s been introduced by crypto-currency miners or solar tech companies?
Faster, Greater, Higher! The sky’s the limit!
So as enough time has passed that the first CTOs and CIOs can look back to their early experiences with moving to cloud services, we can wrap up some key facts if you’re still looking for a few good arguments which may convince you and your team for moving to the cloud.
Scalability with minor effort
Overwhelming traffic is not a show stopper anymore
Auto scaling infrastructure is auto money for your pocket!
Reduce the multi region infrastructure challenge
Security
Cloud storage takes less risks than lost devices!
Stop to worry about underlying infrastructure security & maintenance
The disaster can come! Well proven Disaster Recovery Systems of great cloud providers are probably better than yours!
Speed & Performance
Increase your server-side performance on-demand as you go
Handle request peaks without costly overprovisioning your hardware
Reduced Costs = Greater Profit
Pay-as-you-go is better than buy-and-deprecate!
Financial advantages through transformation from CAPEX to OPEX
Makes your financial model a lot easier to plan and calculate
High chance to reduce your Time to Market
Emerging markets can perhaps make you even fly higher
Agility & User Experience
Stop your team from leaving their key competences
Increase your team performance with the latest collaboration tools
Satisfy your team by giving them a state-of-the-art feeling
Give more freedom for physical appearance
Ecological Footprint
Less eWaste for deprecated hardware
Savings on power supply and emissions
And if you’re doing it right, you earn a lot of FLEXIBILITY which is nowadays more important than ever! Our technological environment changes so fast that you may be outdated before you finish your project – ok, probably not that fast – but you should be prepared! If your cloud provider can’t handle your SLA or gets outdated by himself, you may move on to a new provider but be aware: cloud providers want to keep you as a customer, for sure! How that may end up badly for you we’ll discuss in the next section.
Pitfalls you better leave for others
So if you’re convinced now by the advantages of the cloud, we reveal you some pain points where others have been stumbling, so keep them in mind and don’t make these mistakes again!
Readiness of Organization One of the greatest mistakes you can do while dreaming of your cloud project is to underestimate the know-how that is necessary to do the job right. Many teams stumbled over the fact that their ideas and plans were great but they forgot about the people who finally have to do the job. Your change management team also has to be aware that going to the cloud can also have a negative impact on other teams, so other teams in your company may start fighting against that change. Imagine for example your customers have now direct access to your order- and payment system, will your sales team like that? Your change can also impact your hierarchical structure and long cherished employees lose their major value for the team. So it’s probably likely that not everybody in your organization will like your project, be aware of that!
Readiness of Application It doesn’t matter if you’re developing a new or transforming an existing application, you have to keep the drawbacks in mind that can come along with cloud services. Too many technical leaders made the mistake of thinking a simple lift-and-shift strategy will work out and they had to learn by hard that some refactoring is always necessary. This can start by underestimating the threshold of your cloud provider, continuing with disregarding upcoming latency that destroys your user experience or dependency libraries that are restricted to be used in the cloud (rare, but reported). So you better make a new concept first and think from the very beginning which components can be reused in which way. Think about your build steps, your deployment, your test environment, your architecture, current infrastructure, previous concurrency issues, clusters you want to use – convince yourself first and not later!
Data Privacy & Compliance Especially with the release of the new data privacy regulations from 2018, many companies got messed up and reported big question marks on how to continue with their services. So you can be lucky to start your cloud project after this disrupting wave but you shouldn’t think that the current data privacy agreements will last forever, this was rather just the beginning of regulations that were missed since the beginning of our digital lifestyle. Data privacy for your customers and data protection imposed by your company compliance will harden your way to the cloud – that’s for sure!
Remember your Business Idea! It doesn’t matter if you start a new project or lift some existing functionality to the cloud, if it creates advantages, any motivated employee will create further and further ideas how to improve and scale the impact of your cloud movement. So if you impress your environment, you can likely be sure that it doesn’t take a long time until the first ideas from other teams or team members land on your desk! So keep your primary business goals in mind and finish them first before you start working on other issues. If you’re a technical leader or you’re the one who pushes the cloud movement forward, take the advice to reflect yourself frequently and check if the tasks that you’ve created also have a value to your business. Too many become starry-eyed for cloud transformations and lift them to a holy-grail, which it isn’t for sure! So think twice if the current steps are really necessary and profitable for you and your team!
Vendor Lock-In It’s too sad to be true but in reality it’s a fact. We have two strongly diverging mindsets in our industry. There are the ones who believe everything should be free and open sourced and the others try to catch you in their cage and won’t ever let you go. So you got to be aware of the fact that it’s common under cloud providers that they will try to trap you with special offers, extra support, personal training sessions and individual implementation opportunities just to keep you as a customer as long as possible. So if you start to implement services based on proprietary interfaces it will be harder for you to relocate your codebase one day. Or if you build up a special knowledge customized to their services, you may wake up one day and realize that this knowledge is now worthless. So in other words, if their ship goes down, you’ll go down too! So be aware to avoid a vendor lock-in as much as you can!
Cloud Provider Evaluation Especially those who are new to this topic may stumble across a thing called SLA (Service Level Agreement) which defines the contract you sign with your cloud provider. If you don’t know exactly what it contains and where the borders of your cooperation are, you may run into trouble sooner or later. So it’s a good advice to get somebody for your team who was able to collect some experience in the cloud environment before and protects you from making the same mistakes that others did. It’s not about missing some contract details, it’s more about finding the right values and parameters that fit to your application and finally to your business case. As long as your project works fine you’ll probably never get in touch with your SLA that much, but once you got your customers or your boss in your neck, if there is an unexpected downtime, drastically rising costs due to a miss-configuration or a security lack, or even a disaster with a never ending recovery time, you will understand why knowing your SLA details is important. Take it as a good advice, especially at the beginning of your cooperation, that measurement and control is better than relying on trust. So the job is to monitor and measure as much as you can until you get a good feeling of what’s going on and how much responsibility you can take for it. Expect the unexpected!
Your guide through the jungle
So as far as you are familiar with the greatest advantages and biggest show stoppers now, we can start to create a cloud strategy that fits to your needs. Get your weapons locked and loaded and follow me through the jungle!
What do you want to create?
Remember your business goals and remember how others have lost the trail and you don’t want that the same happens to you, right? So define your project and especially your project scope precisely. What do you want to create? What is what you want to offer to your customer? Who is your customer and what do they really need? Think about your use-cases and match them accordingly. To make it a little bit more confusing, maybe you’ve heard about XaaS or *aaS which is one of the most blown up cloud buzzwords ever! What had once started with Software as a Service (SaaS), which basically means software delivered by cloud services, later transformed into a term that is almost used for any kind of service. Firewall as a Service, Payment as a Service, Analytics as a Service and even non- technological ones. But the idea behind the services-oriented model is great! So if you want to market your later product right, think in terms of services! Remember how you changed your business model from CAPEX to OPEX and your future customers want to have that too! They don’t want to buy a product, they have a goal and they are looking for the right service to reach their target as fast and as best as they can. And that’s a fact for any kind of customer. So you are serving them your best to let them reach their best! They other way around you have to think of the same terms for your goals. But this is a matter of your skills, compliance and amount of responsibility you want to share. Netflix for example thought they could fly high and even higher, so they built up their own infrastructure and data centers until they had to realize that they can never get this job done any better than Amazon does. From that point on they focus on what they really want to do: delivering awesome series! So think about the services you want to deliver, what kind of services would be a help for you, short term and long term! Imagine your project growth stages from time to time, where is your starting point and where do you finally want to get and find a match with a cloud provider that has all the abilities you need for your scale.
If you’re willing to create services that are mainly for corporate users, think of a private cloud with outsourced hosting facilities which is in fact technically nothing else than a restricted public cloud for your private usage. If your compliance or your business case doesn’t force you to keep your data in-house, guard it with a virtual private network access (VPN) and you are ready to go. If you once need to open some services for public usage you can still transform it to a hybrid cloud by removing the according restrictions or routing it through a public cloud.
Checklist
To not leave you unarmed, we’ve created a checklist that you may consider at the start of your cloud project. Critics are always welcome, if you’ve noticed some missing points or made different experiences throughout the years, let us know and write it in the comments down below!
What’s my business goal? Which requirements does it take?
How can I accelerate my time to market? Is the market ready for my concept?
How ready is my application or existing architecture?
Is there a need for refactoring? Does it also consider potential cloud
drawbacks like latency, downtime, noisy neighbours, migration, licenses and so on?
How ready is my organization and team? Who can do the job?
Will it change my business model? Does it disrupt my team?
Which restrictions do exist? Compliance? Security? Data privacy policies?
Where do I need help in terms of services? Long term? Short term?
Where do I want to grow? How much responsibility can I share?
Does the SLA match my requirements? How about Disaster Recovery?
Do the project requirements match my budget? Can it be done at the given time?
And as a final word: Don’t over engineer and don’t over manage it all! Keep it simple and straightforward and learn from your mistakes. Never lay back, be proactive, expect the unexpected and create your plan b – than you’re prepared for whatever it takes 😀
Annika Strauß – as324 Julia Grimm – jg120 Rebecca Westhäußer – rw044 Daniel Fearn – cf056
Introduction
As a group of four students with little to no knowledge of cloud computing our main goal was to come up with a simple project which would allow us to learn about the basics of software development for cloud computing. We had decided a simple game would do the trick. And to make it a little more challenging it should be a two player online game. First we thought of Tic Tac Toe but that seemed too simple. Then we took a look at the Chinese game of Go, but that was too complicated. In the end we agreed on Connect4. Not too simple. Not too complicated.
Getting started / Prerequisites / Tech Choices
In our first group meeting we sat down and brainstormed on all the requirements, features and technologies we would require to realize the game. We also tried to avoid coming up with too many additional features that would be nice to have but exceed our possible workload. The main focus was to get something running in the cloud.
Our version of Connect Four therefore should have a simple user login, so that one can play a game session with other players, interrupt the game and come back later to finish where one left off, which also means game sessions need to be saved between two players. We thought about matching players via a matching algorithm, in order to ensure that players of about equal strength get matched, but then we realized that was way too much effort and our focus really should stay on getting something done. So we decided on simply make it a random match up, or adding friends and connect via a friend list, since this is a study project, not a commercial game.
Of course we don’t really assume that millions of people are going to play our game at once, but we want to learn about scalability, so we are going to act as if no one has ever played it before and everyone in the world is going to act like it’s the new Pokémon Go. In reality it will probably just be the four of us and whomever we show it to.
We already knew beforehand that we would want to program the game in Python, simply because it is getting pretty popular in the web application field and we want to get some experience with it. Also, we finally want to learn something other than Java. But we also want to make sure Python actually is a good choice, so we’re going to check the pros and cons just to be sure and possibly decide on something else later.
We also agreed that we want to use a NO-SQL Database for saving game sessions, simply because they can store arrays and we want to get more experience with it. Again, we must check first if it’s a viable option.
Next question we needed to ask, is what infrastructure, what platform would we put this on. AWS? Our own hardware? IBM Cloud? What are the options there? Also, what requirements does our game bring with it for the platform? Do we need micro services? Should we use Docker? How is the scalability going to be handled? So many questions. It was time to ask the Master for some wisdom. So we talked to lecturer Thomas Pohl, who gave us some very useful insights.
Luckily for us IBM Cloud is available for free for students. And as Thomas works for IBM and has plenty of experience with it, it’s kind of a no brainer to go with IBM Cloud. As we found out, IBM Cloud already provides a great deal of infrastructure and platform, all handled by IBM, which allows us to simply focus on deployment. The service we are looking at in particular is called Cloud Foundry. It handles all the scaling, load balancing and everything automagically for you in the background, so that we can focus on simply getting our game running. It comes with a great variety of tools for almost any technology requirements we desire.
This is definitely the sandbox we want to be playing in. With some help from Thomas we came up with this relatively simple architecture:
So what exactly do we require from the cloud foundry? To answer this we first need to ensure we have all the technology choices figured out. Meaning, programming language, database, user login authentication and so on.
First we’ll start with the programming language. Python. Is it a good choice?
To get a clear impression of whether or not to use Python some research is necessary. After reading some online articles and watching some videos, we came to the conclusion that it is a good choice. Why? Similar to Java, it is an interpreted language and is easy to use. It is portable and has a huge library and lots of prewritten functions. The main drawback is that it is not too mobile friendly, but we’re not worried about that. And it is so simple, that other languages may seem to tedious in comparison. But it will get the job done quickly and it is simple to maintain. Debugging works very well, it has built in memory management and most importantly, it can be deployed in Cloud Foundry. So, Python is definitely a winner. Other languages may also be a good choice, but the point is not to find the best language, but to confirm that Python is not a bad choice. Cause, we want to learn Python, and we just want to be sure it’s not a waste of time. And from what I’ve read, Python is a pretty good choice.
SQL or NoSQL? We want to use NoSQL, but does that make sense? To answer this question we need to have a look at what kind of data we are actually storing and what the advantages and disadvantages are of using NoSQL and see how it compares to SQL.
SQL is great if one has complex data that is very interwoven and a write would mean updating several different places, which SQL manages well by merely linking the data via relations instead of duplicating it. NoSQL stores all relevant data in one place, which makes reading really fast, but making any changes can mean, having to replace all duplicates of the data. So, NoSQL is efficient if changes only need to take place in one area.
Now, the data we are storing is basically just a game session. This mainly consists of two users, which will not likely change. Then the game state, which only changes in the saved game. And once the game is done, the entire game session no longer needs to be stored anyway and can probably be deleted.
The player data will be coming from a UserDataBase supplied by App ID, a user login service provided by IBM. Feeding this data into either DB Type should be no problem. But the main reason for using a NoSQL Database remains arrays. As we will be storing the game progress in form of arrays, getting them translated into something a normal SQL database could handle would be way too tedious and NoSQL can handle storing arrays no problem.
We’ve also just touched on the topic of handling user accounts. As already mentioned IBM provides a called App ID. Rather than using a social media login service such as facebook provides or worse, coming up with a whole system on our own, we were very happy to discover this tool already existed in the IBM cloud foundry. So we gladly decided to use that.
Getting the show on the road. Sort of.
Now that we had decided on all of our technical resources it was time to actually make it all. So, we all created our IBM Cloud accounts and started setting everything up. Now we had to look around Cloud Foundry and figure out where and how we would find all the services we needed and which ones were the right ones for us. Our main services needed to be: a place for the main python app to live – Python Cloud Foundry a NO-SQL database – Cloudant a user login manager – AppID
Everything was relatively simple to find and set up. Our main app would be living in the Python Web App with Flask. A basic web serving application. For the NO-SQL database would use IBMs Cloudant. For the login of users we set up AppID. Now, I’m not going to go into detail on how we set it up, since it was pretty simple and anyone with a basic understanding of clicking through a webpage could have done most of this.
Everything we needed was in place. Our little playground was ready. Now came the really fun part. Actually writing the code. And with it, all the problems we needed to overcome, which would help us learn the ins and outs of cloud computing.
Random Problems we encountered
Merging Cloud Foundry Python Server with our Python-Flask Server
IBM Python Cloud Foundry delivers the following server.py:
We wanted to load our Flask Templates instead. How to do that? The server.py sets the folder “static” as the starting point, which contains the static index.html. But just putting the path onto the templates is not enough, as Flask is initalized with app.run(). Could we just replace server.js with a Python-Flask server? What on the server.js do we need so that the server still runs in the cloud. We picked out the port and added it in our app.js (Python-Flask server).
Read port selected by the cloud for our application:
Adding the port in the app start:
So we added this in the first row:
Success! It worked!
Couldn’t start the app from the Cloud anymore The app wouldn’t start anymore after pushing it to the cloud. We got an error message: “[errno 99] cannot assign requested address in namespace”. This meant that our app couldn’t be found under the URL in the cloud. The mistake was that when we loaded locally we were using “localhost:8000”, but in the cloud, that doesn’t work of course. What was the correct address in the cloud? Adding host=’0.0.0.0′ into app.run solved the issue. Now we could run the app from the cloud and locally.
Explanation If you bind localhost or 127.0.0.1 it means you can only connect to the service locally. 10.0.0.1 cannot be connected, as it is not ours. We can only connect to IPs which belong to our computer. We CAN bind 0.0.0.0 though, because this means, all IPs are on our computer, so that every IP can connect to it.
Templates not found!!! While two of us were working with Visual Studio Code, the templates for the front end html stuff were working quite nicely locally. But one of us was using PyCharm. And PyCharm did not know where the templates were and kept saying, wrong path, template not found. On our quest for answers we were victorious and found ProfHase85 in one of the threads on stackoverflow.com. We followed his wisdom and did as he said: “Just open the project view (View –> Tool Windows –> Project). Once there, though shalt right-click on your templates folder. Not left-click. Not double-click. And most certainly not center-click. No. Though shalt right-click on it. There you will find Mark as Direcory and from there you will find the Template Directory. It is there, that thee will find salvation. There you will set the path of your template and all shall come to life.” So, that worked great. Thank thee, ProfHase85.
The NO-SQL Cloudant DB
When getting our Cloudant DB running we immediately ran into several problems. When we tried to create the DB, as it said in the tutorial from IBM, nothing happened. We then found that the code checks to see if the instance is a cloud instance. And we were trying to run the code locally. So we needed to push our code first and run it from the cloud. The we needed to enter the domain name in the manifest.yml, which took us a while to find, which turned out to be eu-gb.mybluemix.net. Then everything decided to freeze anytime when the password was being prompted. Apparently we made a mistake when we created the DB when configuring the authentication method. We said to only use IAM. So we went back and created the database again, and during setup the authentication method to use both legacy and IAM. Now we also had our in our service credentials and creating the DB and connecting to the Cloudant service finally worked.
Using the DB statically as well as dynamically Now we were able to use Cloudant locally and in the cloud and could add static data in form of documents to our DB. The next problem we were facing was getting data like the username out of the user input into the document and making that data accessible again. Unfortunately our course on accessibility didn’t help in this case. Unfortunately the documentation on what is possible with Cloudant didn’t seem to be very expansive. Functions such as checking if a file exists were only possible locally, but not from the Python-Cloudant extension in the cloud. After several days of trying around we finally had the idea that maybe it was the accessibility functionality that was causing this. Maybe we needed to use IAM to access the Cloudant DB from the Python-Cloudant extension.
After a small issue with finding the right username we tried to connect using:
But were greetet with: Error: type object ‘Cloudant’ has no attribute ‘iam’
IAM requires at least Python-Cloudant version 2.9.0 and for whatever reason the version had in our requirements was 2.3.1. Problem solved. Connection finally established. And then the next problem came flying along. When updating a file: 409 Client Error: Conclict document update at document.save() What? OK. More reading up to do. Went and read through this article: https://developer.ibm.com/dwblog/2015/cloudant-document-conflicts-one/ This didn’t bring us much further, but it seemed to be better to use document.updatefield() rather than trying to go directly into the DB, in order to avoid simultaneous calls.
How to sort data in CloudantDB Three SQL programmers went into a NO-SQL bar. They came back out after five minutes because they couldn’t find a table.
In Cloudant data is stored in completely independent documents. This makes everything more flexible, but also very cluttered and difficult to differentiate when reading. Without any kind of sorting, all data needs to be searched for a specific ID. For our project we needed the following data structures:
We had to differenciate between users and game sessions. How could we accomplish this in Cloudant?
Use views? A view makes a query quick and easy. But anytime a document gets updated, so does the whole view, which is counterproductive with big data sets.
Partitions? We found out, one can create a partitioned DB in Cloudant, whereby naming the ids as follows: <partition>:<documentid>
In our case: games:gameID123 and users:userAbc
This way one can add the partition to the queries, resulting in much better search performance. And also the DB looks a lot tidier.
Search query example:
And that was that.
AppID – The “simple” user login service for web apps
IBM offers a nice little web app called AppID. Easy to integrate. Made for the cloud. Great security features. Easy, right? Well, they have all the code you need for Java or node.js. But not Python. So, a few more lines of code, research and effort. How hard can it be? AppID is based on OIDC (OpenID Connect). Since we used Python we needed to fall back on Flask-pyoidc. This module is a OIDC client for Python and the Flask framework which interacts with AppID for authentication.
Configuring the OpenID Connect Client
The metadata in “appIDinfo” serves as input for configuring the OIDC client.
Securing Web Routes After configuration the OpenID client can be used to secure single pages or sections (“Routes”) of the web-app. This is achieved by attaching a decorator to the rout definition:
“@auth.oidc_auth” ensures that the code only gets executed for authenticated users.
The first problems with using AppID already arose with establishing a connection. First we tried connecting with a direct approach via the create button, which show a connection in the browser, but not when pushing the app to the cloud. So, we created the service again directly in the project via the command line. And voila. The next test push got a connection.
Creating an instance of the AppID service We connected to “ibmcloud resource service-instance-create connect4AppID appid lite eu-gb”. After that an alias of the service instance is created in Cloud Foundry. Then we had “ibmcloud resource service-alias-create connect4AppID –instance name connect4AppID
And we had a finally established a connnection between our app and the AppID service. Seemed like things were coming along. But then of course we encountered the next problem. Turns out the redirect_uri doesn’t work with secured connections.
And then the next problem was that the AppID login widget was probably not going to work with our Python-Flask app either. So, we decided not to use AppID after all. Instead we created our own user login in python.
Sometimes something that looks like plug and play turns out to be plug and pray. And in this case our prayers weren’t heard. But now we know that one needs to thoroughly check the capabilities of these services before trying to implement them.
The heart of the game/Game Engine
Probably the most challenging part of creating this game was writing the main application, as the first slew of questions arose. How do you write a game for two players? How to connect the database? How do make it refresh when a player makes a move so the opponent can see it? How do make taking turns work, so the opponent is blocked from making another move?
Reloading the window after a player made a move After some online research we figured out we could use a socket server to handle the multiplayer functionality. But that seemed like way too much overhead as it meant possibly having to learn an entire new framework. The first issue we needed to takle was getting the web page to update/reload for both players, anytime one made a move. With Python-Cloudant one can listen to changes in the DB, but unfortunately this loop blocks all other actions in Python. Were cloud functions maybe the answer? They are like serverless event listeners. The function gets triggered when a watched event occurs. And fortunately the was even a quickstart template available from IBM Cloud. You can create an action sequence and a trigger on the DB. We would need to call a Flask template in the cloud function. But it was unclear if that was possible. So we tried a Python-Cloudant only approach instead. Same as before, but this time asynchronous. That way the feed can run continuously and listen for changes in the DB. But now the problem is that the asynchronous loop, which is waiting for changes in the DB, cannot be executed at the same time as the return render_template and is blocking. Which also means that it’s blocking server side, which is causing the website to freeze. According to a post on stackoverflow threading is a better solution. One can deamonize a thread and thereby make it run in the background. But, then it was time for a new approach. Getting a better understanding. What is a feed? What is a trigger? Several documentations and coffees later we finally came up with a proper solution.
The Solution to Cloud Functions After enough reading we finally uncovered the proper way to do this. Under cloud.ibm.com/functions/actions select “Trigger” 1. Create new Trigger. Select a premade Cloudant Trigger. Select your DB and the actions selected in the next step will be triggered anytime something changes in the DB. 2. Create the actions, which are supposed to be triggered. 3. Create a sequence, which executes the actions in order. 4. Add the actions to the sequence. 5. Add the sequence to the trigger. 6. Call the API link from getDocument in Javascript: wait with Async await and recursive timeout functions till the trigger gets triggered. If the DB change occures during the current game, the waiting players window gets reloaded. As we’ve never done anything with async await, cloud functions or APIs we definitely learned a lot getting this done.
Javascript code:
The Architecture
The final architecture
In the end, this is what our final architecture looked like. As seen in the square, our main app is comprised of Python-Flask, which handles all the python code and displaying the front-end view. The python code itself is the game engine and also handles all the back end connectivity. Javascript supports front end interactivity and listens to cloud functions. The cloud functions are bound to the DB and are set to trigger when changes in the DB occur. Javascript then reloads the web page to display the current state of the game. The DB contains the user data and current game sessions and is continuously updated by the main app. And the users sit in front of the web browser and enjoy all the magically appearing fun on their screen, without a clue of how much blood sweat it takes to make that magic happen.
Conclusion
What can one say? Everything always sounds so simple and easy in theory. But when it comes down to it, one often gets stuck on little things. Some choices we made in the beginning were good, some were quite challenging and some lead to dead ends. When we started out, we had only developed software for local use on computers or mobile devices. The closest we’ve gotten to something like cloud computing was maybe getting something to run over a network. It is quite challenging getting everything to run in the cloud. It’s a whole new game. Similar, but different rules. And even with all the services provided by IBM, we still ran into many obstacles. Especially when developing locally and then trying to make it work in the cloud. Also, getting all the different types of technology to work together is pretty tricky. Only with experience will one get good at it. Because you won’t know if the service can provide the functionality one requires until you try it. And often we needed additional features or functions we didn’t think of beforehand. Aspects we didn’t consider. The software technologies we’ve encountered may be very powerful, but with great power comes great confusion. But that is where progress happens. Not when everything is going smoothly, but when one is faced with difficult challenges. And we’ve had plenty. I would say that this project, this course, has been one of the most beneficial in our studies at the HdM. We had the opportunity to get our hands dirty, with expert guidance in a safe environment. The experience we’ve gained is priceless. Our understanding of cloud computing and our ability to develop software for such has progressed several levels. And since this was our main goal I would have to say that our project was a complete success.
In the last months, nearly everybody has been talking about Kubernetes. It’s incredible! This semester the Stuttgart Media University even held a training course on this topic. For DevOps or “cloud-computing specialist” mastering Kubernetes and the concepts around it is becoming more and more important.
You must be logged in to post a comment.