<\/p>\n\n\n
Note: you need the below mentioned files in the directory so that the docker-compose setup can run! If you want access to our repository please contact us. <\/em><\/p>\n\n\n If you want to start the setup run this cli command: <\/p>\n At this point you can see the details of the individual files:<\/p>\n\n\n\n If you want to run the testsuite within a container you can adjust the command and nodeenv as follows:<\/p>\n Previously a mongodump was generated into the radcupDevSample.json folder so that some testdata is available. This dump is used by the mongo-importer container which migrates the testdata into the mongodb container of each developer.<\/p>\n This Dockerfile represents our Backend:<\/p>\n Watch the cli-output of docker-compose on your machine. After all necessary images have been downloaded and built, you can access the backend within the web-container. Open a browser tab and enter:<\/p>\n http:\/\/localhost:3000\/api<\/a><\/p>\n You should see a JSON Object with this content:<\/p>\n {“message”:”this will be a beerpong app”}<\/p>\n <\/p>\n Finding:<\/p>\n Now all developers can use this nice setup in a few minutes and can now focus fully on the business logic development. We have also created a good basis for deploying the backend via the docker containers into the cloud and making it available accordingly.<\/p>\n\n\n As mentioned above, there is an ionic-hybrid app which can be linked to the radcup-backend. Since we didn’t want to install ionic and cordova, etc. on our local devices. We decided to move the frontend application to a docker-compose container setup, too. However, since our focus in this project is on the backend, the frontend is not discussed further at this point.<\/p>\n\n\n\n Under Windows 10 docker required \u201cLinux Containers\u201d not \u201cWindows Containers\u201d. This can be easily changed at any time in the Docker settings. At this point we had the problem that the error message was misleading when using Docker <\/strong> containers. The error message said that the version number was incorrect, which lead us onto the wrong track. To solve the issue only changing the containers to \u201cLinux Containers\u201d did the trick.<\/p>\n\n\n\n Docker also needs the Windows feature \u201cHyper-V\u201d. But if it is not installed, docker prompts one with an expressive error message. The Docker environment also needs a shared drive for the docker processes. Fortunately, Docker does this automatically and sets the permissions to the currently logged in Windows user account.<\/p>\n\n\n\n Since the backend, as mentioned before, was developed a long time ago, it was necessary to perform a dependency check. On the one hand to identify the partially obsolete or unneeded dependencies and on the other hand to remove warnings.<\/p>\n\n\n\n Because a manual check of the entries, is very time-consuming, a tool was needed which carries out the analysis. For this reason, various tools were evaluated and tested.\nIn the end, our choice was The following command is sufficient to install it (prerequisite: npm is installed):<\/p>\n<\/div>\n\n\n\n After the installation, you have to navigate to the directory where the Example Output:<\/p>\n By analyzing the dependencies, some updates were performed, some even security relevant, and several dependencies were removed.<\/p>\n Before the analysis:<\/p>\n<\/div>\n\n\n\n After the analysis:<\/p>\n<\/div>\n\n\n\n As you can see from the two outputs, analyzing the dependencies has helped a lot. For example, the number of packages has been reduced and the warnings were removed.<\/p>\n<\/div>\n\n\n\n Finding:<\/p>\n\n\n\n In the middle of the project there was a problem with the npm package ‘event-stream<\/a>‘, this distributed malware and tries <\/strong>to steal from users of the Bitcoin Wallet Copay. Our testing framework is Mocha. If started it will per default run all .js<\/em> files \/ tests defined in the test folder of the application. In the above example you can see a test which checks if a game can be created. The result of the call is checked.<\/p>\n\n\n\n The following code shows a section of our package.json:<\/p>\n\n\n\n The file contains three script sections “start”, “test” and “testfromclusterandrun”. These define how the application can be started. If “test” is selected, the corresponding test is started. (NYC Documentation<\/a>) The Code Coverage can also be tested within the CI-pipeline and depending on the result can cause the test pipeline to fail. To let the test pipeline, fail if a certain threshold is reached, the following command is required. The command was added to the test script in the package.json.<\/p>\n\n\n\n (Documentation<\/a>) With the shown parameters the command checks several coverage types:\n The following picture shows the results of each coverage type and shows that the pipeline fails if the threshold is not met. To show this, the branches threshold was set to 100%:<\/p>\n\n\n\n In the table an additional type Stmts can be seen. This stands for Statements, but the coverage of this type can’t be tested. We also added the following command to get a more precise coverage info as overview:<\/p>\n\n\n\n (Documentation<\/a>)<\/p>\n\n\n\n The following picture shows an additional summary of the results after the table, which can already be seen in the previous picture. This run also ran successfully.<\/p>\n\n\n\n We searched a test that will cause the pipeline to fail (return non-zero) if the javascript implementation, has any errors or warnings. To test the implementation the following command has to be added to the test script in the package.json. Also, the jshint npm package has to be installed and therefore we added a reference to the package.json<\/p>\n\n\n\n (Documentation<\/a>)<\/p>\n\n\n\n As one can see The following picture shows an example of found errors:<\/p>\n\n\n\n At first jshint seemed to run into an endless loop. Because of the npm installation a folder “node_modules” was added that we forgot. And testing this folder just took way more time. Of course, we didn’t want to let jshint test this folder, especially because we can’t really correct errors\/warnings in it and also because it isn’t in the scope of our project.\nBut we also didn’t want to leave it at just testing specified files, because we of course don’t want to reference each file here, that will be added to the project.\nTo solve this issue one can add a file named If there is a warning which can’t be solved it can be marked, in the code, so that it will be ignored as follows:<\/p>\n\n\n\n As a short side node, everything that is in between those ignore comments is completely ignored, which means if one for example splits a several lines long command with those comments, jshint will see the part outside of the comment as incomplete.<\/p>\n\n\n\n A Single line could also be ignored like this:<\/p>\n\n\n\n In short Swagger<\/a> can be used to design and document an API. We already had an API and therefore we only needed to document it.<\/p>\n\n\n\n Swagger documents APIs and while doing this it is also an interactive tool to try out the API, so one can see for himself how the API works.<\/p>\n\n\n\n In this picture one can see a GET-API request which will return a user if given a correct email. This information can be seen and read quickly from the developer who accesses the swagger UI. Also, you can see that this GET call has a required parameter, the email address. An email address can be inserted and the call can be executed. Then you can see the request URL that was used to perform this API call. Also, the return object can be compared to an example value. This example value is also accessible if the request was not sent which means one does not have to execute it.<\/p>\n\n\n\n Whilst trying out the API we also found a security issue with our API. Any logged in user can access any API path, which means any user can access, alter and delete any data.<\/p>\n\n\n\n For the first setup, we found this tutorial<\/a>, very helpful.<\/p>\n\n\n\n We decided to use Swagger-UI-Express<\/a>, because it is actively supported. In the \u201epackage.json\u201c file one must only add the following line to the At the time of this project the current version of the Swagger UI was 4.0.1. With this line NPM will also install the Swagger UI into the Docker image.<\/p>\n\n\n\n In the \u201eserver.js\u201c file, the following lines have to be added: In the same folder as the previous two files, one more file has to be added, the “swagger.json” file. It defines the documentation which will be displayed on the swagger website. For a “how to” see the next topic “swagger.json” below, but for the moment a simple example json suffices.<\/p>\n\n\n\n After building and starting the Docker image we were able to access the Swagger website under http:\/\/127.0.0.1:3000\/api-docs<\/a><\/a>. (We were testing this on a local computer, hence the call to localhost.)<\/p>\n\n\n\n Useful links<\/strong><\/p>\n\n\n\n To create a swagger.json we found the following links helpful:<\/p>\n\n\n\n A short and easy introduction<\/a>.<\/a><\/p>\n\n\n\n The official Swagger Editor<\/a> (optional). Detailed introduction<\/a> into the creation of a “swagger.json”.<\/a><\/p>\n\n\n\n OpenAPI-Specification<\/a>.<\/a><\/p>\n\n\n\n Building a swagger.json\/yml<\/strong><\/p>\n\n\n\n We developed a swagger.yml and then converted it into a swagger.json, because the swagger.yml syntax is easier to use.<\/p>\n\n\n\n We tried to use the swagger.yml directly without the swagger.json but we weren’t able to get it working. If we used the .yml-file in the required command, the .yml-file would be loaded, but the internal parser would throw weird errors. Even though the .yml-file was written according to specification and the swagger editor wouldn’t throw any errors or warnings.<\/p>\n\n\n\n The structure of a swagger.yml(\/.json) can be divided into three parts. The first contains general info about the swagger file and API. The second part contains the API paths itself. And the last part contains the models which are used by the API.<\/p>\n\n\n\n In the following sections we will introduce those three parts and will provide a short overview over each part.<\/p>\n\n\n\n swagger.yml part one:<\/strong><\/p>\n\n\n\n Under swagger.yml part two:<\/strong><\/p>\n\n\n\nbash-3.2$ pwd<\/span>\n\/tmp\/radcup_localdevexample\n\n#we use the tool tree-cli, with which you can see the directory structure of our project:<\/span>\nbash-3.2$ tree\n.\n\u251c\u2500\u2500 Dockerfile\n\u251c\u2500\u2500 docker-compose.yml\n\u251c\u2500\u2500 importer\n\u2502 \u251c\u2500\u2500 Dockerfile\n\u2502 \u2514\u2500\u2500 radcupDevSample.json\n\u2502 \u2514\u2500\u2500 radcupDevelopment\n\u2502 \u251c\u2500\u2500 games.bson\n\u2502 \u251c\u2500\u2500 games.metadata.json\n\u2502 \u251c\u2500\u2500 system.indexes.bson\n\u2502 \u251c\u2500\u2500 users.bson\n\u2502 \u2514\u2500\u2500 users.metadata.json\n\u2514\u2500\u2500 src\n \u251c\u2500\u2500 README.md\n \u251c\u2500\u2500 config.js\n \u251c\u2500\u2500 controllers\n \u2502 \u251c\u2500\u2500 auth.js\n \u2502 \u251c\u2500\u2500 game.js\n \u2502 \u2514\u2500\u2500 user.js\n#--------------SNIP--------------------<\/span>\n# the src directory contains all necessary files of the node.js backend<\/span>\n\nThe content of the individual files (Dockerfile, docker-compose.yml, ...)\nis described in the following blogpost.<\/pre>\ndocker-compose up --build<\/pre>\n\n\n\n
1. Docker-Compose.yml:
<\/h4>\n\n\n\ndb:<\/span> #this is our db-container which uses the default MongoDB Image from Dockerhub<\/span>\n image:<\/span> mongo\n ports:<\/span>\n -<\/span> \"27017:27017\"<\/span> #default exposed mongod ports<\/span>\nmongo-importer:<\/span>\n build:<\/span> .\/importer #change directory to importer folder and build this dockerfile (see below for this file) <\/span>\n links:<\/span> #because the importer-container needs access to the db we have to link it<\/span>\n -<\/span> db\nweb:<\/span> #builds the radcup backend dockerfile (see below for this file) <\/span>\n build:<\/span> .\n links:<\/span> #because the container needs access to the db we have to link it<\/span>\n -<\/span> db\n ports:<\/span>\n -<\/span> \"3000:3000\"<\/span> #exposed api ports<\/span>\n environment:<\/span>\n NODE_ENV:<\/span> development #define the environment <\/span>\n command:<\/span> [\"npm\"<\/span>, \"start\"<\/span>] #define the command which has to be executed in the container<\/span>\n<\/pre> \nNODE_ENV:<\/span> test\ncommand:<\/span> [\"npm\"<\/span>, \"test\"<\/span>, \"--exit\"<\/span> ]<\/pre>\n\n\n\n2.1 mongo-importer Dockerfile:
<\/h4>\n\n\n\nFROM<\/span> mongo\nCOPY<\/span> radcupDevSample.json \/radcupDevSample.json\n<\/span>CMD<\/span> mongorestore -h db \/radcupDevSample.json &&\\\nmongo db\/radcupDevelopment --eval \"db.createUser({user: 'root', pwd: 'toor', roles:[ { role:'readWrite',db: 'dbAdmin'} ] } ) ;\"<\/span> &&\\ \n<\/span>mongo db\/radcupProduction --eval \"db.createUser({user: 'root', pwd: 'toor', roles:[ { role:'readWrite',db: 'dbAdmin'} ] } ) ;\" <\/span> &&\\\nmongo db\/radcupTest --eval \"db.createUser({user: 'root', pwd: 'toor', roles:[ { role:'readWrite',db: 'dbAdmin'} ] } ) ;\"<\/span> <\/pre>\n\n\n\n2.2 Web-Dockerfile:
<\/h4>\n\n\n\nFROM<\/span> node:jessie #use node, based on a Debian jessie image<\/span>\nCOPY . \/home\/app #copy the backend source files to \/home\/app<\/span>\nWORKDIR \/home\/app\/src #set the default workdirectory for the container<\/span>\nRUN npm install #install dependencies with nom<\/span>\nUSER<\/span> node #set the execution user to node! (This is a important security feature - if you do not do this the app-process is executed as root user! <\/span>\nEXPOSE<\/span> 3000<\/span> #expose the api port <\/span><\/pre>\n\n\nFrontend
<\/h4>\n\n\n\nDocker under Windows 10<\/h3>\n\n\n\n
Every time the runtime environment is started, docker needs access to the “Windows Hosts Files”. However, antivirus software can block this access, depending on your settings. Unfortunately, the solution seems to be to change the antivirus software settings.
<\/p>\n\n\n\nStep 2: Dependency Update<\/h2>\n\n\n\n
npm-check<\/code><\/a>.<\/p>\nnpm install -g npm-check<\/pre>\n\n\n\n
package.json<\/code> is located. If the command npm-check<\/code> is executed, the file and its dependencies will be analyzed.<\/p>\n![\"npm-check](\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2019\/01\/npm-check.png\")
<\/p>\nStep 4\/6 : RUN npm install\n---> Running in 24e38874f36f\n\n> nodemon@1.18.7 postinstall \/home\/app\/src\/node_modules\/nodemon\n> node bin\/postinstall || exit<\/span> 0\n\nLove nodemon? You can now support the project via the open collective:\n> https:\/\/opencollective.com\/nodemon\/donate\n\nnpm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.4 (node_modules\/fsevents):\nnpm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.4: wanted {\"os\"<\/span>:\"darwin\"<\/span>,\"arch\"<\/span>:\"any\"<\/span>} (current: {\"os\"<\/span>:\"linux\"<\/span>,\"arch\"<\/span>:\"x64\"<\/span>})\n\nadded 356 packages from 616 contributors and audited 2535 packages in 15.28s\nfound 0 vulnerabilities<\/pre>\n\n\n\nStep 4\/6 : RUN npm install\n---> Running in 9fb8e1bf3f43\nadded 141 packages from 498 contributors and audited 283 packages in 6.056s<\/pre>\n\n\n\n
As a result, npm-install stopped running and the affected package had to be updated to a newer version without malware.<\/p>\n\n\n\nStep 3: Testsuite improvements<\/h2>\n\n\n\n
Our test setup<\/h4>\n\n\n\n
(Documentation<\/a>)<\/a>
Here is one example of a test, the shortened version of the game tests:<\/p>\n\n\n\n[...]\n\/* Game Testsuite *\/<\/span>\ndescribe('Games API Testsuite.'<\/span>, function<\/span>(<\/span>) <\/span>{\n [...]\n it('should create a game'<\/span>, function<\/span>(done<\/span>) <\/span>{\n request(server.app)\n .post('\/api\/games'<\/span>)\n .auth(user.email, user.password)\n .send( game )\n .end(function<\/span>(err, res<\/span>)<\/span>{\n expect(err).to.eql(null<\/span>);\n expect(res.status).to.eql(200<\/span>);\n expect(res.body.desc).to.eql(game.desc);\n expect(res.body.results.winner).to.eql('No Winner'<\/span>);\n expect(res.body.results.endTime).to.eql(null<\/span>);\n expect(res.body.players.one).to.eql(user.username);\n done();\n });\n });\n [...]\n});<\/pre>\n\n\n\n{\n \"name\"<\/span>: \"radcup\"<\/span>,\n \"version\"<\/span>: \"1.1.1\"<\/span>,\n \"private\"<\/span>: true<\/span>,\n \"scripts\"<\/span>: {\n \"start\"<\/span>: \"node radcup\"<\/span>,\n \"test\"<\/span>: \"NODE_ENV=test nyc .\/node_modules\/.bin\/mocha --exit --timeout 3000; nyc report --reporter=text-summary; nyc check-coverage --branches 20 --functions 60 --lines 50; jshint '.\/' --verbose\"<\/span>\n \"testfromclusterandrun\": \".\/node_modules\/.bin\/mocha --exit --timeout 3000; NODE_ENV=prodDBatlas node radcup\"<\/span>\n },\n[...]<\/pre>\n\n\n\n
We use the nyc package to handle mocha. “testfromclusterandrun” is used to start the application on the cluster. Before that a test is started, to see if everything works on the cluster.<\/p>\n\n\n\n
(NYC with Mocha<\/a>)<\/p>\n\n\n\nNODE_ENV=test<\/code>: This variable has to be set to test<\/code> for the integrated babel-plugin-istanbul<\/code> of the NYC package to work. This is required so that NYC fully works with Mocha.<\/p>\n<\/div>\n\n\n\nnyc .\/node_modules\/.bin\/mocha --exit --timeout 3000;<\/code> starts the mocha framework and therefore the test. The exit command is necessary because otherwise this command will not return. We also defined a time-out in case it still doesn’t return, for example if the tests have an issue. The other commands will be explained in further detail, in the following sections.<\/p>\n<\/div>\n\n\n\nHow to check Code Coverage?<\/h4>\n\n\n\n
nyc check-coverage --branches 20 --functions 60 --lines 50;<\/pre>\n\n\n\n
<\/p>\n\n\n\nbranches >= 20%, functions >= 60%, lines >= 50%<\/code><\/p>\n<\/div>\n\n\n\n![\"\"](\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2019\/01\/tests.png\")
<\/figure>\n\n\n\n
<\/p>\n\n\n\nnyc report --reporter=text-summary;<\/pre>\n\n\n\n
![\"\"](\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2019\/01\/summary.png\")
<\/figure>\n\n\n\nHow to test the javascript implementation?<\/h4>\n\n\n\n
jshint '.\/'<\/span> --verbose;<\/pre>\n\n\n\n'.\/'<\/code> the command references the folder where it is started from, in our case this is the "src" folder of our project.<\/p>\n<\/div>\n\n\n\n![\"\"](\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2019\/01\/jshint-error.png\")
<\/figure>\n\n\n\n
<\/p>\n\n\n\n.jshintignore<\/code>, in the directory the jshint command will be executed (the src folder in our case). In the file one can specify which files\/folders should be ignored. Those exceptions will be interpreted using the minimatch npm module. Also, the --exclude-path<\/code> flag needs to be added once, above those path patterns. Here one can see an example:<\/p>\n<\/div>\n\n\n\n--exclude-path\n.\/node_modules<\/pre>\n\n\n\n
\/* jshint ignore:start *\/\n\/\/ code over several lines (for example a function) that causes a warning\n\/* jshint ignore:end *\/<\/pre>\n\n\n\n
oneLineOfCode(); \/\/ jshint ignore:line<\/pre>\n\n\n\n
Step 4: Swagger integration<\/h2>\n\n\n\n
What is Swagger?<\/h4>\n\n\n\n
![\"\"](\"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2019\/01\/user-by-mail-executed.png?fit=656%2C603&ssl=1\")
<\/figure><\/li><\/ul>\n\n\n\nSwagger setup<\/h4>\n\n\n\n
dependencies<\/code>:<\/p>\n<\/div>\n\n\n\n\"swagger-ui-express\": \"^4.0.1\"<\/span><\/pre>\n\n\n\n
<\/p>\n\n\n\nvar<\/span> swaggerUi = require<\/span>('swagger-ui-express'<\/span>);\nvar<\/span> swaggerDocument = require<\/span>('.\/swagger.json'<\/span>);<\/pre>\n\n\n\nswagger.json<\/h4>\n\n\n\n
(This website comes with an detailed example Petstore<\/em>. This example can be downloaded as an swagger.yaml or swagger.json.)<\/p>\n\n\n\nswagger:<\/span> \"2.0\"<\/span>\ninfo:<\/span>\n version:<\/span> \"1.1.1\"<\/span>\n title:<\/span> \"radcup\"<\/span>\n description:<\/span> \"radcup backend API\"<\/span>\nhost:<\/span> \"159.122.181.248:30304\"<\/span>\nbasePath:<\/span> \"\/api\"<\/span>\ntags:<\/span>\n- name:<\/span> \"Game\"<\/span>\n description:<\/span> \"API for games in the system\"<\/span>\n- name:<\/span> \"Statistic\"<\/span>\n description:<\/span> \"API for statistics in the system\"<\/span>\n- name:<\/span> \"User\"<\/span>\n description:<\/span> \"API for users in the system\"<\/span>\nschemes:<\/span>\n-<\/span> \"http\"<\/span>\nconsumes:<\/span>\n-<\/span> \"application\/json\"<\/span>\nproduces:<\/span>\n-<\/span> \"application\/json\"<\/span>\n[...]<\/pre>\n\n\n\ninfo<\/code> belongs general information about the API that is in use. host<\/code> and basePath<\/code> are used to define the URL for each API call. The host<\/code> address should of course be set to the address of the server. tags<\/code> is used to group the API calls and one can also describe each group\/tag.<\/p>\n<\/div>\n\n\n\n[...]\npaths:<\/span>\n[...]\n \/games\/{game_id}:\n[...]\n