Cryptomining Malware – How criminals use your devices to get wealthy!

Has your computer ever been slow and you couldn’t tell what the problem was? Nowadays, illicit cryptomining can cause those performance problems. It dethroned ransomware as the top cybersecurity threat in 2018. (Webroot Threat Report 2018) A simple website visit can start the mining process as a javascript running in the background of the browser or an accidentally installed malware on your computer. These two examples for different modes of illicit cryptomining are called browser-based cryptojacking and binary-based cryptomining. In both cases hash-rates can be up to medium-sized mining farms. This blog article will give an overview over binary-based cryptomining malware. In that case the mining process is embedded in the payload of a malware. Criminals hide it as good as possible which makes it hard to detect to gain a massive income. All the tools they need to start a malicious cryptomining business are easy to get in underground markets. For example Malware can be purchased for a few dollars (e.g. the average cost for an encrypted miner for Monero XMR is 35$). We will also take a quick look at how companies are legally using cryptomining to monetize web content as an alternative business model.

Source: “A First Look at the Crypto-Mining Malware
Ecosystem: A Decade of Unrestricted Wealth”
by S.Pastrana and G.Suarez-Tangil

Basics

In this part we will have a look on basics which are required for this article.

Mining Pools

Since more and more computational power is required to calculate cryptocurrencies mining pools are popular. A mining pool is a collection of miners who pooled their resources together to mine a cryptocurrency and share their rewards for every calculated block. But there are advantages and disadvantages of mining pools. One main advantage is a more stable income by using mining pools due better chances to solve a cryptographic puzzle for the next block. On the other hand miners have to share their rewards which can be seen as a disadvantage, but without enough resources the outcome is potentially lower. (Mining Pools and How They Work 2019)

Cryptocurrency Wallets

Cryptocurrency wallets are not exactly like wallets we know from daily life. Users can  monitor their balance and send money or execute other operations. The virtual wallets contain a private and public key to perform operations. The keys are used to access the public blockchain adresse and confirm a transaction. The private key is used for the transaction of the wallet owner an the public key is similar to a international Bank Account Number. For example, if someone wants to transfer money to your wallet this person needs your public key, but you don’t get actual money on your account. The transaction is only identified by a transaction record on the blockchain and a balance change in your cryptocurrency wallet. Important to know is that the private key is totally unique and in the case of a lost of it the wallet won’t be accessible anymore for its owner. (What is a wallet 2019)

Binary-based Mining

Binary-based mining is the common way to mine cryptocurrency. Users install a program or application on a device to mine. That would be the legitimately way as the user gets the rewards for accomplished performance. It gets illicitly if a malicious actor gains access to the users computer power through a malware and mines for their own benefits. The mining software would be installed on the computer and drains the CPU performance of the victim and the payments for the rewards are going to the wallet of the attacker.

Browser-based Mining

In addition to the two types of illicit cryptomining we will have a brief look at browser-based cryptojacking. Illicit browser-based mining is continually rising in the past years. As in the introduction mentioned it is really simple to run into it. As long as an user navigates on a website and uses the services the mining process is running. The browser of the victim performs scripts which execute the mining progress. It is only illicit if the user is not aware of it. There are some websites that use this method to generate money legally for maintenance, as donations or as a substitute for advertising. For example the UNICEF organization in Australia used this method to provide donations. (UNICEF Donation 2019)

Source: thehopepage.org

UNICEF notifies the users about the procedure and started the mining operation after an agreement to the terms on the devices of the users which makes the activity legitimate.

Key Enablers of Illicit Cryptomining

The factors of key enablers of the malicious actors to conduct were analyzed by the cyber threat alliance in 2018 (The illicit Cryptocurrency Mining Threat 2018). Let’s have a look on these factors :

  • It’s more profitable since the increased value of cryptocurrencies.
  • Cryptocurrencies with anonymity for transactions, such as Monero and Ethereum that can be mined with personal computers or IoT devices and create a potential attack surface.
  • Malware and browser-based exploits are easy to use and easily available.
  • The number of mining pools is increasing, facilitating the pooling of resources and providing a scalable method for mining.
  • Enterprises and individuals with inadequate security measures are targets for malicious actors and are unaware of the potential impact on their infrastructure and operations.

Most popular Cryptocurrency

Since the popularity of Bitcoin dropped for illicit cryptomining over time, because of the increased amount of time to calculate a single coin, underground economies focus other cryptocurrencies like Monero (XMR). Monero is the most popular cryptocurrency for illicit cryptomining, because of the use of innovative ringsturcutres and decoys to retain transactions completely untraceable. (Webroot Threat Report 2019) Researchers found out that 4,32% of the circulating XMR was mined with cryptomining malware which has an estimated revenue of nearly 57 million USD. (First Look 2019)

Damage caused by Cryptomining

Cryptomining can cause serious damage in different ways. It is draining the CPU usage which could be detected easily during the use of an infected computer, but criminals use distinct methods to evade detection of the mining process. These methods will be explained later in the article. Another main damage is the increased power supply of the CPU or GPU which cause high electricity bills. Through the excessive load of computer components during the process the hardware deteriorates rapidly.

How Criminals spread the Malware

The common approach to spread the malware is to host it in public cloud storage sites such as Amazon Web Services (AWS), Dropbox, Google Drive, Github and so on. Criminals often hide the malware in stock mining software for instance xmrig or xmr-stak to get access. Another approach is the use of botnets which are offered as pay-per-install (PPI) services in the deep web markets. (First Look 2019)

Source: “A First Look at the Crypto-Mining Malware
Ecosystem: A Decade of Unrestricted Wealth”
by S.Pastrana and G.Suarez-Tangil

A Further and probably the oldest approach to transfer these executables to a user is to deliver malicious spam or exploit kits by email. The malware starts to infect the computer after opening the attachment. Once the machine installs the malicious mining software it starts to mine cryptocurrency. In some cases the malware begins to scan the network for more accessible devices and tries to infiltrate them with an exploit.

Mechanisms to evade Detection

As earlier mentioned most of the cryptomining malware make use of stealth techniques. The more difficult it is to detect them, the longer the malware can utilize the computing power. The method idle mining starts the mining process only when the computer is in idle state and no operations are running for a certain time. For example if you leave your computer without turning it off for a longer time the mining process starts and lasts as long as there is no interaction with the computer. After an interaction the process shuts down and the performance is free for the user. The programmers of the malware take care in many ways to evade detection. There are cryptomining malwares with different modes for desktop  and laptop to get the best computing power for the infected device. For instance the malware on the laptop would take just as much performance as possible to keep the fans quiet. Another technique is the execution stalling code which makes the process almost invisible when Task Manager is running. If the Task Manager is running the mining process is slowing down the CPU utilization. It is possible to bypass this execution stalling code by using other process monitoring applications. Furthermore cryptomining campaigns use domain aliases (e.g. CNAME) to prevent blacklisting of mining pools.

Source: coindesk.com [Accessed 4. September 2019]

The image above shows how the execution stalling of the malicious miner called Norman works. It is based on a XMRig-based crypto-miner and avoids detection. After the Task Manager opens the malware stops operating and re-injects itself as soon as the Task Manager is closed.

Source: “THE ILLICIT CRYPTOCURRENCY MINING THREAT” by the Cyber Threat Alliance

In the figure above we can see another stealth technique which was described by Palo Alto Networks. This cryptomining malware uses only 20 percent of the machines CPU. The benefits of using this method is to persist longer on the infected machine and avoid detection as the mining performance is lower than possible.

Campaigns

Source: “A First Look at the Crypto-Mining Malware
Ecosystem: A Decade of Unrestricted Wealth”
by S.Pastrana and G.Suarez-Tangil

If we have a look on the illicit cryptomining campaigns we see a small number of actors that monopolize the cryptomining malware ecosystem. It is common to see campaigns mining in various pools. The most popular are crypto-pool, dwarfpool and minexmr and there are successful campaigns that are running for over 5 years without getting detected. In the next part we will have a look on the most profitable campaigns which were still active in 2018 and were analysed by Sergio Pastrana of the Carlos III University of Madrid and Guillermo Suarez-Tangil of the King’s College London and which this article is based on.

The Freebuf Campaign

The Freebuff Campaign was and probably is still active since 2016 and has mined over 163K XMR (approx. 18 million USD). It is named “Freebuf” because of the main domain xt.freebuf.info. Statistics of two banned wallets have shown that they were connected from 5,352 and 8,009 different IPs and had mined 362.6 and 1,283.7 XMR. The campaign used 7 wallets which are connected to the mining pools minexmr and crypto-pool by using domain aliases. After the ban of the two wallets the operator changed to another mining pool.

Source: “A First Look at the Crypto-Mining Malware
Ecosystem: A Decade of Unrestricted Wealth”
by S.Pastrana and G.Suarez-Tangil

In the figure above we can see the structure of the Freebuf campaign. The green nodes are malware miners and are connected to wallets shown as blue nodes. Gray and pink nodes represent the infrastructure of the campaign. Therefore the gray nodes represent the contacted domain server and the pink node shows the malware hosts. The red and orange nodes symbolize additional malware. As earlier mentioned the campaign uses 7 wallets which we can see in this graph. All the malware miners are connected to one of the wallets and linked to one mining pool which is hidden behind a CNAME alias domain. We can see three different domain servers in this graph: xt.freebuf.info, x.alibuf.com and xmr.honker.info. All of them have been aliases of common used mining pools. For example xt.freebuf.info and xmr.honker.info are aliases for minexmr and x.alibif.com for crypto-pool.

The USA-138 Campaign

The USA-138 has mined at least 6,709 XMR (approx. 651K USD) using 5 wallets. An interesting point about this campaign is it has mined the cryptocurrency Electroneum (ETN) with earnings of 314.18 ETN in late 2018. It was worth less than 5 USD, but it was a speculative for the future.

Source: “A First Look at the Crypto-Mining Malware
Ecosystem: A Decade of Unrestricted Wealth”
by S.Pastrana and G.Suarez-Tangil

The figure above shows the structure of the USA-138 campaign. The meaning of the nodes are the same as previously described in the Freebuf campaign chapter.

Countermeasures

The simplest method to prevent cryptomining malware is to keep the anti virus updated and avoid to download tools from suspicious websites. Furthermore the operating system should stay updated to seal vulnerabilities and prevent injections. Another possibility is to track the network data transfers and web-proxies to detect attacks. In case of suspicions that the computer performance is slower than normal and illicit cryptomining might drain the CPU/GPU load it’s useful to monitor the activities and analyse if any suspicious services are running. (Cryptominer Protection 2019)

The most successful approach to stop illicit cryptomining was the change of the Monero PoW (Proof-of-Work) algorithm in 2018 which stopped approximately 73% to 90% of the campaigns, because their malware couldn’t adjust to the changes.

Conclusion

The fact is that cyberattacks with cryptomining malware is constantly rising and the enterprises and individuals are most of the time not aware of the situation. It causes enormous performance problems and hardware deterioration. The attackers are getting more and more creative with the use of stealth techniques which makes it hard to detect. They got an almost anonymous platform to generate money on victims devices with the cryptocurrency and cryptomining. That’s why it is unlike ransomware, where the victim is aware of the situation and can deal with it. Cryptomining attacks are most of the time silent and without an awareness of this problem it will go on. As an common user you can only have a monitor your CPU/GPU performance if there are any suspicious performance drops. Keep your antivirus software and operating system updated.

References

The (in)security about speaker legitimacy detection

For the most of us, voices are a crucial part in our every-day communication. Whether we talk to other people over the phone or in real life, through different voices we’re able to distinguish our counterparts, convey different meanings with the same words, and – maybe most importantly – connect the voice we hear to the memory of a person we know – more or less.

In relationships lies trust – and whenever we recognize something that’s familiar or well-known to us, we automatically open up to it. It happens every time we make a phone call or receive a voice message on WhatsApp. Once we recognize the voice, we instantly connect the spoken words to that person and – in case of a friend’s or partner’s voice – establish our connection of trust.

But what if that trusty connection could be compromised? What if a voice could be synthesized by a third person in a way that makes it indistinguishable from the original one?

There are some very interesting studies that explore the possibility of “speech synthesis” in the matter of “speaker legitimacy” – the art of determining the authenticity of a voice heard. By the way, that doesn’t only affect us as humans. There are a number of systems that use a voice to recognize a person in order to grant access to sensitive data or controls – think about your digital assistant on your smart phone, for example.

Today, there are several ways to synthesize a voice – purely artificially or based on human input. To give you a quick overview: There is the articulatory approach, where basically the human speech apparatus is mimicked in order to modify a sound signal through different parameters, like the position of the tongue, lips or jaw. This approach is by far the most difficult to achieve due to the vast number of sensor measurements that have to be taken in several iterations of a speaker analysis. To this day, a complete speech synthesis system based solely on this approach doesn’t exist.

Another approach is the signal modelling approach. Where before, the signal was based on the question of “how does a human create it”, this approach raises the question “how the signal actually sounds” – so the acoustic signal itself is being modified here. This is basically done through applying several filters with specific settings in a specific order – the best results can mostly be achieved with a “convolutional neural network” (CNN), but there are many speech signals necessary for training the engine, and it comes with high computational cost.

The by far most successful way to create a realistic-sounding voice is by applying the approach of “concatenation”. Here, fitting segments of a existing, recorded (“real”) voice a taken and put together to create syllables, words and eventually whole sentences. Think about your GPS navigation system – it would probably take forever to record all the street names that exist in your country or region of language. But if you had just the right number of syllables in different pitches, they can be concatenated in a way where every possible combination of street names can be pronounced in a realistic way.

But how can all of this be used to attack me and my phone calls?

This rather shocking example is based on a study by D. Mukhopadhyay, M. Shirvanian, and N. Saxena. They tried to impersonate a voice by a threat model that includes three steps:
First, samples from the voice of a “target victim” are collected. That can be done in numerous ways, either through wiretapping phone calls, recording the victim in it’s surrounding or simply use voice samples shared on social media.
In a second step, an attacker speaks the same utterance of the victim into a voice morphing engine – that way, he receives a model of the voice of the victim. The engine now basically knows “what was said”, and “how did it sound”. That model can now be used by the attacker to speak any utterance, while the morphing engine is able to apply the model built before to make the attacker’s voice sound like the target victim.
Note that the term “voice morphing”: It is a technique where a source voice can be modified to sound like a desired target voice, by applying the respective different spectral features between the two voices. This process makes use of signal modelling and concatenation, that were mentioned before.
The image below illustrates the described threat model:

Source: “All Your Voices Are Belong to Us” by D. Mukhopadhyay et al.

If you want to listen into a short sample of the result of a voice morphing software, watch this little video.

As shown in Phase III of the threat model, the fake utterance of Bob’s voice will be used to attack both a machine-based, as well as a human-based legitimacy detection capability.

The machine-based setup was targeting the “Bob SPEAR Speaker Verification System”, a Python-based open source tool for biometric recognition. Two different speech datasets (Voxforge – short 5 second samples in high quality, and MOBIO – longer samples of 7-30 seconds, recorded with basic laptop microphone) were used to train the engine, which was in this case the “Festvox” conversion system.
The results of this attack system were startling:

Source: “All Your Voices Are Belong to Us” by D. Mukhopadhyay et al.

This data shows how the system responded to the original voices as well as the faked one’s. To clarify the overall accuracy of the system, for each dataset a “different speaker attack” as well as a “conversion attack” as made – the different speaker attack means that the voice used to authenticate itself was a completely different one on purpose. The conversion attack however is the attacker’s voice morphed into the original speaker’s one.
The “False Acceptance Rate” (FAR) shows that in both conversion attack scenarios the system granted access to more than 50% of the voices played back – enough to say that the system fails significantly against a voice conversion attack. It also shows that there is indeed a difference in the results based on the quality of the conversion samples.

Having tested the machine-based speaker verification it is kind of eagerly awaited to see how the human-based verification will perform.
For this setup, online workers from Amazon Mechanical Turk (M-Turk, a crowdsourcing marketplace) were recruited to give their voices to build a model for the attack. The setup consisted of two parts: A “famous speaker study”, and a “briefly familiar speaker study”. The former aimed for an attacker to mimic the voice of a popular celebrity – one that many participants knew and would be able to recognize more easily. For that scenario, the voices of Morgan Freeman and Oprah Winfrey were used by collecting samples from some of their speeches. The latter intended to re-create the situation where somebody received a call from a person he or she met just briefly before – like at a conference. The participants from both studies conducted the tests and were asked, after listening to each of the vocal samples, to state whether the voice they just heard belonged to one of the famous speakers – or in the second case, to one of the briefly familiar speakers. The results from both of these studies are shown below:

Source: “All Your Voices Are Belong to Us” by D. Mukhopadhyay et al.

They show that the participants were a bit more successful in detecting a “different speaker” (an unknown voice), than verifying the original one – but the rate of successfully detecting a conversion attack was around 50%, which is not really a comforting value. The indicator “not sure”, that the participants were able to state shows, that they got confused. If this scenario should happen in real life, it is to be expected that this confusion could highly affect a person’s ability to verify a speaker’s identity.
With the briefly familiar speakers, the success rate of detecting a conversion attack was about 47%, which means that also over 50% of the users could not say for sure if an attack was present.

Let’s recap for a moment – we’ve seen that with modern means of technology it is rather easy and accessible to mimic a voice and trick people into believing that it is actually the real voice of a person familiar to them – with a possible success rate over 50%, depending on the quality of the samples used.

But why can we be tricked so easily? Isn’t there a way to sharpen our subconscious decision-making when it comes to speaker legitimacy detection?

Well, relating to the first question, another study by A. Neupane, N. Saxena, L. Hirshfield, and S. Bratt tried to find a biological relation to the rather poor test results.
In their paper – that describes a brain study based on the same tests from the studies described before – they try to find that relation.
Why a brain study? Previous studies have found differences in neural activation in the human brain in similar areas when users were viewing real and counterfeit items like websites and Rembrandt paintings.
In their study, Neupane and his team tried to confirm that some specific “and other relevant brain areas might be activated differently when users are listening to the original and fake voices of a speaker”.

To investigate this, they conducted the same tests, but monitored the users’ brain activities using a neuroimaging technique called “fNIRS” (Functional Near-Infrared Spectroscopy), by which activities in neural areas of interest can be inferred by examining changes between oxy-Hb and deoxy-Hb.
There are basically only a few neural activation areas of interest for this kind of scenarios. They are listed below:

Source: “The Crux of Voice (In)Security:
A Brain Study of Speaker Legitimacy Detection” by A. Neupane et al.

For brevity’s sake, only the applicable abbreviations are used furtherly.

You can see the three test runs where first the Original Speaker Attack is perceived, the second frame shows the Morphed Voice Attack and the third one the Different Speaker Attack. During the tests, the active regions around DLPFC, FPA and STG (working memory and auditory processing) show that the participants were actively trying to decide if the voice they heard was real or fake.

Following their hypothesis, the team tried to prove that there should be a difference in the Orbitofrontal Area (OFA), where the decision making and trust processes take place, especially when comparing the original speaker vs. the morphed voice.
But surprisingly, there were no such statistically significant differences! That suggests that the morphed voices may have sounded identical enough to the original voices to remain untroubled by skepticism on the part of the human brain. Further, a higher activation in FPA and MTG were observed when the participants were listening to the voice of a familiar speaker, compared to an unfamiliar speaker. This illustrates that the human brain processes familiar voices differently from the unfamiliar ones.

To sum up, here’s what we learned from all of that:

  • Human voice authenticity can easily be breached
  • People seem to detect attacks against familiar celebrities voices better than briefly familiar voices, but still an uncertainty of about 50% remains
  • The brain study surprisingly shows that even though users put considerable effort in making real vs. fake decisions, no significant difference is found in neural areas of interest with original vs. morphed voices

Still wonder what that means for you?

Well, first, we should all be aware of the fact that a vocal impersonation of individuals is indeed possible, even with reasonable effort. That could target politicians as well as family members, friends or employees of your bank. Voice phishing via phone becomes a real threat, especially when an attacker is able to perform an attack where his or her voice can be morphed “on the fly” (without prior rendering or preparation of spoken statements).

It is also important to mention that the studies described were conducted with young and healthy participants. Imagining older people or people with hearing disabilities becoming victims of such attacks, the might perform even worse against those than the participants of the studies.
Finally, voice morphing technologies will probably advance faster in time than our brains evolve – our very own “biological weakness” remains.

Now, isn’t there anything we can do about that?

Probably the most important thing about all of these findings is to become aware of the possibilities of such attacks. It helps not to rely only on information given to you via phone, especially when it comes to handling sensitive information or data.
With social media becoming a growing part of your lives, we should nevertheless be wary about posting our audio-visual life online, especially not in a public manner, where samples of our voices become available to everyone.

A tip against voice phishing is to never call back to provided phone numbers. If the caller claims to be from your bank – look up the phone number online, it might be a much safer option.

Conclusively, voice is not the only way of biological identification that contains flaws – even though in our own perception it is kind of unique. Regardless, it should never be used solely to ascertain a person’s identity.
But even with security through strongly encrypted private keys, at some point in human interaction the link between machine and human needs to happen – and it is where we will continue find weak spots.

References

  • “All Your Voices Are Belong to Us: Stealing Voices to Fool Humans and Machines” by D. Mukhopadhyay, M. Shirvanian, N. Saxena
  • “The Crux of Voice (In)Security: A Brain Study of Speaker Legitimacy Detection” by A. Neupane, N. Saxena, L. Hirshfield, S. E. Bratt
  • “Sprachverarbeitung: Grundlagen und Methoden der Sprachsynthese und Spracherkennung” by B. Pfister and T. Kaufmann
  • https://krebsonsecurity.com/2018/10/voice-phishing-scams-are-getting-more-clever/
  • http://www.koreaherald.com/view.php?ud=20190317000115

Social Engineering – Learn From the Best!

Kevin David Mitnick, Social Engineering, Hacker, Manipulation

It isn’t always necessary to attack by technical means to collect information or to penetrate a system. In many cases, it’s more effective to exploit the human risk factor. To successfully protect yourself and your company from social engineering, you’ve to understand how a social engineer works. And the best way to do this is by listening to the world’s most wanted hacker Kevin David Mitnick. Nowadays, the former social engineering hacker uses his expert knowledge to advise companies on how to protect themselves against such attacks. This blog entry is based on his bestseller “The Art of Deception: Controlling the Human Element of Security”. It sheds light on the various techniques of social engineering and enumerates several ways in which you can arm yourself against them.

Continue reading

Convenient internet voting using blockchain technology

src: https://www.extremetech.com/wp-content/uploads/2018/10/511382-how-to-register-to-vote-online.png

Within this century the use of digital technology has probably never been as high and as convenient as of today. People use the internet to access encyclopedias, look up food recipes and share pictures of their pets. It doesn’t matter whether you are at home, standing in an aisle at the grocery store or even flying on an airplane. Our devices provide unlimited access to modern technology and even somewhat changed the way we used to do things. For instance, it is now a matter of minutes, sometimes even seconds for us to buy some products online or quickly check our balance on banking accounts, whereas those things used to require you to at least leave the house for some time. In some cases, we even narrowed down our involvement for buying products to simply pushing down a button. In comparison to the older day methods for those actions this seems like a huge improvement. And it is. But maybe not in all regards.

Continue reading

How does Tor work?

Written by Tim Tenckhoff – tt031 | Computer Science and Media

1. Introduction

The mysterious dark part of the internet – hidden in depths of the world wide web, is well known as a lawless space for shady online drug deals or other criminal activities. But in times of continuous tracking on the Internet, personalized advertising or digital censorship by governments, the (almost) invisible part of the web promises to bring back lost anonymity and privacy as well. This blogpost aims to shed light into the dark corners of the deep web and primarily deals with the explanation of how TOR works.

Reference: Giphy, If Google was a person: Deep Web
  1. Introduction
  2. The Deep Web
    1. 2. 1 What is the Tor Browser?
  3. The Tor-Network
    1. 3.1 Content
    2. 3.2 Accessing the Network
    3. 3.3 Onion Routing – How Does Tor Work?
  4. Conclusion – Weaknesses
  5. References

2. The Deep Web

So, what exactly is the deep web? To explain this, it makes sense to cast a glance at the overall picture. The internet as most people know it, forms only a minimal proportion of the overall 7.9 Zettabyte (1 ZB = 10007 bytes = 1021 bytes = 1000000000000000000000 bytes 
= 1 trillion Gigabytes?) of data available online (Hidden Internet 2018). This huge amount of data can be separated into three parts:

Separation of the worldwide web, Reference: (Search Engines 2019)

As seen in the picture above, we are accessing only 4% available on search engines like Google or Bing. The remaining 96% (90% + 4%) are protected by passwords, hidden behind paywalls or can be accessed via special tools (Hidden Internet 2018). But what separates the hidden parts into Deep Web and Dark Web by definition?

The Deep Web is fundamentally referred to data which are not indexed by any standard search engines as e.g. Google or Yahoo. This includes all web pages that search engines cannot find, such as user databases, registration-required web forums, webmail pages, and pages behind paywalls. Thus, the Deep Web can, of course, contain content that is totally legal (e.g. governmental records). The Dark Web is a small unit of the Deep Web – which refers to web pages that cannot be found by common search engines. The collection of websites that belongs to this dark web​ only exists on an encrypted network that cannot be reached by regular browsers (such as Chrome, Firefox, Internet Explorer, etc.). In conclusion, this area is the well-suited scene of cybercrime. Accessing these Dark Websites requires the usage of the Tor Browser.

…hidden crime bazaars that can only be accessed through special software that obscures one’s true location online.

– Brian Krebs, Reference: (Krebs On Security 2016)

2. 1 What is the Tor Browser?

The pre-alpha version of the Tor Browser was released on September 2002 (Onion Pre Alpha 2002 and the Tor Project, the company maintaining Tor, was started in 2006. The name Tor consists of three subterms and is the abbreviation of The onion router. The underlying Onion Routing Protocol was initially developed by the US Navy in the mid-1990s at the U.S Naval Research Laboratory (Anonymous Connections 1990). The protocol basically describes a technique for anonymous communication over a public network: By encapsulating each message carried in several layers of encryption and redirecting Internet traffic through a free, worldwide overlay network. It is called onion routing because of the layers in this network and the layers of an onion. Developed as free and open-source software for enabling anonymous communication, the Tor-Browser still follows the intended use today: protecting ​personal privacy and communication by protecting internet activities from being monitored.

With the Tor Browser, barely anyone can get access to The Onion Router (Tor) network by downloading and running the software. The browser does not need to be installed in the system and can be unpacked and transported as portable software via USB stick (Tor Browser 2019). As soon as this is done, the browser is able to connect to the Tor network. This is a network of many servers, the Tor nodes. While surfing, the traffic is encrypted by each of these Tor nodes. Only at the last server in the chain of nodes, the so-called​ exit node, the data stream is decrypted again and normally routed via the Internet to the target server, which is located in the address bar of the Tor browser. In concrete terms, the Tor browser first downloads a list of all available Tor servers for the connection over the Tor network and then defines a random route from server to server for data traffic, which is called Onion Routing as said before. These routes consist of a total of three Tor nodes, with the last server being the Tor exit node (Tor Browser 2019).

Conncetion of a Web-Client to Server via Tor Nodes, Reference: (Hidden Internet 2018)

For the reason that traffic to the Onion service runs across multiple servers from the Tor Project, the traces that users usually leave while surfing with a normal Internet browser or exchanging data such as email and messenger messages become blurred. Even though the payload of normal Internet traffic is encrypted, e.g. via https, the header containing routing source, destination, size, timing etc. can simply​ be spied by attackers or Internet providers. Onion routing in contrast​ also obscures the IP address of Tor users and keeps their computer location anonymous. To continuously disguise the data route, a new route through the Tor network is chosen every ten (Tor Browser 2019) minutes. The exact functionality of the underlying encryption will be described later in section Onion Routing – How Does Tor Work?.

3. The Tor-Network

For those concerned about the privacy of their digital communications in times of large-scale surveillance, the Tor network provides the optimal obfuscation. The following section explains which content can be found on websites hidden in the dark web, how the multi-layered encryption works in detail, and what kind of anonymity it actually offers.

3.1 Content

Reference: Giphy: SILK ROAD GIF BY ANTHONY ANTONELLIS

Most of the content in relation to the darknet involves nefarious or illegal activity. With the provided possibility of anonymity, there are many criminals trying to take advantage of it. This results in a large volume of darknet sites revolving around drugs, darknet markets (sites for the purchase and sale of services and goods), and fraud. Some examples found within minutes using the Tor browser are listed in the following:

  • Drug or other illegal substance dealers: Darknet markets (black markets) allow the anonymous purchase and sale of medicines and other illegal or controlled substances such as pharmaceuticals. Almost everything can be found here, quite simply in exchange for bitcoins.
  • Hackers: Individuals or groups, looking for ways to bypass and exploit security measures for their personal benefit or out of anger for a company or action (Krebs On Security 2016), communicate and collaborate with other hackers in forums, share security attacks (use a bug or vulnerability to gain access to software, hardware, data, etc.) and brag about attacks. Some hackers offer their individual service in exchange for bitcoins.
  • Terrorist organizations use the network for anonymous Internet access, recruitment, information exchange and organisation (What is the darknet?).
  • Counterfeiters: Offer document forgeries and currency imitations via the darknet.
  • Merchants of stolen information offer e.g. credit card numbers and other personally identifiable information can be accessed and ordered for theft and fraud activities.
  • Weapon dealers: Some dark markets allow the anonymous, illegal purchase and sale of weapons.
  • Gamblers play or connect in the darknet to bypass their local gambling laws.
  • Murderers/assassins: Despite of existing discussions about whether these services are real or legitimate, created by the law enforcement or just fictitious websites, some dark websites exist, that offer murder for rent.
  • Providers of illegal explicit material e.g. child pornography: We will not go into detail here.
Screenshot of the infamous Silk Road (platform for selling illegal drugs, shutdown by the FBI in October 2013) , Reference: (Meet Darknet 2013)

But the same anonymity also offers a bright side: freedom of expression. It offers the availability to speak freely without fear about persecution in countries where this is no fundamental right. According to the Tor project, hidden services allowed regime dissidents in Lebanon, Mauritania and the Arab Spring to host blogs in countries where the exchange of those ideas would be punished (Meet Darknet 2013). Some other use-cases are:

  • To use it as a censorship circumvention tool, to reach otherwise blocked content (in countries without free access to information)
  • Socially sensitive communication: Chat rooms and web forums where rape and abuse survivors or people with illnesses can communicate freely, without being afraid of being judged.

A further example of​ that is the New Yorker’s Strongbox, which allows whistleblowers to upload documents and offers a way to communicate anonymously with the magazine (Meet Darknet 2013).

3.2 Accessing the Network

The hidden sites of the dark web can be accessed via special onion-domains. These addresses are not part of the normal DNS, but can be interpreted by the Tor browser if they are sent into the network through a proxy (Interaction with Tor 2018). In order to create an onion-domain, a Tor daemon first creates an RSA key pair, calculates the SHA-1 hash over the generated public RSA key, shortens it to 80 bits, and encodes the result into a 16-digit base32 string (e.g. expyuzz4waqyqbqhcn) (Interaction with Tor 2018). For the reason that onion-domains directly derive from their public key, they are self-certifying. That implements, that if a user knows a domain, he automatically knows the corresponding public key. Unfortunately, onion-domains are therefore difficult to read, write, or to remember. In February 2018, the Tor Project introduced the next generation of onion-domains, which can now be 56 characters long, use a base32 encoding of the public key, and includes a checksum and version number (Interaction with Tor 2018). The new onion services also use elliptic curve cryptography so that the entire public key can now be embedded in the domain, while it could only be the hash in previous versions. These changes led to enhanced security of onion-services, but long and unreadable domain names interfered the usability again (Interaction with Tor 2018). Therefore, it is a common procedure, to repeatedly generate RSA keys until the domain randomly contains the desired string (e.g. facebook). These vanity onion domains look like this for e.g. Facebook (facebookcorewwwi.onion) or the New York Times (nytimes3xbfgragh.onion) (Interaction with Tor 2018). In contrast to the rest of the Worldwide Web, where navigation is primarily done via search engines, the darknet often contains pages with lists of these domains for further navigation. The darknet deliberately tries to hide from the eyes of the searchable web (Meet Darknet 2013)

3.3 Onion Routing – How Does Tor Work?

So how exactly does the anonymizing encryption technology behind Onion Routing work? As said before, the Tor browser chooses an encrypted path through the network and builds a circuit in which each onion router only knows (is able to decrypt) its predecessor and the successor, but no other nodes in the circuit. Tor thereby uses the Diffie-Hellman algorithm to generate keys between the user and different onion routers in the network (How does Tor work 2018). The algortihm is one possible application of Public Key Cryptography that makes use of two large prime numbers which are mathematically linked:

  1. A public-key — public and visible to others
  2. A private-key — private and kept secret

The public key can be used to encrypt messages and the private key is in return used to decrypt the encrypted content. This implicates, that anyone is able to encrypt content for a specific recipient, but this recipient alone can decrypt it again (How does Tor work 2018).

Tor normally uses 3 nodes by default, so 3 layers of encryption are required to encrypt a message (How does Tor work 2018). It is important to say, that every single Tor packet (called cell) is exactly 512kb large. This is done for the reason, that attackers cannot guess which cells are larger cells e.g images/media (How does Tor work 2018). On every step, the transferred message/package reaches, one layer of encryption is decrypted, revealing the position of the next successor in the circuit. This makes it possible, that nodes in the circuit do not know where the previous message originated or where its final destination is (How does Tor work 2018). A simplified visualization of this procedure can be seen in the picture below.

Removing one layer of encryption in every step to the next node, Reference (How does Tor work 2018)

But how does the network allow different users to connect without knowing each other’s network identity? The answer are so-called “rendezvous points”, formerly known as hidden services. (Onion Service Protocol 2019). The following steps are mainly extracted and summarized from the official documentation of Tor about the Onion Service Protocol 2019 and describe the technical details of how this is made possible:

Step 1: Before a client is able to contact an onion service in the network, it needs to broadcast its existence. Therefore, the service randomly selects relays in the network and requests them to act as introduction points by sending its public key. The picture below shows these circuit connections in the first step as green lines. It is important to mention, that these lines mark Tor circuits and not direct connections. The full three-step circuit makes it hard to associate an introduction point with the IP address of an onion server: Even though the introduction point is aware of the onion servers identity (public key) it does never know the onion server’s location (IP address)(Onion Service Protocol 2019).

Step 1: Reference: (Onion Service Protocol 2019)

Step 2: Step two: The service creates a so-called onion service descriptor that contains its public key and a summary of each introductory point (Onion Service Protocol 2019). This descriptor is signed with the private key of the service and then uploaded to a distributed hash database table in the network. If a client requests an onion domain as described in section Accessing the Network the respective descriptor is found. If e.g. “abc.onion” is requested, “abc” is a 16 or 32 character string derived by the service’s public key as seen in the picture below.

Step 2: Reference: (Onion Service Protocol 2019)

Step 3: When a client contacts an onion-service it needs to initiate the connection by downloading the descriptor from the distributed hash table as described before. If that certain descriptor exists for the address abc.onion, the client receives the set of introduction points and the respective public key. This action can be seen in the picture below. At the same time, the client establishes a connection circuit to another randomly selected node in the network and​ asks it to act as a rendezvous point by submitting a one time-secret key (Onion Service Protocol 2019).

Step 3: Reference: (Onion Service Protocol 2019)

Step 4: Now the client creates a so-called introduce message (encrypted with the public key of the onion service), containing the address of the rendezvous point and the one-time secret key. This message is sent to one of the introduction points, requesting the onion service as its final target. For the reason that the communication is again realized by a gate circuit, it is not possible to uncover the clients IP address and thus its identity.

Step 4: Reference: (Onion Service Protocol 2019)

Step 5: At this point, the onion service decodes the introduce message including the address of the rendezvous point and the one-time secret key. The service is then able to establish a circuit connection to the now revealed rendezvous point and communicates the one-time secret in a rendezvous message to the node. Thereby, the service remains with the same set of entry guards for the creation of new circuits (Onion Service Protocol 2019). By application of this technique, an attacker is not able to create his own relay to force the onion service to create an optional number of circuits, so that the corrupt relay might be randomly selected as the entry node. This attack scenario which is able to uncover the anonymity in the Deep Web networks was described by Øverlier and Syverson in their paper (Locating Hidden Servers 2006).

Step 5: Reference: (Onion Service Protocol 2019)

Step 6: As seen in the last picture below, the rendezvous point informs the client about the successfully established connection. Afterwards, both the client and onion service are able to use their circuits to the rendezvous point to communicate. The (end-to-end encrypted) messages are forwarded through the rendezvous point from client to the service or vice versa (Onion Service Protocol 2019). The initial introduction circuit is never used for the actual communication for one important reason mainly: A relay should not be attributable to a particular onion service. The rendezvous point is therefore never aware of the identity of any onion service (Onion Service Protocol 2019). Altogether, the complete connection between service and onion service and client consists of six nodes: three selected by the client, whereas the third is the rendezvous point and the other three are selected by the service.

Step 6: Reference: (Onion Service Protocol 2019)

4. Conclusion – Weaknesses

Different from what many people believe (How does Tor work 2018) Tor is no completely decentralized peer-to-peer system. If it was, it wouldn’t be very useful, as the system requires a number of directory servers that continuously manage and maintain the state of the network.

Furthermore, Tor is not secured against end-to-end attacks. While it does provide protection against traffic analysis, it cannot and does not attempt to protect against monitoring of traffic at the boundaries of the Tor network (the traffic entering and exiting the network), which is a problem that cyber security experts were unable to solve yet (How does Tor work 2018). Researchers from the University of Michigan even developed a network scanner allowing identification of 86% of worldwide live Tor “bridges” with a single scan (Zmap Scan 2013). Another disadvantage of Tor is its speed – because the data packages are randomly sent through a number of nodes, and each of them could be anywhere in the world, the usage of Tor is very slow. Despite its weaknesses, the Tor browser is an effective, powerful tool for the protection of the user’s​ privacy online, but it is good to keep in mind that a Virtual Private Network (VPN) can also provide security and anonymity, without the significant speed decrease of the Tor browser (Tor or VPN 2019) . If total obfuscation and anonymity regardless of the performance play a decisive role, a combination of both is recommended.

5. References

Hidden Internet [2018], Manu Mathur, Exploring the Hidden Internet – The Deep Web [Online]
Available at: https://whereispillmythoughts.com/exploring-hidden-internet-deep-web/
[Accessed 27 August 2019].

Search Engines [2019], Julia Sowells, Top 10 Deep Web Search Engines of 2017 [Online]
Available at: https://hackercombat.com/the-best-10-deep-web-search-engines-of-2017/
[Accessed 24 July 2019].

Krebs On Security [2016], Brian Krebs, Krebs on Security: Rise of Darknet Stokes Fear of The Insider [Online]
Available at: https://krebsonsecurity.com/2016/06/rise-of-darknet-stokes-fear-of-the-insider/
[Accessed 14 August 2019].

Anonymous Connections [1990], Michchael G.Reed, Paul F. Syversion, and David M. Goldschlag Naval Research Laboratory Anonymous Connections and Online Routing [Online]
Available at: https://www.onion-router.net/Publications/JSAC-1998.pdf
[Accessed 18 August 2019].

Onion Pre Alpha [2002], Roger Dingledine, pre-alpha: run an onion proxy now! [Online]
Available at: https://archives.seul.org/or/dev/Sep-2002/msg00019.html
[Accessed 18 August 2019].

Tor Browser [2019], Heise Download, Tor Browser 8.5.4 [Online]
Available at: https://www.heise.de/download/product/tor-browser-40042
[Accessed 29 August 2019].

Interaction with Tor [2018], Philipp Winter, Anne Edmundson, Laura M. Roberts, Agnieszka Dutkowska-Zuk, Marshini Chetty, Nick Feamster, How Do Tor Users Interact With Onion Services? [Online]
Available at: https://arxiv.org/pdf/1806.11278.pdf
[Accessed 16. August 2019].

What is the darknet?, Darkowl, What is THE DARKNET? [Online]
Available at: https://www.darkowl.com/what-is-the-darknet/
[Accessed 22. August 2019].

Meet Darknet [2013], PCWorld: Brad Chacos ,Meet Darknet, the hidden, anonymous underbelly of the searchable Web [Online]
Available at: https://www.pcworld.com/article/2046227/meet-darknet-the-hidden-anonymous-underbelly-of-the-searchable-web.html
[Accessed 23. August 2019].

Onion Service Protocol [2019], Tor Documentation, Tor: Onion Service Protocol [Online]
Available at: https://2019.www.torproject.org/docs/onion-services
[Accessed 8. July 2019].

How does Tor work [2018], Brandon Skerritt, How does Tor *really* work? [Online]
Available at: https://hackernoon.com/how-does-tor-really-work-c3242844e11f
[Accessed 8. July 2019].

Locating Hidden Servers [2006], Lasse Øverlier, Paul Syverson, Locating Hidden Servers [Online]
Available at: https://www.onion-router.net/Publications/locating-hidden-servers.pdf
[Accessed 8. August 2019].

Zmap Scan [2013], Peter Judge, Zmap’s Fast Internet Scan Tool Could Spread Zero Days In Minutes [Online]
Available at: https://www.silicon.co.uk/workspace/zmap-internet-scan-zero-day-125374
[Accessed 21. August 2019].

Tor or VPN [2019], Bill Man, Tor or VPN – Which is Best for Security, Privacy & Anonymity? [Online]
Available at: https://blokt.com/guides/tor-vs-vpn
[Accessed 8. August 2019].

Progressive Web Apps – Wer braucht noch eine native App?



Beispiele zum Einstieg

Progressive Web Apps sind schon weiter verbreitet wie man denkt. Auch große, innovative Unternehmen Twitter, Airbnb, Spotify oder Tinder setzen auf Progressive Web Apps.

Abb. 1: Eine Auswahl von Progressive Web Apps [1]


Wer sich ein tolles Beispiel anschauen möchte, dem empfehle ich https://riorun.theguardian.com/ (auf mobile) zu testen. Nach einiger Zeit erscheint ein Popup, das fragt, ob man die App zum Startbildschirm hinzufügen möchte. Bestätigt man diese Abfrage, wird die PWA im Hintergrund auf dem Gerät installiert und ist ab sofort wie jede andere App auf dem Gerät verfügbar.

Abb. 2: Rio Run App von the Guardian [2]

Weitere tolle PWAs finden sich auf den Übersichtsseiten von:
https://pwa.rocks/
https://progressivewebapproom.com/

Historische Entwicklung

Kurz nach der Einführung des ersten iPhones hatte Steve Jobs schon die nächste Vision für sein Schmuckstück. Für ihn war nicht das Gerät an sich, viel mehr der Browser, der Hafen zum Tor der neuen Welt. Seiner Vision nach, sollte ein Smartphone bestenfalls nur noch einen Browser beinhalten. Entwickler können Apps über das Web bereitstellen, die sich vollständig in Safari integrieren lassen und sich selbstständig im Hintergrund updaten. [3]. Die Idee ist gut, allerdings, typisch Jobs, ihrer Zeit voraus. Die Erfolgsgeschichte des iPhones zwang Apple an den Rand der Kapazitäten, so dass vermutet werden kann, dass Apple schlichtweg keine Zeit und Ressourcen hatte das Thema PWA weiter zu verfolgen. Mit der Masse an Apps, die in den Store drang, hatte man mehr als genug zu tun. Die ersten, die nach Steve Jobs das Thema wieder aufgegriffen haben, waren Mozilla, mit der Veröffentlichung des FirefoxOS im Jahr 2013. Ein Betriebssystem, das es ermöglicht Web Apps als native Apps auf dem Endgerät laufen zu lassen. [3] In den Jahren 2006-2013 hat sich auch sonst viel getan: Endgeräte haben unfassbar an Performance gewonnen, die Webentwicklung hat JQuery und PHP hinter sich gelassen. Neue Möglichkeiten, neue Frameworks und ausgefuchste CSS-Kniffe ermöglichen seitdem eine völlig neue Form der User Experience im Web. Und Web? An wen denken wir da im Web? Genau, Google! Und Google hat ein Problem: Auch wenn sie selbst einen App Store betreiben und sich Kniffe haben einfallen lassen, wie sich Apps indexieren lassen, ist eine native App nicht in der Form auswertbar für ihre Suchmaschine, wie der Content einer Website. So fordert Alex Russells, Chrome Engineer bei Google, im Jahr 2015, die Einführung von Progressive Web Apps (PWA), eine Kombination aus aktueller Web Technologie mit den modernsten Möglichkeiten des Browsers [3]. Mit geschicktem Marketing und einer großen User-base überzeugte Google nun auch Apple wieder an dem Thema zu arbeiten, die 2017 mit der Unterstützung vom PWAs in Safari gleichzogen, komplettiert von Microsoft, dem schlafenden Giganten, der 2018 vollen Support für PWAs im Edge Browser verkündete [4]. Somit ist das große Trio vollends am Start und einer Erfolgsgeschichte von PWAs steht nichts mehr im Wege!

Was bedeutet Progressive Web App?

Vom Naming her leitet sich Progressive Web App von Web Apps mit Progressive Enhancement ab. Aber wo liegt da die Innovation? Bei der Entwicklung einer Web App denkt man heute an große Frameworks wie Angular, React, Vue und neueste Browser APIs wie die Geolocation API und viele mehr. Das ist auch bei einer PWA alles einsetzbar, denn Achtung: Eine PWA unterliegt keinem Framework und schließt auch kein Framework aus. Die Limitierung besteht lediglich durch die Gegebenheiten des Browsers.

Progressive Enhancement, zu deutsch progressive Verbesserung, beschreibt den Content-First Ansatz. Ziel dieser Optimierung ist es, dass der First-Meaningful-Paint beim Aufbau einer Seite möglichst früh geschieht. Sprich Content zuerst und weitere Layer wie Skripte, Style und Multimedia Files, werden nach und nach, je nach Netzwerkverbindung geladen und aufgebaut. Das ist aber keine Neuheit, die im Zuge von PWAs entstanden ist, sondern ein Ergebnis der letzten Jahre der Webentwicklung, Suchmaschinen- und User Experience-Optimierung.

Zusammengefasst lässt sich sagen, der Begriff Progressive Web App weist nicht auf die wahre Innovation hin. Web Apps, moderne Browser APIs und progressiver Aufbau wurden nicht mit PWAs erfunden. Frances Berriman, einer der Mitbegründer bei Google geht sogar so weit zu sagen: “The name isn’t for you… The name is for your boss, for your investor, for your marketeer.” [5] 

Worin liegt die Innovation?

Um auf die wahre Innovation hinter PWAs zu kommen, müssen wir zunächst die aktuellen Probleme in der Bereitstellung von Web Apps und nativen Apps betrachten. Web Apps sind eine tolle Sache, aber auf der geschäftlichen Seite fragt man sich, wie man den Benutzer an den Service binden kann. Soll man den Benutzer dazu auffordern, Lesezeichen im Browser zu machen, Social Media Aktivitäten zu verfolgen oder soll man eine weitere App entwickeln, die im Store ausgeliefert wird? Das alles erfordert zusätzliche Motivation beim Benutzer, den Content oder Service zu konsumieren und, auch wenn kein Medienbruch erfolgt, liegt an jeder Stelle der Customer Journey eine kleine Abbruchrate (Bounce Rate) vor. Es gibt auch Hinweise die bestätigen, dass die Downloadzahlen aus den App Stores immer kleiner werden, wobei der Traffic im Internet weiter zunimmt [6]. Dort ist der User und dort will er auch abgeholt werden. Was ist also naheliegender, als den Content, den man sowieso schon über das Web bereitstellt, für den Kunden dauerhaft zur Verfügung zu stellen? Ohne überflüssige Zwischenschritte in der Customer Journey? Das lässt sich zwar mit Web Apps realisieren, aber wenn der Benutzer keine Internetverbindung hat, klingelt auch bei den Entwicklern nichts in der Kasse.
Auch native Apps bringen ihre Probleme auf der geschäftlichen Seite mit. Hohe Entwicklungskosten und straffe Anforderungen der Store Betreiber, sorgen regelmäßig für Frust und lange Nächte beim Release des nächsten Updates. Payment-Optionen werden vordiktiert, undurchsichtige Indexierungen der Store Einträge erlauben keine freien Produktplatzierungen und Marketing. Wie soll man da aus der Masse an Apps herausstechen und sein eigenes Produkt sinnvoll bewerben? Von den Gebühren die Apple und Google beim App-Kauf abkassieren, mal ganz abgesehen. Neben der undurchsichtigen Indexierung ist auch die Auffindbarkeit außerhalb der App Stores problematisch. Zwar lassen sich Apps inzwischen in Suchmaschinen auffinden, werden aber vom Content her nicht so von Web Crawler erfasst, wie herkömmliche Websites. Keine direkte Indexierung bedeutet eine lange Customer Journey bis ein Kunde das Produkt entdeckt und das bedeutet hohe Kosten. 

Diesen Punkten versucht eine PWA entgegenzuwirken. Eine PWA kann online und offline, barrierefreie konsumiert werden. Der Zwang einer App-Installation ist nicht erforderlich, der Kunde kann den Content zunächst über das Web auffinden und konsumieren und sich jederzeit impulsiv für eine App-Installation entscheiden. 

Technik & Funktionsweise

Eine PWA funktioniert grundsätzlich wie eine gewöhnliche Web App. Man kann jede Web App als Basis nehmen, erweitert um Manifest, Service Worker und App Icon. Wichtig zu wissen ist, dass eine PWA nur mit https funktioniert und sie sich auch nur installieren lässt, wenn die Verbindung über https gesichert wurde. Dieser Zwang ist ein willkommener Vorteil, der ein bisschen Sicherheit ins World Wide Web bringt. 

Die nachfolgende Übersicht zeigt einen schematischen Aufbau und Zusammenhänge. In den nachfolgenden Abschnitten schauen wir uns die einzelnen Elemente im Detail an.

Abb. 3: Funktionsweise und Aufbau von PWAs [9]


Manifest

Das Manifest liefert die, für die Installation der App, notwendigen Metadaten. Es muss per Link-Element in den Head des HTML-Dokuments eingebunden werden. In Json formatiert liefert es u.a.:

  • App Name
  • App Beschreibung
  • App Icon in versch. Auflösungen für versch. Endgeräte und Browser
  • Informationen bzgl. des Urhebers
  • App-Scope
  • Start Modus

Eine vollständige Liste aller mögliche Attribute findet sich hier:
https://developer.mozilla.org/de/docs/Web/Manifest

App Scope

Die Property “start_url” beschreibt den Scope der App innerhalb der Domain. Über diesen Parameter wird auch definiert welche Seite beim Start der App angezeigt werden soll.

Start Modus

Die Property “display” beschreibt den Start-Modus der App. Die nachfolgende Bilderreihe zeigt die Auswirkungen der einzelnen Optionen. Mit der Standalone-Option kann man den Look einer nativen App perfekt imitieren. Über den Parameter Theme Color lässt sich zusätzlich die Gestaltung der Statusleiste beeinflussen.

Abb. 4: Browser Modes [7]

Service Worker

Der Service Worker ist der Hintergrunddienst einer PWA. Er ist verwandt mit dem Web Worker, läuft in einem eigenen Thread und erlaubt keine direkte DOM-Manipulation, sondern nur die Kommunikation über eine definierte Schnittstellen. 

Er ist auch bei geschlossener Anwendung lauffähig, legt sich schlafen und erwacht bei eintreffenden Informationen! Mit seiner Hilfe wird eine PWA offlinefähig. Seine Aufgabe ist es, alle Requests die aus dem eigenen Scope ins Netzwerk geschickt werden, abzufangen und zu beurteilen, ob er mögliche Anfragen aus dem eigenen Cache beantworten kann oder nicht. Technisch betrachtet fungiert er quasi als Pseudo-Proxy. In welchem Umfang Requests ins Netzwerk geschickt werden oder aus dem eigenen Cache beantwortet werden, liegt im ermessen des Entwicklers. So ist über das Install-Event des Service Workers, dass bei App-Installation getriggert wird, ein vollständiger Download des App-Contents möglich, was die App damit gänzlich offlinefähig machen würde. Über die Wake-Up Funktion werden Push Notifications auf dem Endgerät ermöglicht, womit man der User Experience einer nativen App wieder einen großen Schritt näher kommen kann. 

Weitere Informationen und eine Übersicht von möglichen Events auf die gelauscht werden kann, findet man unter:
https://developer.mozilla.org/de/docs/Web/API/Service_Worker_API/
Using_Service_Workers


Skandal – Sind Service Worker die neuen Cookies?

Das eine App einen Hintergrunddienst benötigt, ist aus der nativen App- Entwicklung betrachtet, nichts ungewöhnliches. Async-Tasks, Services, Background Tasks, etc. sind jedem Android- oder iOS-Developer ein Begriff. In der Entwicklung von gewöhnlichen Websites (keine Web Apps), sind Hintergrunddienste allerdings eher selten von Bedarf. Der Benutzer weiß inzwischen durch Aufklärungsmaßnahmen, dass der Besuch einer Website in seinem Browser Spuren hinterlässt, sei es Cache, Cookies usw. ABER und jetzt kommt der Skandal: Viele Benutzer wissen nicht (!), dass ihr Browser ebenso durch Service Worker belastet wird, die sich wie eine Zecke in den Browser schleichen, jederzeit aus dem Schlaf erwecken lassen und die die Performance des ganzen Gerätes durch Hintergrunddienste beeinträchtigen können! Wenn man den Artikel bis hierher verfolgt hat, könnte man nun meinen, das Service Worker nur für PWAs eingesetzt werden, aber das Web war schon immer dafür bekannt, dass man sämtliche Tricks ausnutzt, die irgendwie möglich sind. Ich kann an dieser Stelle nur jedem Chrome-Benutzer, egal ob auf Mobile oder Desktop empfehlen, mit dem nachfolgenden Link zu überprüfen, wieviele Service Worker sich unwissend in den eigenen Browser geschlichen haben. Eine vergleichbare Schnittstelle zum überprüfen der registrierten Service Worker bieten andere Browser derzeit nicht an, obwohl sie sie unterstützen! Über Service Worker lässt sich beispielsweise das heimliche Crypto Mining auf Kosten von unwissenden Benutzern realisieren.

Check:  chrome://serviceworker-internals

Push Notifications

Eine Push Notification, die über eine installierte PWA gesendet wurde, lässt sich nicht von einer nativen Notification unterscheiden. Auch jede Referenz auf den zugrundeliegenden Browser wird verborgen. Notifications lassen sich über den Service Worker triggern, der wiederum von App oder Server dazu angestoßen wird.

Abb. 5: PWA Push Notifications vs. native Notifications [9]

Splash Screen

Um dem Start einer App eine bessere User Experience zu verpassen, hat man sich entschlossen, dass eine PWA mit einem Splash Screen startet, während im Hintergrund der Content aufgebaut wird. So wird auf geschickte Art und Weise performance suggeriert und langweilt den Benutzer nicht mit einem Whitescreen. Splash Screens lassen sich nicht an- oder ausschalten, aber customizen. Per Default sind App Icon und App Name (Android) gesetzt.

Abb. 6: Splash Screen auf Android (links) und iOS (rechts) [8]

Optimize & Debug

Wer eine PWA programmieren und das beste an Performance herauskitzeln will, der findet in den Developer Tools des Chrome Browsers das beste Hilfsmittel. Über die Audit-Tests startet der Browser auf Knopfdruck eine Reihe an Tests, die eine PWA in Bezug auf Vollständigkeit (PWA-Anforderungen), Performance, Best Practises, Accessibility und Semantik (SEO) hin bewertet. Auf einer Skala von 1-10 wird ein Scoring ermittelt und aufgelistet wo Verbesserungspotential besteht.

Abb. 7: Audits in der Chrome Developer Console [9]

Exkurs – Browser APIs

Um sich bei Funktionalität und User Experience einer nativen App anzunähern, liefern moderne Browser heute schon jede Menge APIs, die man vielleicht gar nicht kennt. Darunter auch einige hardwarenahe Schnittstellen, die häufig noch gar keinen Gebrauch finden, aber die Zukunft mitprägen können. Um sicherzustellen, dass entsprechende APIs auch schon in weit verbreiteten Browsern implementiert sind, bietet sich an, diese vor Implementierung, mit Hilfe des Can I Use – Services zu prüfen.

Eine Auswahl eher unbekannter APIs:

  • Sensors API
    • Ambient Light, Proximity, Accelerometer, Magnetometer, Gyroskop
  • Battery Manager API
  • Web Payments API
  • Gamepad API
  • Geolocation (GPS) API
  • Vibration API
  • Web Speech API (Text-To-Speech, Speech-To-Text, Grammar)
  • Bluetooth API
  • Push Notification API
  • USB Device API
  • WebVR API
  • Indexed Database API
  • File System API


PWA vs. Native

Wir wir bisher erfahren haben, ermöglichen moderne Browser schon mehr Features, wie man vielleicht vermuten mag. Wenn man betrachtet, dass PWAs erst seit 2018 von allen großen Browsern unterstützt werden, kann man schlussfolgern, dass das ganze Thema noch in den Kinderschuhen steckt. Aber bereits jetzt, durch die herausragende Performance-Steigerung mobiler Endgeräte in den letzten Jahren, sind tolle Anwendungen im mobile Web möglich, so dass native Apps aus Kostengründen wohl immer unattraktiver werden, sofern keine native Performance für das Produkt erforderlich ist. Auch der PWA-Standard wird sich in den nächsten Jahren weiterentwickeln und immer mehr Möglichkeiten bieten. Es bahnt sich vielleicht eine spannende Welle der Veränderungen an, die den Mobile-App-Markt durcheinander bringen könnte!

Zum Abschluss eine Übersichtsmatrix die zur Entscheidungsfindung hilfreich sein kann. Die meisten Punkte erhält die Umsetzungsart, die am besten abschneidet.

Web AppPWANative App
Speicherbedarf auf dem Endgerät◼️◼️◼️
Offlinefähigkeit◼️◼️◼️◼️
Auffindbarkeit in Suchmaschinen◼️◼️◼️◼️◼️
Installation, Updates, Wartung, Versionierung◼️◼️◼️◼️◼️
Performance, Ladezeiten◼️◼️◼️
Sicherheit(kann)◼️◼️(kann)
Entwicklungskosten◼️◼️◼️
Natürlicher Traffic◼️◼️◼️◼️◼️
Hardwarenahe Features◼️◼️◼️◼️
User Experience(kann)(kann)◼️◼️


In 10 Schritten zur ersten PWA

Hier noch einen Quickguide zur ersten Progressive Web App:

  1. Web App programmieren
  2. Sicher stellen, dass die Kommunikation zum Server ausschließlich über https läuft
  3. Web Manifest anlegen mit mindestens:
    • Scope
    • Start URL
    • App Icon und App Name
  4. Web Manifest im Head des HTML-Dokuments einbinden
  5. ServiceWorker.js (o.ä.) in root Folder der PWA anlegen
  6. Service Worker im JS der Web App registrieren
  7. ServiceWorker.js je nach Bedarf implementieren
  8. Mit Audits testen ob alle empfohlenen Qualitätskriterien einer PWA erreicht sind
  9. Deployen 🙂



Quellenverzeichnis

[1] Great examples of progressive web apps in one room
https://progressivewebapproom.com/
Zugegriffen am 05.08.2019, 22:08 Uhr

[2] Rio Run App von the Guardian,
https://riorun.theguardian.com/
Zugegriffen am 05.08.2019, 22:08 Uhr

[3] Wikipedia
https://en.wikipedia.org/wiki/Progressive_web_applications
Zugegriffen am 05.08.2019, 22:08 Uhr

[4] Welcoming Progressive Web Apps to Microsoft Edge and Windows 10
https://blogs.windows.com/msedgedev/2018/02/06/welcoming-progressive-web-apps-edge-windows-10/
Zugegriffen am 05.08.2019, 22:08 Uhr

[5] Naming Progressive Web Apps
https://fberriman.com/2017/06/26/naming-progressive-web-apps/
Zugegriffen am 05.08.2019, 22:08 Uhr

[6] Why Build Progressive Web Apps
https://developers.google.com/web/ilt/pwa/why-build-pwa
Zugegriffen am 05.08.2019, 22:08 Uhr

[7]  Progressive Web App Challenges
https://www.slideshare.net/grigs/progressive-web-app-challenges
Zugegriffen am 05.08.2019, 22:08 Uhr

[8] Progressive Web App Splash Screens
https://medium.com/@applification/progressive-web-app-splash-screens-80340b45d210
Zugegriffen am 05.08.2019, 22:08 Uhr

[9] Eigene Leistung

JavaScript Performance optimization with respect to the upcoming WebAssembly standard

Written by Tim Tenckhoff – tt031 | Computer Science and Media

1. Introduction

Speed and performance of the (worldwide) web advanced considerably over the last decades. With the development of sites more heavily reliant on JavaScript (JS Optimization, 2018), the consideration of actions to optimize the speed and performance of web applications grows in importance. This blogpost aims to summarize practical techniques to enhance the performance of JavaScript applications and provides a comparative outlook regarding the upcoming WebAssembly standard.

Let´s jump into hyperspeed…

  1. Introduction
    1. JavaScript 
    2. Why optimize? 
  2. How can the JavaScript Speed and Performance be increased?
    1. Interaction with Host Objects
    2. Dependency Management
    3. Event Binding
    4. Efficient Iterations
    5. Syntax 
  3. Is WebAssembly replacing JavaScript?
  4. References

JavaScript

Back in the days, in 1993 a company called Netscape was founded in America. Netscape aimed to exploit the potential of the burgeoning World Wide Web and created their own web browser Netscape Navigator (Speaking JavaScript 2014). As Netscape realized in 1995, that the web needed to become more dynamic, they decided to develop a scripting language with a syntax similar to Java’s to rule out other existing languages. In May 1995, the prototype of this language was written by the freshly hired software developer Brendan Eich within 10 days. The initial name of the created code was Mocha, which was later changed by the marketing to LiveScript. In December 1995, it was finally renamed to JavaScript to benefit from Java’s popularity (Speaking JavaScript 2014).

Today, every time the functionality of a web page exceeds to just display static content, e.g. by timely content updates, animated graphics, an interactive map or the submission of an input formular, JavaScript is probably involved to solve the given complexity. Thus it is the third layer cake of the standard web technologies, which adds dynamic behavior to an existing markup structure (e.g. defining paragraphs, headings, and data tables in HTML), customized with a definition of style rules (e.g. defining paragraphs, headings, and data tables in CSS). If a web browser loads a webpage, the JavaScript code is executed by the browser’s engine, after the HTML and CSS have been assembled and rendered into a web page (JavaScript 2019). This ensures that the content of the required page is already in place before the JavaScript code starts to run, as seen in Figure 1.

Figure 1: The execution of HTML, CSS and JavaScript in the Browser (JS Optimization, 2018)

The script language often abbreviated as JS includes a curly-bracket syntax, dynamic typing, object orientation, and first-class functions. Initially only implemented in and for web browsers on the client side, today’s JavaScript engines are now embedded in many other types of host software, including web servers (e.g. NodeJS), databases, or other non-web use-cases (e.g. PDF software).

Why optimize?

Speaking about the performance optimization in networks, most developers think about it in terms of the download and execution cost – sending more bytes of JavaScript code takes longer, depending on the users internet connection (JS Optimization 2018). Therefore, it can be said beforehand, that it generally makes sense to reduce the transferred network traffic, especially in areas where the available network connection type of users might not be 4G or Wi-Fi. But one of JavaScript’s heaviest costs regarding the performance is also the parse/compile time. The Google Chrome browser visualizes the time spend in these phases in the performance panel as seen in Figure 2. But why is it even important to optimize this time?

Figure 2: Parse and compile time of JavaScript (JS Optimization 2018)

The answer is, that if more time is spent parsing/compiling, there might be a significant delay in the user’s interaction with the website. The longer it takes to parse and compile the code, the longer it takes until a site becomes interactive. According to a comparison in an article by Google Developer Addy Osmani, JS is more likely to negatively impact a pages interactivity than other equivalently sized resources (JS Optimization 2018). As an example seen in Figure 3, the resource processing time of 170kb JavaScript bytes and the same amount of JPEG bytes require the same amount of 3,4 seconds network transmission time. But as the resource processing time to decode the image (0,064 seconds) and rasterize paint in the image (0,028 seconds) are relatively low, it takes much longer to parse the JavaScript code (~2 seconds) and execute it (~1,5 seconds). Due the fact that website users differ not only regarding their provided network connection, but also in the hardware they have, older hardware may also have a bad influence on increased execution time. This shows how much more JS can potentially delay the interactivity of a website due to parse, compile and execution costs and proves the need for optimization.

Figure 3: The difference between JavaScript and JPEG resource processing (JS Optimization 2018)

2. How can the JavaScript Speed and Performance be increased?

The question appearing at this point is the following: What needs to be practically done during the development of JavaScript to optimize websites or web applications regarding speed and interactivity. The depths of the world wide web expose several blogs and articles about this kind of optimization. The following section aims to sum up found techniques categorized by the underlying performance problem:

Interaction with Host Objects

The interaction with “evil” host objects needs to be avoided as Steven de Salas says in his blog (25 Techniques 2012)

As described before, JavaScript code is compiled by the browser’s scripting engine to machine code – offering an extraordinary increase in performance and speed. However, the interaction with host objects (in the browser) outside this native JavaScript environment raises a loss in performance, especially if these host objects are screen-rendered DOM (Document Object Model) objects (25 Techniques 2012). To prevent this, the interaction with these evil host objects needs to be minimized. Let’s take a look at how this can be done.

Rotation via CSS

A first useful approach is the usage of CSS-classes for DOM animations or interactions. Unlike JavaScript code, solutions like e.g. CSS3 Transitions, @keyframes, :before, :after are highly optimized by the browser (25 Techniques 2012). As shown above, the rotation of the small white square can either be animated by addressing the id via document.getElementById(“rotateThisJS”), triggered by a button starting the rotate() function below…

Figure 4: Rotation via JavaScript (How not to…)

…or by adding a CSS class (as seen in Figure 5) to the <div> element which works much more efficiently, especially if the respective website contains several animations. Except from animations, CSS is also able to handle interactions like e.g. hovering elements, thus it can be said, that the way JavaScript is optimized this time is not to use it in certain cases.

Figure 5: Animation via CSS

Speaking of selectors to pick DOM elements, the usage of jQuery allows a highly specific selection of elements based on tag names, classes and CSS (25 Techniques 2012). But according to the online blog article by Steven de Salas, it is important to be aware that this approach involves the potential of several iterations through underlying DOM elements to find the respective match. He states that this can be improved by picking nodes by ID. An example can be seen in Figure 6:

Figure 6: Different ways of DOM element selection

What also increases the (DOM interaction) performance, is to store references to browser objects during instantiation. If it can be expected, that the respective website is not going to change after instantiation, references to the DOM structure should be stored initially, when the page is created not only when they are needed. It is generally a bad idea to instantiate references DOM objects over and over again. That’s why it is rather advisable to create few references to objects during instantiation which are needed several times (25 Techniques 2012). If no reference to a DOM object has been stored and needs to be instantiated within a function, a local variable containing a reference to the required DOM object can be created. This speeds up the iteration considerably as the local variable is stored in the fastest and most accessible part of the stack (25 Techniques 2012).

The general amount of DOM-elements is also a criteria with respect to the performance. As the time, used for changes in the DOM is proportional to the complexity of the rendered HTML, this should also be considered as an important performance factor (Speeding up 2010).

Another important aspect regarding the DOM interaction is to batch (style) changes. Every DOM change causes the browser to do a re-rendering of the whole UI (25 Techniques 2012). Therefore, it should be avoided to apply each style change separately. The ideal approach to prevent this, is to do changes in one step, for example by adding a CSS class. The different approaches can be seen in Figure 7 below.

Figure 7: How to batch changes in DOM

Additionally it is recommended to build DOM elements separately before adding them to a website. As said before, every DOM requires a re-rendering. If a part of the DOM is built “off-line” the impact of appending it in one go is much smaller (25 Techniques 2012). Another approach is to buffer DOM content in scrollable <div> elements and to remove elements from the DOM that are not displayed on the screen, for example, outside the visible area of a scrollable <div>. These nodes are then reattached if necessary (Speeding up 2010).

Dependency Management

Figure 8: Checking the dependency Management of www.stuttgart.de

Looking at different pages in the www, e.g. the cities’s website of Stuttgart, it can be observed that the screen rendering is delayed for the user until all script dependencies are fully loaded. As seen in Figure 8, some dependencies cause the delayed download of other dependencies that in return have to wait for each other. To solve this problem the active management and reduction of the dependency payload are a core part of performance optimization.

One approach to do so, is to reduce the general dependency on libraries to a minimum (25 Techniques 2012). This can be done by using as much in-browser technology as possible. For example the usage of document.getElementById(‘element-ID’) instead of using (and including) the jQuery library. Before adding a library to the project, it makes sense to evaluate whether all the included features are needed, or if single features can be extracted from the library and added separately. If this is the case, it is of course important to check whether the adopted code is subject to a license – to credit and acknowledge the author is recommended in any case (25 Techniques 2012).

Another important approach is the combination of multiple JavaScript files to bundled ones. The reason behind this is, that one network request with e.g. 10kb of data is transferred much faster than 10 requests with 1kb each (25 Techniques 2012). This difference is caused by lower bandwidth usage and network latency of the single request. To save additional traffic, the combined bundle files can also be minified and compressed afterwards. Minification tools, such as UglifyJS or babel-minify remove comments and whitespacing from the code. Compression tools as e.g. gzip or Brotli are able to compress text-based resources to smaller memory size. (JS Optimization 2018)

A further way to optimize the dependency management, is the usage of a post-load dependency manager for libraries and modules. Tools like Webpack or RequireJS allow, that the layout and frame of a website appears before all of the content is downloaded, by post-loading the required files in the background (JS Optimization 2018). This gives users a few extra seconds to familiarise themselves with the page (25 Techniques 2012).

By maximizing the usage of caching, the browser downloads the needed dependencies only at the first call and otherwise accesses the local copy. This can be done by manually adding eTags to files that need to be cached, and putting *.js files to cache into static URI locations. This communicates the browser to prefer the cached copy of scripts for all pages after the initial one (25 Techniques 2012).

Event Binding

To create interactive and responsive web applications, event binding and handling is an essential part. However, event bindings are hard to track due to their ‘hidden’ execution and can potentially cause performance degradation e.g. if they are fired repeatedly (25 Techniques 2012). Therefore it is important to keep track of the event execution throughout various use cases of the developed code to make sure that events are not fired multiple times or bind unnecessary resources (25 Techniques 2012).

To do so, it is especially important to pay attention to event handlers that fire in quick repetition. Browser events such as e.g. ‘mouse move’ and ‘resize’ are executed up to several hundred times each second. Thus, it is important to ensure that an event handler that reacts to one of these events can complete in less than 2-3 milliseconds (25 Techniques 2012). The box below visualizes the amount of events that are fired when the mouse is moved over an element.

hover me!

Another important point that needs to be taken care of, is the event unbinding. Every time an event handler is added to the code, it makes sense to consider the point when it is no longer needed and to make sure that it stops firing at this point. (Speeding up 2010) This avoids performance slumps through handlers that are bound multiple times, or events firing when they are no longer needed. One good approach to prevent this, is the usage of once-off execution constructs like jQuery.one() or manually adding/coding the unbind behavior at the right place and time (25 Techniques 2012). The example below, shows the usage of jQuery.one – an event binding on each p element that is fired exactly once and unbinds itself afterwards. The implementation of this example can be seen in Figure 9.

Click the boxes to trigger a jQuery.one event!
Fired once. Also fired once. This is also fired only once.
Figure 9: The usage of jQuery.one()

A last important part of the event binding optimization is to consider and understand the concept of event bubbling. A blog article by Alfa Jango describes the underlying difference between .bind(), .live(), and .delegate() events (Event Bubbling 2011).

Figure 10: Propagation of a click event through the DOM (Event Bubbling 2011)

Figure 10 shows what happens if e.g e a link is clicked that fires the click event on the link element, which triggers functions that are bound to that element’s click event: The click event propagates up the tree, to the next parent element and then to each ancestor element that the click event was triggered on one of the descendent elements (Event Bubbling 2011). Knowing this, the difference between the jQuery functions bind(), live() and delegate() can be explained:

.bind()

jQuery scans the entire document for all $(‘a’) and binds the alert function to each of these click events.

.live()

jQuery binds the function to the $(document) tag including the parameters ‘click’ and ‘a’. If the 
event is fired, it checks if both parameters are true, then executes the function.

.delegate()

Similar to .live(), but binds the handler to a specific element, not the document.root.

The article says that .delegate() is better than .live(). But why?

$(document).delegate(‘a’, ‘click’, function() { blah() });

$(‘a’).live(‘click’, function() { blah() });

According to the blog entry (Event Bubbling 2011), delegate() can be preferred for two reasons:

Speed: $(‘a’) first scans for all a elements and saves them as objects, this consumes space and is therefore slower.

Flexibility: live() is linked to the object set of $(‘a’) elements, although it actually acts at the $(document) level..

Efficient Iterations

The next topic is the implementation of efficient iterations. As seen in Figure 11 below, the execution time for string operations grows exponentially during long iterations (String Performance 2008). This shows why iterations can often be the reason for performance flaws. Therefore it always makes sense to get rid of unnecessary loops, or calls inside of loops (25 Techniques 2012) .

Figure 11: Comparative String Performance Analysis (String Performance 2008)

One technique to avoid unnecessary loops, is to use JavaScript indexing. Native JavaScript objects can be used to store quick-lookup indexes to other objects, working in a similar way to how database indexes work (25 Techniques 2012). As seen in Figure 12 below it also speeds up finding objects by using e.g. the name as an index. This is highly optimized and avoids long search iterations.

Figure 12: Finding Objects by JavaScript indexing

Additionally it is always a good idea to use native JavaScript array functions such as push(), pop() and shift(), especially working with arrays. These functions also have a small overhead and are closely connected to their assembly language counterparts (25 Techniques 2012).

The difference between reference and primitive value types, also comes up in terms of efficient iterations. Primitive types such as String, Boolean or Integer are copied if they are handed over to a function. Reference types, such as Arrays, Objects or Dates are handed over as a light-weight reference. This knowledge should be considered if a reference is handed over to a function, running in an iteration: Obviously, it is better to avoiding frequent copying of primitive types and pass lightweight references to these functions.

Syntax

A final option to consider for optimization is the usage of correct JavaScript syntax. Writing simple function patterns without being familiar to advanced native ECMAScript can lead to inefficient code. It is recommendable to try to stay up to date and learn how to apply these constructs.

One easy example (25 Techniques 2012) regarding this, is to prefer the usage of native, optimized constructs over self-written algorithms: Functions as e.g. Math.floor()or new Date().getTime() for timestamps don’t need to be rewritten. The operator === instead of == provides an optimized, faster type-based comparison (25 Techniques 2012). Furthermore, the switch statement can be used instead of long if-then-else blocks to provide an advantage during compilation, to name just a few examples.

3. Is WebAssembly replacing JavaScript?

On the 17th of June 2015, Brendan Eich (the creator of JavaScript) announced a new project that aims to make it easier to compile projects written in languages like C and C++ to run in browsers and other web related JavaScript environments (Why we need Web Assembly). The developing team consists of members from Apple, Google, Microsoft, Mozilla and others collective under the name of the W3C WebAssembly Community Group (Why we need Web Assembly). A blog article by Eric Elliot comments on these release announcements and states that „the future of the web platform looks brighter than ever“ (What is WebAssembly?).

But what exactly is WebAssembly? Elliot further explains that WebAssembly, often shortened as WASM, is a new language format. The code defines an AST (Abstract Syntax Tree) represented in a binary format that can be edited/developed as readable text. The instruction format has been built to compile languages such as C, C++, Java, Python and Rust and allows the deployment on the web and in server applications. Through WebAssembly it is now possible to run the respective code on the web at a native speed (WASM replace JS 2018). But WASM is also an improvement to JavaScript: The performance critical code that needs to be optimized can be implemented in WASM and imported like a standard JavaScript module (What is WebAssembly?). The blog entry by Elliot additionally explains that WASM is also an improvement for browsers. Through the fact, that browsers will be able to understand the binary code that can be compressed to smaller files than currently used Javascript files, smaller payloads would lead to faster delivery and make websites run faster (What is WebAssembly?).

But listing all these advantages, does WebAssembly have the potential to replace JavaScript in the nearest future? And is WebAssembly compilation really so much faster? A blog article from Winston Chen compares the performance of WebAssembly vs JavaScript and comes to a surprising result (WASM vs JS 2018). The performed experiment involved several implementations of matrix multiplications as a simple and computationally intensive way to compare JavaScript and C++ (in WASM). In summary it can be said that JavaScript performed better than WebAssembly on smaller array sizes and WebAssembly outperformed JavaScript on larger ones. Chen concludes, that outgoing from his results, JavaScript is still the best option for most web applications. According to him, Web Assembly was therefore “best used for computationally intense web applications, such as web games” (WASM vs JS 2018).

Among different articles throughout the internet, dealing with the question whether WASM is going to replace JavaScript, it is not possible to find a precise answer or prediction. But Brendan Eich, the creator of JavaScript himself, finds a relatively clear answer to the question if he was trying to KILL JavaScript (by developing WASM): “…We’re not killing JavaScript. I don’t think it’s even possible to kill JavaScript”(Why we need wasm 2015). The JavaScript ecosystem currently supports all major browser and most developers write libraries and frameworks in it (e.g. React, Bootstrap or Angular)(WASM replace JS 2018). In order to overtake JavaScript, any competitor (as WASM) would need to provide replacement options for all these libraries. Furthermore, it is not easily feasible to replace the existing code base of JavaScript based projects. With growing popularity in calculation intense projects as browser-based games, WebAssembly can possibly decrease the market share of JavaScript, but is not able to replace these already existing JS applications. It can be said, that the native speed improvements of WebAssembly are rather a complementation of the existing JavaScript features (WASM replace JS 2018). By using both, (e.g. by using WebAsembly run alongside JS using WASM JavaScript APIs) developers can benefit from the flexibility of JS and the native speed advantages of WASM in combination. Wrapping this up, the creator was right and is very unlikely that WASM is going to overtake JavaScript – still the single, dominating language of the web.

4. References

Speaking JavaScript 2014, Axel Rauschmeyer, Speaking JavaScript: An In-Depth Guide for Programmers
Chapter 4. How JavaScript Was Created
[Accessed 24 July 2019].

WebAssembly 2015, Eric Elliot, What is WebAssembly? [Online]
Available at: https://medium.com/javascript-scene/what-is-webassembly-the-dawn-of-a-new-era-61256ec5a8f6
[Accessed 28 July 2019]. 

Why we need wasm 2015, Eric Elliot, Why we Need WebAssembly [Online]
Available at: https://medium.com/javascript-scene/why-we-need-webassembly-an-interview-with-brendan-eich-7fb2a60b0723
[Accessed 28 July 2019]. 

WASM replace JS 2018, Vaibhav Shah, Will WebAssembly replace JavaScript? [Online]
Available at: https://dev.to/vaibhavshah/will-webassembly-replace-javascript-or-will-wasm-make-javascript-more-valuable-in-future-5c6e
[Accessed 28 July 2019]. 

WASM vs JS 2018, Chen Winston, Performance Testing Web Assembly vs JavaScript [Online]
Available at: https://medium.com/samsung-internet-dev/performance-testing-web-assembly-vs-javascript-e07506fd5875
[Accessed 27 July 2019]. 

25 Techniques 2012, Steven de Salas, 25 Techniques for Javascript Performance Optimization [Online]
Available at: https://desalasworks.com/article/javascript-performance-techniques/
[Accessed 27 July 2019].

JS Optimization 2018, Addy Osmani, JavaScript Start-up Optimization [Online]
Available at: https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/javascript-startup-optimization/
[Accessed 27 July 2019].

JavaScript 2019, Chris David Mills (MDN web docs), What is JavaScript? [Online]
Available at: https://developer.mozilla.org/en-US/docs/Learn/JavaScript/First_steps/what_is_javascript/
[Accessed 27 July 2019].

Speeding up 2010, Yahoo Developer Network, Best Practices for Speeding Up Your Web Site [Online]
Available at: https://developer.yahoo.com/performance/rules.html#min_dom/
[Accessed 27 July 2019].

String Performance 2008, Tom Trenka, String Performance: an Analysis [Online]
Available at: https://www.sitepen.com/blog/string-performance-an-analysis/
[Accessed 27 July 2019].

Event Bubbling 2011 Alfa Jango, THE DIFFERENCE BETWEEN JQUERY’S .BIND(), .LIVE(), AND .DELEGATE() [Online]
Available at: https://www.alfajango.com/blog/the-difference-between-jquerys-bind-live-and-delegate/
[Accessed 24 July 2019].


Mobile Security – How secure are our daily used devices?

Nowadays, the usage of mobile devices has become a part of our everyday life. A lot of sensitive and personal data is stored on these devices, which makes them more attractive targets for attackers. Also, many companies offer the possibility to work remotely, which results in storing confidential business information on private phones and therefore increases the organizations’ vulnerability. The following content shows what kind of attacks the mobile platform is facing and how secure we really are.

Continue reading

Crypto-Trading: Securing Million Worth API-Keys in a Distributed System

The teaching of IT security is often about trust boundaries – these are drawn at the interface of the system to the outside world. While this view is dangerous even with a monolithic system, it is simply wrong with a distributed system. Especially when the system’s data is so delicate that you don’t even want to trust all your own microservices.

In this essay, an approach is discussed to restrict access to data in a distributed system by means of cryptography. However, this is not only about security but also about the practicability and effects on the development.

Continue reading

How to create a K8s cluster with custom nodes in Rancher

Don’t you find it annoying not to be able to manage all your Kubernetes clusters at a glance? Ranger 2.0 offers an ideal solution. 

The following article is less a scientific post than a how-to guide to creating a new Kubernetes cluster with custom nodes in Ranger 2.0. 

But before we drift into the practical part, let’s take a look at what’s going on with Rancher and Kubernetes. We’ll briefly discuss the differences, origins, and use of the two tools. 

What is Kubernetes? 

Kubernetes, also often abbreviated as K8s, is a container orchestration system. An open source system for automating the deployment, scaling and management of container applications. In contrast to other orchestration systems, no fixed workflow is implemented. The actual state of the container system is continuously monitored and changed to the target state. K8s was originally developed by Google in 2014 and subsequently donated to the Cloud Native Computing Foundation (CNCF). 

So-called ‘pods’ representing the smallest unit which are being orchestrated. These are wrappers that consist of either a single container or several containers. They encapsulate these containers from the outside (like a legume). These pods do run on worker nodes (either physically or as virtual machines in a cluster). The cluster is controlled by one (or more) dedicated machine(s), the ‘Kubernetes Master’ (‘Master-Nodes’). 

What is Rancher? 

Rancher 2.0 is an open source container manager for managing multiple K8s clusters. It doesn’t matter if a cluster consists of custom nodes or a hosted Kubernetes cluster like GKE (Google Kubernetes Engine), EKS (Amazon Elastic Kubernetes Service) or AKS (Azure Kubernetes Service). Rancher is able to manage all these clusters centrally. Especially user administration and RBAC (Role Based Access Control), policy and project management, resource management, backup and recovery of clusters, monitoring, logging and much more are covered. 

As we have now seen, we are able to integrate multiple K8s clusters into Rancher. In our particular case, we don’t want to integrate a cluster from the big cloud supplier, but use our own hosts. So let’s take a look at what we need.

Requirements

The following servers all run CentOs 7. Of course you can also use other operating systems.

We used seven hosts. One is only used for rancher. Three are used as master nodes and another three are used as worker nodes.

nunki[01:03]10.208.215.24[1:3]Master-Nodes (etcd)
nunki[04:06]10.208.215.24[4:6]Worker-Nodes
rana0110.208.215.247Rancher

Prerequisite

Docker needs to be installed on all hosts that are to serve as master and worker nodes in the cluster and on the host on which our rancher instance is running?runs. Once Docker is installed, you need to start the Docker daemon. Most Linux distributions use systemctl (systemctl start docker) to start services. If you don’t have systemctl, use the service command. If you want Docker to automatically start at boot (recommended) you should enable the docker service as well. 

Further Setup
In this tutorial, we will not cover ‘SELinux’ configuration for Docker, so we disable it on every host. Edit the /etc/selinux/config file to set the SELINUX parameter to disabled, and then reboot the server. Execute getenforce to see the current mode of SELinux

Since we need certain ports for our Kubernetes environment, we deactivate the firewall completely for our tests. Of course this should not be done in production!
‘Firewalld’ is a complete firewall solution that has been made available by default on all CentOS 7. 

  • First, stop the Firewalld service with:
    • systemctl stop firewalld
  • Then Disable the Firewalld service
    • systemctl disable firewalld
  • To check the status of firewalld, run the following command
    • systemctl status firewalld

We will run Rancher 2.0 in the container using the docker image rancher/ranche r:v2.2.3. To start it, we use the following command:

  • docker run -d –restart=unless-stopped -p 80:80 -p 443:443 rancher/ranche r:v2.2.3

By means of parameter -d we tell the docker deamon that the container should run ‘detached’, in other words: in the background. –restart=unless-stopped allows us to automatically restart the container after a restart (the service must be enabled for this). With parameter -p we define ports, which can be used to access our container from the outside. 

Create a k8s cluster with custom nodes

The following section describes the process of creating a new cluster. For this we call the Rancher web interface in the browser. 

From the Clusters page, click Add Cluster

Add a new Cluster

  1. Choose Custom
  2. Enter a Cluster Name
  3. Use Member Roles to configure user authorization for the cluster.
    • Click Add Member to add users that can access the cluster.
    • Use the Role drop-down to set permissions for each user
    • Use Cluster Options to choose the version of Kubernetes, what network provider will be used and if you want to enable project network isolation
  4. Click Next

Assign nodes to roles

  1. Under Node select the role that certain nodes should fulfill. If, for example, the master nodes are to be created, only ‘Control Plane’ and ‘etcd’ may be selected. 
  2. Optional: Click Show advanced options to specify one or more IP address(es) to use when registering the node, override the hostname of the node or to add labels to the node.
  3. Copy the displayed command to your clipboard.
  4. Log in to your Linux host using your preferred shell, such as PuTTy or a remote Terminal connection. Run the command copied to your clipboard.
  5. After all commands have been executed on your Linux host(s), then click Done.

Result:

  • Your cluster is created and assigned a state of Provisioning. Rancher is standing up your cluster.
  • You can access your cluster after its state is Active.

Active clusters are assigned to two Projects, Default (containing the namespace default) and System (containing the namespaces cattle-system,ingress-nginx,kube-public and kube-system, if present).

In addition to the configured settings, the available resources can also be taken from the Cluster Dashboard.
It also shows the current workload of the nodes.

For more information about the used nodes, select Nodes.
The table shows the current status, assigned role, available resources and their utilization. Nodes can also be restarted or deleted in this overview.
In our example we selected 3 Master Nodes (Role ‘etcd’ and ‘Control Plane’) and 3 Worker Nodes.

As you could see, setting up a cluster is uncomplicated. Within minutes, a Kubernetes cluster with custom nodes can be created and managed within Ranger 2.0. In the Rancher web interface you are able to display all clusters, their projects and namespaces clearly.

You should try it out.