The teaching of IT security is often about trust boundaries – these are drawn at the interface of the system to the outside world. While this view is dangerous even with a monolithic system, it is simply wrong with a distributed system. Especially when the system’s data is so delicate that you don’t even want to trust all your own microservices.

In this essay, an approach is discussed to restrict access to data in a distributed system by means of cryptography. However, this is not only about security but also about the practicability and effects on the development.

---

1. Introduction

This semester a concept for solving a fictitious scenario should be developed. The object of research should be a software product that fulfills two characteristics. First, it should be a data-intensive application that requires a distributed system due to the sheer amount of data. In addition, the data to be processed should be delicate and therefore particularly protection-needy.

In this work, therefore, a fictitious software product from the crypto area is chosen. The sector seems appropriate, as it is repeatedly exposed to a large number of attacks. Furthermore, there have been very specific and sophisticated attacks lately, making it even more of an interest. Of course, there is also a lot of data to process and these are in any case sensitive, as they represent the financial portfolio of the users.

The research object of this essay should be a platform that aggregates several accounts of a user and joins the data in a unified overview. Comparable real-world platforms would be Blockfolio, Cointracker or Blox.io. All of these have in common that the user’s portfolio is loaded from several providers, processed and then displayed. A provider is the operator of an exchange or wallet. In this way, the user can check his complete portfolio at a central location. The APIs of the providers shall be used to load the data.

Scenario

The fictitious crypto portfolio management solution Portfolio-Tracker.io shall represent the research object and be briefly described. The product consists of various microservices. The user only interacts with the Client service, most of the data is in the control of the microservice API Server. Connected to this are among others a Trading and an Importer service.

User Journey

At the beginning, there is authentication, a task that is characterized by being critical but also very generic. It is therefore outsourced to a service provider. Since the login is based on the open standard OAuth, it can be easily integrated into the project. Participating developers will find good documentation and a wide range of information on the Internet and have often already worked with it. A password is never transmitted directly to Portfolio-Tracker.io.

The user then enters the API keys obtained from the individual providers in an onboarding process. These API keys ideally have only read permission – following the principle of least privilege. Nevertheless, write permissions are necessary for certain functions on the page and will therefore be activated by some users. Accordingly, it can be assumed that the API keys offer at least read legitimation or even rights to trade on behalf of the user. Therefore the keys are to be regarded as the most critical data of the system.

After the user has connected all wallets and exchanges, a background job is started that imports the entire user’s trading portfolio. The portfolio data are the second critical asset. Since these data provide very accurate information about the financial situation of the user and also include every single transaction made.

To unlock additional features like tax generation, the user can also choose between three different packages that are available for different prices on the website. This is again a process that handles critical data (address, identity, payment information, etc). However, this task is very generic again, so there are ready-made services such as Shopify. According to the scenario, Software-as-a-Service (SaaS) was used again.

Technology

The target project should have an ordinary modern stack. Several microservices, scaled horizontally, are deployed on top of Kubernetes. The services are developed by individual teams and communicate with each other either synchronously via GRCP or decoupled via queues. Data are exclusively controlled by one microservice – shared nothing.

The stack consists primarily of JavaScript-based components. In the client, React is used while the server and most other microservices rely on NodeJS.

To make matters even more difficult, the company behind the product is a start-up that depends on the ability to adapt paradigms such as moving fast. The product is developed iteratively, involving the users. Therefore, many features are often fundamentally revised or even discarded. This not only increases the susceptibility of code to errors but also influences the philosophy of development: Why should a service be perfectly tested when it might be removed without replacement after a few weeks?

The security situation is therefore at least difficult if not critical.

Objectives

In addition to the entire portfolio, the API keys are of course particularly in need of protection. This work should accordingly conceive a solution, how to secure external API keys of users in a distributed system.

2. API Keys

API keys are rarely even surpassed directly into the hands of the end-user – commonly API keys are acquired in the background of processes like OAuth. They are usually used to map identification and authentication in machine-to-machine communication. However, in the crypto scene, it is quite common for users to give applications access to their data by manually requesting an API key from the offering platform and entering it in the consuming platform. In this way, the user transfers his identity to the consuming party and thus grants the capability to act on his behalf.

3. Recent Attacks

As already mentioned, the crypto scene is confronted with attacks a lot. One reason for this is the low age of the whole scene. Most platforms and tools are not yet fully matured. In addition, there is high market dynamics, new products are constantly popping up while others are disappearing again.

This ignites the already critical security situation, as the good that the platforms take care of is money. And not just money, but money that can be very easily withdrawn.

Agama Wallet

One of the attacks finally executed is an attack that has certain parallels to spear phishing. Only that instead of user data should be subtracted, malicious code should be integrated. For this purpose, a seemingly useful package called electron-native-notify was published on the Node Package Registry (NPM). When the open-source crypto wallet Agama integrated this library into its code, a new version containing malicious code was released shortly afterwards [1].

The actual attack leaves open whether Agama was specifically selected as the target or whether the attacker simply published the package and then waited until a worthwhile target used it. However, thanks to the history of the repository, the operators of the Agama Wallet were able to determine that the attacker spent several months making useful contributions to Agama, until he finally installed the malicious code [2].

The attack is special, because first a developer identity was established, which actively participated in the project. In addition, the malicious code was actually deployed and caused damage.

Copay Wallet

Another attack was noticed, again by the NPM team in late November 2018. The origin of the attack can be traced back to social engineering. [3]. The attacker gained access by taking over the maintainership of the event-stream packet. He then released a new version of the package which in turn added a dependency to event-stream including malicious code. But this was only one of the steps taken to disguise the attack. Among other things, the malicious code was embedded damaged to prevent detection in a dynamic analysis. Only during the build process was the code then corrected again and the actual malicious code could be integrated. The loaded code was then integrated into the created Copay Wallet bundle aimed for capturing crypto assets from end-users once installed. The code was only activated if 100 Bitcoin (1 BTC = $5,290 as of Nov 19, 2018) or 1,000 Bitcoin Cash (1 BCH = $384 as of Nov 19, 2018) were found in the wallet. It was also ensured that the malicious code was only included if the specific development environment of the Copay Wallet was found – so that the attack wouldn’t generate extra attention in other projects using the code.

The attack is therefore characterized by being very covert. Not only was the malicious code only reloaded by another dependency and could only become active after a correction, but also only individual, high target, victims were selected and the attack was executed specifically and exclusively for this wallet.

Coinbase Sim Port

Another very specific attack took place in May 2019. A user of Coinbase, who had a $100,000 worth portfolio at his disposal there, was relieved of his credit. Although he had actually secured everything in an exemplary manner. This also included two-factor authentication via SMS.

Around 10 pm the user notices that his mobile phone no longer had cellular service. Shortly afterward, a Google prompt opens and asks for a login. This fails because the attacker had already changed the password of the Google account. The user decides to solve the problem the next day. Only 50 minutes later, the attacker had already triggered the password reset flow on Coinbase, the e-mails received were deleted in order to conceal further traces. This is necessary because Coinbase only resets passwords with a 24-hour delay and the victim regains access to the Google account in the meantime – by visiting a shop of the network operator and equipping himself with a new sim card in order to perform the password reset of the Google account.

Since the network operator did not report a sim-port attack as a possible cause and the reset e-mails have been deleted, the user is still not aware of the ongoing attack.

Then the events of the previous evening are repeated, the cellular connection is interrupted at 10 pm, the attacker ported the SIM card again, reset the password of the Google account and can now complete the initiated password reset flow of the Coinbase account. The Coinbase account is then cleared.

Only during the second visit to the network operator’s shop does he notice unusual access from another state and the sim-port attack is identified. Since the money has now been transferred to an account outside Coinbase, their support team cannot assist either.

This attack is so extensively documented because the victim is an engineering manager at a blockchain company himself and published the course after the attack [4]. The attack is characterized by the fact that it targets a particular end-user and also uses him as an entry point. The attack is very individual and it can be assumed that the victim has been specifically selected. The attacker must have gained access to personal data in advance and could thus successfully take over the Google account. This is a worthwhile target, accounts on other platforms can be identified quickly via the e-mail history. Then either a password reset or even a direct login can be carried out if the account is connected to the Google account.

4. Passwords

An obvious solution to encrypt user data is to do this with a password. This method is also used for Tools like VeraCrypt or BitLocker. However, the password is actually only used to encrypt another key. The actual data are then encrypted with this key. This additional abstraction prevents that the complete area of the hard disk must be re-encrypted in case of a password change, instead only the encryption of the generated key is exchanged.

From the point of view of the use case, password-based encryption tends to be unsuitable here. After all, the user must actively enter a password to decrypt the data. This means that it would not be possible to carry out any processing in the background, for example, to update the user’s portfolio. But there are other reasons against passwords, which will be briefly outlined [5].

Passwords are not bound to a scope

Passwords, and thus inevitably encrypted data, are not bound to an origin or scope. This makes them vulnerable to man-in-the-middle attacks.

Passwords are shared secrets

Passwords practically transfer the complete identity. Since the receiving party gets access to the password in plain text, it can theoretically use this password to falsely authenticate itself as a user wherever the access data matches.

Passwords are hard to create and maintain

Due to security factors such as entropy and length, passwords are usually not easy to create. Also, remembering appropriate passwords is likely to overwhelm many users and it is questionable whether password managers can be the solution for the general public, as they also represent a hurdle in usability and lag behind many usage scenarios, especially in the cross usage perspective.

Passwords are easy to re-use

It is precise because strong passwords are so difficultly handled that users tend to reuse their passwords. However, this is extremely questionable from the point of view of identity transfer. An analysis shows that close to all users reuse their passwords [6].

Passwords are easy to steal

This point is specifically powered by the re-use aspect. No matter how secure a system may be, it cannot prevent users from using the same combination of email and password for other services.

Passwords are hard to secure and to encrypt

Even within the area of responsibility of a product, passwords are difficult to secure. They must at least be encrypted and enriched with an individual salt to prevent decryption with the help of a rainbow table.

But this is only the first step, if passwords are in the system they have to be protected at every request. Just placing the password in an HTTP GET request can lead to caching or logging [7]. Especially the unintentional logging of the password in plain text is very common, even in large companies such as Facebook [8], GitHub [9] or Twitter [10]. In addition, any resources generated with the password may also be in need of protection, such as tokens or session cookies.

Future of passwords

They have come to stay, even if the originator of the passwords has reservations by now [11]. But the future seems promising, with WebAuthN a standard has been created that targets many problems of passwords, especially the shared secret and the bound-to-origin aspects. Also, OpenID’s Project You Only Login Once (YOLO) alleviates the problem by having only one central place where a secret is used by the user [12].

As mentioned earlier, passwords are not suitable for this application. However, they cannot be completely banished from the system due to the password-based login.

5. Solution

The investigations have shown that the login should at least be regarded as vulnerable. From this, it can be concluded that the solution should be designed in such a way that the user can never see API keys once they have been entered.

But it’s not just the end-user or hacker who poses a problem. Developers also make mistakes and in a distributed system with independent teams, fast release cycles and questionable low test-coverage it must be assumed that sooner or later mistakes will happen. This should also be included in the solution, here the attack surface can be cryptographically significantly minimized by only giving access to specific parties. This means for the fictitious example that only the Client and the microservice that imports the data needs access to the API keys. For all other components the data could be removed, but this would be error-prone again. So the data should be encrypted instead. This ensures that only the two microservices Client (entry point) and Importer (usage point) need to be closely monitored. If an intermediate service logs the data unintentionally, it is encrypted and unusable.

Passwords are inappropriate, not only because Auth0 does not transmit any passwords to the system, but also because it is not suitable for users and developers to handle them. Instead, asymmetric encryption should be used to ensure that the Client can only encrypt data, but cannot decrypt it later. On the other hand, the Importer should be able to decrypt the data encrypted by the client. In addition, the end-user should be able to select in the client which of the microservices – Trader, Importer or both – should have access to the data. So an explicit capability should granted given which cannot be changed without the user’s involvement.

Sealed Secrets

Sealed Secrets originate from the Dev-Ops movement and aim to secure secrets using asymmetric cryptography [13]. This puts them in direct competition with approaches like HasiCorps Vault.

Instead of keeping secrets in a central secure bastion and regulating access with the help of access-control-lists (ACL), the entity to be secured is encrypted with the public key of the future consumer. The latter, equipped with the private key, can then decrypt the data.

The decisive advantage is that encrypted data are harmless.¹ So they can be possibly stored in a publicly accessible repository.

While the concept in the Dev-Ops movement aims at storing all configurations in a self-contained repository and thus integrating the infrastructure into the code, the concept is also suitable for cryptographically preventing access to data – e.g. within a third microservice. This makes it extremely interesting for this application because it minimizes the attack surface to the point of entry and point of use.

1 Insofar as an appropriate complexity has been chosen.

JSON Web Encryption

The challenge of encryption within the scenario is that the entry point should be in the client. Accordingly, encryption must be performed there using the server’s public key. Many JavaScript implementations which are primarily offered for NodeJS still use native bindings. These cannot be loaded in the client – a compilation to WebAssembly is unsuitable due to the practicability. With the Web Cryptography API [14], however, there is a standard that is now also supported in the most common browsers – 92% at the time of writing [15]. JSON Web Encryption (JWE) is the ideal exchange format. Due to the widespread use of the related standardizations JWT, JWS, JWK and JWA, there are sufficient libraries available – even those that run in the browser.

Encryption Keys

The key used for encryption plays an important role in the chosen approach. Accordingly, the private key has to be secured well. There are different methods for this, each with different advantages and disadvantages.

From a security point of view, it would be best if the private key is only located in one hardware module on the server on which the Importer microservice is operated. However, this raises several questions. What happens for example in case of a hardware problem, it would not be practicable if all users had to re-enter their data. A hardware module would also be a strict commitment to a specific environment that offers these modules. A further complication is that the Importer microservice is also to be scaled horizontally, all of which would severely limit the choice of supplier for operation.

Instead, the key access is to be mapped via the authorization system of a key management solution. Possible products would be for example AWS Key Management Service (KMS) or GCP’s Cloud Key Management Service (KMS). The client only has access to the public key, while the Importer microservice is granted access to the private key.

Role of the client

The client has immense importance. After all, it represents the single point of entry into the system. This is the easiest place for attackers to access unencrypted data. To make matters worse, the source code is freely available. This increases the risk that attackers can check exactly which third-party libraries are used.

The client is written in JavaScript and relies on the popular package manager NPM. The default behavior in NPM is that installed packages, following semantic versioning, are entered with a version range (^2.0.0) [16]. NPM then allows, if there is no package-lock.json file, to install new versions that are released as a patch (e.g. 2.0.1) or minor (e.g. 2.1.0) update. Usually, there is an automatically generated lock file, but this often creates merge conflicts and is then regenerated. In this case, a developer would have unintentionally increased several versions.

Revisiting the attacks described in the introduction, it quickly becomes clear that there is a security risk here. To prevent an unintentional patch or minor update, the default range modifier (^) must be removed. This is the only way to guarantee that always the desired and reviewed version of a library is included. It should not be overlooked that in most cases a quick update of patches and minor versions is desired. Ultimately, these can receive important security updates.

Total Cost of Ownership

In the previous chapters, it was already mentioned that a Cloud Key Management Service is used to map authorizations and key handling. The reasons for this decision will be explained in more detail. Especially under the aspects of Total Cost of Ownership.

It will certainly become clear that the securing of users API keys is a very specific use-case no cloud provider can offer a complete solution to. Nevertheless, in this scenario, it is not an alternative to design and write this code oneself. This quickly becomes clear when you recall the fictitious company: a startup, keyword moving fast.

If features are possibly removed again, code is written quickly, the team is changed often, etc, then all this is an excellent environment for mistakes. Accordingly, and associated with the effort of maintenance and operation, one should clearly distance oneself from an implementation that important and sensitive.

Instead, the very specific problem of encrypting API keys can be solved by generic building blocks – encryption, key handling, etc. – again. These, in turn, are wonderfully suited to be solved with ready-to-use cloud services. The advantage is obvious, the own code base remains narrow, errors become obvious and while no cloud service should be blindly trusted, the security is better compared to startup code.

But cloud services also come with a risk. Service operators are sometimes very agile when it comes to changing existing services. Therefore, a certain amount of maintenance effort is to be expected here as well. Even the worst-case should be taken into account: the complete discontinuation of the service.

6. Conclusion

The data is hot. The security situation is correspondingly critical. The team must be aware that the API keys of the end-users are attractive data for possible attackers. Especially because cryptocurrencies are so easy to move and so hard or impossible to recover.

Due to the context of the company (startup) it is extremely difficult to guarantee reasonable security. It would be irresponsible to just assume that the system is completely secure. Therefore, in addition to securing the system, safety also includes planning the worst case: data loss to a malicious third party. Hence, it must be ensured that the damage is minimized as far as possible by a fast and automated procedure. For example by a key rotation as well as notifications to end-users and platform operators. These procedures must be well prepared and ready-to-use.

When planning this fictitious scenario, some aspects are particularly outstanding. They should, therefore, receive special attention in the context of this conclusion:

Context

The solution outlined here makes sense for this example. However, as soon as a single parameter is changed in the fictitious scenario, the solution may lose its validity. This is because the solution is directly linked to the context. If the company were a bank instead of a startup, the heavy use of SaaS could make less sense.

This also reveals that the designed solution should be questioned regularly, even if the requirements remain the same – after all, other characteristics such as company goals, size or legal regulations may have changed.

Development Process & Dependencies

In order to design a safe software product, above all, an appropriate underlying process is required. The security requirements must be incorporated into this process to ensure that they are consistently applied.

In software development, tools such as a continuous integration pipeline are used to establish standardized procedures. And there were a lot of positive developments here recently. Gitlab, for example, now integrates concepts such as Code-Ownership and Protected Environments [17]. This means that security requirements can not only be automated but also integrated seamlessly into the development process.

While the review process of own code is comparatively simple, checking dependencies poses a greater challenge. But as the recent attacks (chapter ) show, this is necessary. The conflict is as follows: On the one hand, one would like to integrate patches as quickly as possible; after all, they could contain security-relevant improvements. On the other hand, each update requires a new review to ensure that no malicious code is included.

But the situation is highly complex: manual reviews are time-consuming, automated reviews are even easier to trick. The published code in the package source does not necessarily have to match the original source code in the repository, and the attack could be in a dependency of the actual dependency or included in obfuscated or compiled code.

Hence, the only acceptable solution for dependencies is applying fixed versions which are constantly updated in a separate process with manual and automated reviews.

Software as a Service

Software as a Service is an enabler. It is particularly suitable for generic tasks such as a shopping cart or authentication. But it can also be helpful for highly individual tasks. In the latter case, it is just not possible to utilize the high-level approach that maps exactly to what is needed. Instead, low-level building blocks and APIs can be used. As a result, the share of self-implementation increases slightly but is still in striking proportion to the total own contribution.

Enabler not only in the sense that a service can be put into operation without development time, maintenance costs, improvement iterations, and co, but also an enabler in the sense that the team can concentrate on its domain and the team’s profile does not have to be broadened with additional experts. Also, the size of the team can be small and thus the manageability of the team can be guaranteed. A smaller team has other positive influences such as proven higher productivity [18].

But services are also interesting from a security point of view. While libraries are hacked almost daily, attacks on services are rarer, as they are developed and distributed by vendors. If these vendors are considered trusted, one can assume that the code was already security reviewed and thus can be used.

Summary

Absolute security cannot be achieved in this context. After all, security aspects must be combined with business goals and usability. Nevertheless, the attack surface could be minimized and the critical points identified. This allows the teams to develop more agilely as appropriate guidelines are applied depending on the microservice. It remains open how long a company like the fictitious could operate in reality without being hacked.

References

[1] I. npm, “The npm blog — plot to steal cryptocurrency foiled by the npm…” https://blog.npmjs.org/post/185397814280/plot-to-steal-cryptocurrency-foiled-by-the-npm, Jun-2019.

[2] D. Pigeon, “Update regarding vulnerability discovered in komodo’s agama wallet.” https://komodoplatform.com/update-agama-vulnerability/, Jun-2019.

[3] I. npm, “The npm blog — details about the event-stream incident.” https://blog.npmjs.org/post/180565383195/details-about-the-event-stream-incident, Nov-2018.

[4] S. Coonce, “The most expensive lesson of my life: Details of sim port hack.” https://medium.com/coinmonks/the-most-expensive-lesson-of-my-life-details-of-sim-port-hack-35de11517124, May-2019.

[5] S. Raman, “The web authentication api – imagine a world without passwords.” https://www.youtube.com/watch?v=MesXuMg0WKo&t=351s, Oct-2018.

[6] keeper, “Consumer mobile security app use.” https://keepersecurity.com/assets/pdf/Keeper-Mobile-Survey-Infographic.pdf, 2017.

[7] D. Wind, “When ebay implements a keylogging feature – slashcrypto’s page.” https://slashcrypto.org/2016/06/29/Ebay/, Jun-2016.

[8] J. Kastrenakes, “Facebook stored hundreds of millions of passwords in plain text – the verge.” https://www.theverge.com/2019/3/21/18275837/facebook-plain-text-password-storage-hundreds-millions-users, Mar-2019.

[9] C. Cimpanu, “GitHub accidentally recorded some plaintext passwords in its internal logs.” https://www.bleepingcomputer.com/news/security/github-accidentally-recorded-some-plaintext-passwords-in-its-internal-logs/, May-2018.

[10] S. Gallagher, “Twitter alerts users: Please change your passwords, we’ve seen them | ars technica.” https://arstechnica.com/information-technology/2018/05/twitter-advises-users-to-reset-passwords-after-bug-posts-passwords-to-internal-log/, May-2018.

[11] L. Eadicicco, “Inventor of the password – business insider.” https://www.businessinsider.com/inventor-of-the-password-2014-5?IR=T, May-2014.

[12] OpenId, “Openid/openyolo-web: Web protocol for credential exchange and update – “you only login once”.” https://github.com/openid/OpenYOLO-Web, Mar-2015.

[13] A. Lees, “Bitnami engineering: Sealed secrets: Protecting your passwords before they reach kubernetes.” https://engineering.bitnami.com/articles/sealed-secrets.html, Jul-2017.

[14] W3C, “Web cryptography api.” https://www.w3.org/TR/WebCryptoAPI/, Jan-2017.

[15] caniuse, “Can i use… Support tables for html5, css3, etc.” https://caniuse.com/#feat=cryptography.

[16] I. npm, “About semantic versioning | npm documentation.” https://docs.npmjs.com/about-semantic-versioning.

[17] “GitLab 11.3 released with maven repository and protected environments | gitlab.” https://about.gitlab.com/2018/09/22/gitlab-11-3-released/.

[18] D. Rodriguez, M. Sicilia, E. Garcia, and R. Harrison, “Empirical findings on team size and productivity in software development.” JOURNAL OF SYSTEMS AND SOFTWARE, vol. 85, no. 3, pp. 562–570, Aug. 2011.