Introduction

Over the last five years, the use of cloud computing services has increased rapidly, in German companies. According to a statistic from Bitkom Research in 2018, the acceptance of cloud-computing services is growing.

Cloud-computing brings many advantages for a business. For example, expenses for the internal infrastructure and its administration can be saved. Resource scaling is easier, it can be done when the appropriate capital is available. In general the level of innovation within companies is also increasing, so this can support new technologies and business models. Cloud-computing can also increase data security and data protection, because the big providers must comply with standards and obtain certification through the EU-DSGVO General Data Protection Regulation, which came into force in May 2018. There is even special cloud providers that offer such services, so these are working based on the regulations of the TCDP (Trusted Cloud Data Protection) certification.

However, not all that glitters is gold. Based on a study by KPMG Bitkom Research, 19 percent of the interviewed companies reported that data security incidents have happened in their company by the use of public cloud solutions within the last 12 months. But in my opinion, 19% is far too high, because we as experts, have the task to increase this security even more!

Figure 1: secure cloud computing

In the end, there is the question:
Which areas or components within the cloud-computing technologies are affected in terms of data-security?

To be honest, there are a lot of components involved, these are:

Cloud-infrastructures:
1. SaaS – Software as a Service
2. PaaS – Platform as a Service
3. IaaS – Infrastructure as a Service
Devices that use cloud-services and therefore also
The user / developer / administrator (The human)

That’s many, right?

DevOps and Cloud Security

In recent years, the job title DevOps was established.
It is a composition of the words “Development” and “Operations” and it focuses on the improvement of software development. This approach should help increase the general quality of the product, and as well the speed of software delivery.
That’s what Rob England said about DevOps in 2017:

DevOps is agile IT operations delivery, a holistic system across all the value flow from business need to live software. […]
DevOps is a philosophy, not a method, or framework, or body of knowledge, or *shudder* vendor’s tool.
DevOps is the philosophy of unifying Development and Operations at the culture, system, practice, and tool levels, to achieve accelerated and more frequent delivery of value to the customer, by improving quality in order to increase velocity.
Culture=behaviour, teamwork, responsibility/accountability, trust/empowerment…
Practice=policy, roles/RACI, processes/procedures, metrics/reporting, KPIs/improvement…
Tools=shared skills, toolmaking for each other, common technology platforms…

These following practices often help to achieve the goal of DevOps:

CI – Continuous Integration: (Version management + CI server with test suites)
CD – Continuous Delivery: (Delivering increments with manual release of an authorized person to the production system)
CD – Continuous Deployment: (Automated delivery of increments to the production system)

But what does the DevOps approach have to do with the cloud?

David Linthicum said:

Thus, DevOps won’t have much value without the cloud, and the cloud won’t have much value without DevOps. This fact alone is being understood inside enterprises that once thought they could move to one or the other, and that no dependency existed. We’re finding that dependencies between DevOps and cloud do indeed exist.

DevOps often use tools and cloud architectures for their work. Sometimes even the operation part shifts towards the DevOps. They must ensure that the security of the product is given and that the cloud-computing technologies for that platform is used correctly.

This blog entry is intended to give a brief overview of what DevOps and administrators can do to secure their applications and their cloud-platforms. It is quite clear that not everything can be covered in this article. Therefore I always recommend, do your own extended research based on your individual topics and needs!

The here explained recommendations are focusing on the behaviour of software in general and are divided around the following topics:

First steps
Secrets management
Data encryption
Logging and monitoring

At the end of the article there is a short example about secret management with the tool git-crypt. This is intended to develop a basic understanding of the use of such secret management tools.

First steps

The first thing to say is that you ALWAYS need to use common sense. So my first recommendation: Use your brain! This also includes an extended understanding of security for employees in the IT environment and training in this area, if they are not familiar with this. It should be clear, that someone with full root access always uses strong passwords in their work and also handles their keys responsibly. No one in IT should be fooled by spam messages with phishing or other harmful links. Passwords, keys, tokens and all confidential information should never be shared within the IT department or even within the company! The use of secret management is also helpful here (see below). Each person should have personalized access. When hiring new employees, it does not hurt to check their background in advance to see if they are trustworthy. Clear and secure authorizations concepts must also be used. Not everyone should have full root access. However, humans can also be regarded as a source of errors in the area of software development. For example, an API-key accidentally uploaded to github.com in version control is also a security leak. Here are some advice’s to get this problems under control:

Repo-security-scanner: CLI tool which finds secrets (passwords, private keys) that were accidentally committed to a repository.
Gitrob: is similar to a repo security scanner. Offers the possibility to search for “sensitive” data in repositories on an organizational basis. Provides results in a web interface.
Another tool that can also search repositories is TruffleHog. It works with the search method to find strings with high entropies – which usually reflect API keys.
By using merge requests and code reviews within a project, security-critical source code passages can be identified.
Do not use public services such as Github for versioning. Gitlab software can be hosted within companies themselves.

Developers can also use the OWASP Dependency Check Tool to check their software packages for security vulnerabilities.

The hardware used such as laptops, PCs, servers, etc. must be additionally protected. This succeeds through:

Use of distributions with SE-Linux / AppArmor. These work according to the MAC (Mandatory Access Control) principle. Applications may only perform activities defined for them. Only the necessary software packages should be installed. Unused programs and services must be uninstalled.
Osquery is a tool that makes operating systems searchable like databases. With the help of appropriate packages, security holes and strange behavior can be detected.
Use 2-factor authentication wherever it’s possible. Use strong passwords and encryption of hard drives. If a laptop is forgotten e.g. in a cafe, strangers cannot easily access sensitive data. The chapter Data Encryption contains detailed information about this topic.
Use firewalls and IDS/HIDS systems such as iptables, fail2ban and OSSEC.

In the architecture and conception of the server landscape/cloud and its network, individual services should be installed separately on different machines. This is achieved through firewalls, DMZ principles and various host systems. If possible, the software used, should be separated into microservices. These measures help to keep possible attack vectors small. However, if a break-in occurs, the damage is not as big, as when everything is executed on one monolithic system.

The BSI (German Federal Office for Information Security) makes the following proposals:

Security zone for cloud management, storage network and live migration, if server virtualization is used
Separate customer’s virtual machines at IaaS into security zones
The management network of the CSP (Cloud Solution Provider) should be isolated from the data network.

An important point for DevOps is to secure the CI/CD pipeline. The reason for this is, that sensitive activities such as building source code takes place here. If an attacker succeeds in hijacking such a system, he has the possibility to integrate and execute his own malicious code! The recent incident in which Eric Holmes gained access to the Homebrew repository at github.com within 30 minutes showed how extremely dangerous an unsecured Jenkins pipeline is. In addition, pipelines should be isolated in a separate network and also to other servers. If containers (Docker, RKT) are used, only secure images from trusted registries should be pulled, which also perform additional security scans!

Secrets Management

What are secrets?

The word secret has become a buzzword for keys, certificates, passwords, tokens and API-Keys in the recent years. Actually, everything that needs protection or provides access, is called a secret. But there are various software implementations, that can help you protect your data. A distinction can be made between selfhosted secret management and cloudbased secret management. However, it is not excluded that a selfhosted tool is running within a cloud architecture, like on a virtual machine. The cloudbased tools are mostly implementations provided by the cloud platform vendors themselves.

The capabilities of these tools are different. The tools could be ranked in 3 levels as shown in the figure.

Figure 2: secret levels

Level 1: Only the access to the secrets is restricted, the secrets themselves are not encrypted.
Level 2: The access is restricted again, but the secrets are encrypted.
Level 3: Like Level 2, but with additional features such as Time To Live (the duration how long a secret can be used, till it is invalid) or ACL’s (Access Control List). With this technology an user is able to assign finely granular access-rights to the various secrets in use.

Here is an overview of such tools, but not sorted by their levels.

Selfhosted-tools:

Git-crypt: File-based encryption within a git repository
Hashicorp Vault: Offers many features like rolling secrets (changing the secret within a certain period of time), saving logfiles (when was a secret used by whom, which secret was used, … )
Docker Secrets: Can be used inside docker containers, only in combination with Docker Swarm. Secrets are stored securely within a raft consensus cluster and can be injected into containers.

Cloudbased-tools:

AWS KMS: Secret management of Amazon Web-Services.
Azure Key Vault: Secret management of Microsoft Azure.
Google Cloud KMS: Secret Management of Google Cloud Platform.

Due to the complexity, it is very difficult to make a recommendation for one of the tools. It always depends on the existing architecture. In addition, you have to differentiate between the gained security and the complexity of introducing these new tools in your company.

Furthermore, an additional layer is added to the architecture of the system. For example, if the secret management fails, the deployment process of an application can also fail!

If you want to learn more about the different tools, I recommend this overview and this one.

Data encryption

There is a thing called Data at Rest. This is when data is persistently saved in a storage, like in a Hard Drive, SSD or on tapes and so on. The opposite of this is data, which is used right at the time it is needed like processed data within a CPU or stored in the RAM. This is called Data at Use.

When it comes to the encryption of data, Data at Rest is usually meant. Under the keywords “Data at Rest Encryption” you will find a lot of tooling to help you with this challenge. Encrypting data makes sense, because it helps in cases of loss or theft. If this is done the data cannot be read directly by other people without any further hacks or software. Encryption also avoids unwanted external access if a software is misconfigured. This way the attacker can access a service, but the encrypted data is worth nothing as long as he does not get the information the data is keeping of the service. A distinction is made between “data volume encryption” and software feature encryption. For example, the data of a database can be encrypted directly by using their software feature. Data volume encryption usually encrypts the entire hard disk. The best practice with volume data encryption is:

Use powerful AES/RSA algorithms
Keys should be changed regularly
Keys should be kept separate from the actual data storage
Periodic checking of encryption/decryption is essential
Only necessary data should be stored and encrypted

Some operating systems such as MacOS offer the possibility to encrypt the hard disk by default. There are also open source solutions such as LUKS (Linux Unified Key Setup), Veracrypt and many more. In addition, there are also commercial solutions such as Portworx, which can especially score in the area of encryption of container volumes such as docker volumes.

Logging and Monitoring

Logging describes the recording of all action of an application process, triggered by users and other systems, distinct the application. The Error.log of the web server Apache is a good example of logging data. If an error occurs, it can be reviewed in this file. Usually, authentication and access to services with the differentiation between the privileges of the users are also performed during logging. Normally log files are written by default for most applications. As DevOps or administrator, the challenge is to collect the log files and manage, evaluate and understand them accordingly.

Monitoring describes the monitoring of computer-services and their availability. Usually performance and availability of a service are checked in real-time, e.g. by the CPU load of a server or the availability, by the ping for an IP address. Monitoring tools offer the possibility to send automated notifications (E-Mail, SMS,…) in case of increased load or if an error occurs. But also graphical elements and the presentation of the collected data in dashboards is handled by monitoring tools.

Monitoring usually includes logging, since decisions etc. cannot be made without the previously gathered data. These two topics are therefore closely related. In the past, the following tools have become established:

The individual cloud provider offer their own monitoring and logging services. AWS calls this Cloudwatch and Google Cloud Platform also offers a variety of management tools in this area. It is also very difficult to make a recommendation here. It always depends on the experience of the employees and which architecture is used. I like Icinga2 because it is a German project and is currently in heavy development. In addition, there is solid support within the monitoring portal forum.

Example “Secrets Management” with git-crypt and key-file detection with truffleHog :

This Example will show a simple procedure of using git-crypt on MacOS. Notice: This is intended for a single user only. If you want to collaborate, please use an extended tutorial like this. You can also use git-crypt with symmetric keys – but then you have to make sure that the transport channel, you share the keys on, has to be secure!

Because of its easiness, homebrew (although it was recently hacked) and github.com is used in this example. You can bypass this technology by using gitlab and installing the tools by hand. Git-crypt was chosen for the example because it is simple. Furthermore you get very fast increased security and it affects almost every developer who works with git-versioning.

For more Examples check the Github-Repository.

Create a fake API-Key and push it to github.com

As you can see i created the example API-Key named “mysupersecretapi.key” through the help of this website.

$ touch mysupersecretapi.key && echo "6ZM86DNgIeXfUkTnq8E3nUI08VkruOv4Zjufs5lh15ZFDRnsydKEU6lBTilb7mm" > mysupersecretapi.key
$ cat mysupersecretapi.key
6ZM86DNgIeXfUkTnq8E3nUI08VkruOv4Zjufs5lh15ZFDRnsydKEU6lBTilb7mm

Then i pushed this file into my Github.com example-repository.

$ git add mysupersecretapi.key && git commit -m "DevOoooooops - i make a mistake" 
$ git push

Install truffleHog and scan the example-repository

For more informations of this Tool visit this repository.

Installation is made with “pip” the python package manager. (If you don’t have pip installed, use this page)

Run the following in the CLI for truffleHog installation and scan the example-repository:

$ sudo pip install truffleHog
$ trufflehog https://github.com/immae1/securesystems_example
~~~~~~~~~~~~~~~~~~~~~
Reason: High Entropy
Date: 2018-08-14 22:04:30
Hash: 55476 ba6c4455213672bec2aa0b35f41f0341e5f
Filepath: mysupersecretapi.key
Branch: origin/master
Commit: DevOoooooops - i make a mistake
@@ -1 +0,0 @@
-6ZM86DNgIeXfUkTnq8E3nUI08VkruOv4Zjufs5lh15ZFDRnsydKEU6lBTilb7mm
~~~~~~~~~~~~~~~~~~~~~

As you can see above – truffleHog found this “accidentally” pushed API-Key!

So now i will show you how to secure this repository.

Install Git-Crypt and GPG – encrypt the repository

If you are using a linux machine, simply install through your package manager the tools gpg an git-crypt. On MacOS first install homebrew (the missing package manager for macOS) and run the following steps for installation:

$ brew install git-crypt
$ brew install gpg

Next Step is to generate a GPG-Key and secure it with a passphrase:

$ gpg --gen-key
gpg (GnuPG) 2.2.9; Copyright (C) 2018 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Note: Use "gpg --full-generate-key" for a full featured key generation dialog.
GnuPG needs to construct a user ID to identify your key.
Real name: SecureSystems
Email address: SecureSystems@test.com
You selected this USER-ID:
   "SecureSystems <SecureSystems@test.com>"
Change (N)ame, (E)mail, or (O)kay/(Q)uit?
$ O
Pass a valid passphrase and make sure you can remember it (<strong>in this case test123</strong>)

For this example we use this Github.com Repository:

Clone this Repo and cd into it.

Now we have to initialize the git-crypt tool with the following command in the CLI:

$ git-crypt init
Generating key...

After this step, verify that git-crypt is correctly added into your repository:

$ cd .git/git-crypt/keys

If there is a file named “default” located you can go further.

Git-Crypt uses the .gitattributes file to determine which files have to encrypted.

In our example we want to encrypt all files which are added into the repository.

So generate the .gitattributes file and add a . like this:

$ touch .gitattributes
$ echo "*.* filter=git-crypt diff=git-crypt" > .gitattributes
$ echo ".gitattributes \!filter \!diff" >> .gitattributes
$ cat .gitattributes
*.* filter=git-crypt diff=git-crypt
.gitattributes !filter !diff

Now we have to commit and push this new git-crypt setup of this repository:

$ git add .gitattributes
$ git commit -m "added git-crypt setup"
$ git push
$ git crypt add-gpg-user SecureSystems
$ git push

The time has come to test the encryption through git-crypt so we create a new testfile and some text into it. Then we commit and push it to github.com:

$ touch secret3000.txt && echo "this is a secret text! we should encrypt it" > secret3000.txt
$ git add secret3000.txt && git commit -m "added a testfile" && git push

If we visit the github.com file explorer of this repository we can see that the pushed file is not longer a .txt . For now on it’s a binary file:

Figure 3: github.com file explorer binary file

After downloading this file we can confirm that the file is stored encrypted on the repository on github.com:

Figure 4: secret3000.txt encrypted

If you clone the repository into a new folder the files in there, are also encrypted:

$ git clone https://github.com/immae1/testgitrypt.git checkcryptedfiles
Cloning into 'checkcryptedfiles'...
remote: Counting objects: 19, <b>done</b>.
remote: Compressing objects: 100% (13/13), <b>done</b>.
remote: Total 19 (delta 2), reused 19 (delta 2), pack-reused 0
Unpacking objects: 100% (19/19), <b>done</b>.

$ cd checkcryptedfiles
$ cat secret3000.txt
%�㉌\�����3�bҐ��%

As you see the secret3000.txt file is encrypted. To decrypt it you have to run:

$ git crypt unlock
      x Passphrase: _____..._______ x   
<strong>use the passphrase “test123”</strong>

Then you see the decrypted text of the file. (Notice: You need the have the previously created gpg-key on your local disk!)

$ cat secret3000.txt
this is a secret text! we should encrypt it

To encrypt again, you can use:

$ git crypt lock

But wait what’s with the pushed “mysupersecretapi.key” file?

If you run the “git-crypt status” in CLI you can see that the committed file is not encrypted (as we know), but the local one is encrypted.

$ git-crypt status
encrypted: mysupersecretapi.key *WARNING: staged/committed version is NOT ENCRYPTED! *
..
Run 'git-crypt status' with the '-f' option to stage an encrypted version

But we want that the staged remote file in the example-repository is also encrypted. To achieve this do the following:

$ git-crypt status -f
mysupersecretapi.key: staged encrypted version
Staged 1 encrypted file.
Warning: if these files were previously committed, unencrypted versions still 
exist in the repositorys history.
$ git status
On branch master
Your branch is up to date with 'origin/master'.Changes to be committed:
(use "git reset HEAD <file>..." to unstage)modified: mysupersecretapi.key$ git commit -m "is now encrypted"
[master 818c328] is now encrypted
1 file changed, 0 insertions(+), 0 deletions(-)
$ git push

Now we can see that the “mysupersecretapi.key” is not longer a readable file. So it was also encrypted through git-crypt.

Figure 5: apikey is encrypted

If you look at the previous commits, you can still see the fake api.key! In practice, you should now clean the repository of this commit. But because this is an example, I’ll let it be.

Figure 6: old commit – secret is still readable!

As you can see, git-crypt is a powerful, easy-to-use tool if you want to secure your repositories! I strongly recommend it, as it means that no information about their secrets is publicly accessible. The used GPG keys must be backed up. In the worst case, if they are lost, you no longer have access to the data in the repository.

Summary

Increasing security within cloud computing is an extremely extensive process. There are a variety of tools and methods that can be useful. It should not be overlooked that this is a fast-growing environment. Therefore, good manpower with high expertise is necessary. Employee training is therefore essential. By so-called Gameday’s and e.g. the use of the Simian Netflix Army the DevOps team can not only be trained, but also gain more experiences. Companies should therefore take time for such actions. It is also essential that a cultural change takes place within companies. Especially in management it must be understood that the security of applications and their platforms must be prior before the speed when using new innovations and products!