Category: System Engineering
Open Source Batch and Stream Processing: Realtime Analysis of Big Data
Abstract Since the beginning of Big Data, batch processing was the most popular choice for processing large amounts of generated data. These existing processing technologies are not suitable to process the large amount of data we face today. Research works developed a variety of technologies that focus on stream processing. Stream processing technologies bring significant…
Autoscaling of Docker Containers in Google Kubernetes Engine
In this blog post we are taking a look at scaling possibilities within Kubernetes in a cloud environment. We are going to present and discuss various options that all have the same target: increase the availability of a service.
Kubernetes: from Zero to Hero with Kompose, Minikube, k3sup and Helm — Part 2: Hands-On
This is part two of our series on how we designed and implemented a scalable, highly-available and fault-tolerant microservice-based Image Editor. This part depicts how we went from a basic Docker Compose setup to running our application on our own »bare-metal« Kubernetes cluster.
Kubernetes: from Zero to Hero with Kompose, Minikube, k3sup and Helm — Part 1: Design
This is part one of our series on how we designed and implemented a scalable, highly-available and fault-tolerant microservice-based Image Editor. The series covers the various design choices we made and the difficulties we faced during design and development of our system. It shows how we set up the scaling infrastructure with Kubernetes and what…
Writing High Performance Code on Modern Hardware
Today, with the use of modern hardware combined with optimized high performant code, it is an easy task to process more than 500 million images per day on a single machine. Small improvements in the underlying implementations can have extreme large impacts on the execution time and are therefore fundamentally important to handle the huge…
Reproducibility in Machine Learning
The rise of Machine Learning has led to changes across all areas of computer science. From a very abstract point of view, heuristics are replaced by black-box machine-learning algorithms providing “better results”. But how do we actually quantify better results? ML-based solutions tend to focus more on absolute performance improvements (measured by metrics) instead of…
Experiences from breaking down a monolith (3)
Written by Verena Barth, Marcel Heisler, Florian Rupp, & Tim Tenckhoff DevOps Code Sharing Building multiple services hold in separated code repositories, we headed the problem of code duplication. Multiple times a piece of code is used twice, for example data models. As the services grow larger, just copying is no option. This makes it…
Migrating to Kubernetes Part 1 – Introduction
Written by: Pirmin Gersbacher, Can Kattwinkel, Mario Sallat
Migrating to Kubernetes Part 2 – Deploy with kubectl
Written by: Pirmin Gersbacher, Can Kattwinkel, Mario Sallat
Migrating to Kubernetes Part 3 – Creating Environments with Helm
Written by: Pirmin Gersbacher, Can Kattwinkel, Mario Sallat