{"id":3788,"date":"2018-07-31T15:08:24","date_gmt":"2018-07-31T13:08:24","guid":{"rendered":"https:\/\/blog.mi.hdm-stuttgart.de\/?p=3788"},"modified":"2023-08-06T21:51:49","modified_gmt":"2023-08-06T19:51:49","slug":"building-a-fully-scalable-architecture-with-aws","status":"publish","type":"post","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2018\/07\/31\/building-a-fully-scalable-architecture-with-aws\/","title":{"rendered":"Building a fully scalable architecture with\u00a0AWS"},"content":{"rendered":"

What I learned in building the StateOfVeganism<\/a> ?<\/h3>\n
\"\"<\/a>
Final setup for the finished project (created with Cloudcraft)<\/figcaption><\/figure>\n

By now, we all know that news and media shape our views<\/strong>on these discussed topics. Of course, this is different from person to person. Some might be influenced a little more than others, but there always is some opinion communicated.<\/p>\n

Considering this, it would be really interesting to see the continuous development of mood communicated towards a specific topic or person in the media.<\/p>\n

<\/p>\n

For me, Veganism<\/a> is an interesting topic, especially since it is frequently mentioned in the media. Since the medias opinion changes the opinion of people, it would be interesting to see what \u201csentiment\u201d they communicate.<\/p>\n

\"\"<\/a><\/p>\n

This is what this whole project is about. Collecting news that talk about or mention Veganism<\/a>, finding out the context in which it is mentioned and analysing whether they propagate negativity or positivity.
\nOf course, a huge percentage of the analysed articles should be classified as \u201cNeutral\u201d if the writers do a good job in only communicating information, so we should keep that in mind, too.<\/p>\n

Of course, this is an incredible opportunity to pick up new toolset, especially when thinking about the sheer number of articles published daily<\/em>.
\nSo, we could think about building a scalable architecture, a scalable architecture that is cheap\/free in the beginning when there is no traffic and only a few articles but scales easily and infinitely once the amount of mentions or traffic increases.
\nI can hear the cloud calling.<\/p>\n

Designing the Architecture<\/h3>\n

Planning is everything, especially when we want to make sure that the architecture scales right from the beginning.<\/p>\n

Starting on paper is a good thing because it enables you to be extremely rough and quick in iterating.<\/p><\/blockquote>\n

 Your first draft will never be your final one, and if it is, you\u2019ve probably forgotten to question your decisions.<\/p>\n

\"\"<\/a>
First Concept with removed components<\/figcaption><\/figure>\n

For me, the process of coming up with a suitable and, even more important, reasonable architecture was the key thing I wanted to improve with this project. The different components seemed pretty \u201ceasy\u201d to implement and build but coming up with the right system, the right communication and a nice, clean data pipeline was the really interesting part.<\/p>\n

In the beginning, I had some bottlenecks in my design which, at one point, would\u2019ve brought my whole system to its knees. In that situation, I thought about just adding more \u201cscalable\u201d services like queues to queue the load and take care of it.
\nWhen I finally had a design which, I guess, could handle a ton of load and was dynamically scalable, it was a mess\u2026 Too many services, a lot of overhead and an overall \u201cdirty\u201d structure.<\/p>\n

When I looked at the architecture a few days later, I realised that there was so much I could optimise with a few changes. I started to remove all the queues and thought about replacing actual virtual machines with FAAS components.
\nAfter that session, I had a much cleaner and still scalable design.<\/p>\n

<\/h4>\n

Think of the structure and technologies, not implementations.<\/h4>\n
\"\"<\/a>
Final Architecture<\/figcaption><\/figure>\n

That was one of the mistakes I made quite early in the project. I started out by looking at what services IBMs BlueMix can offer and went on from there. Which ones could I mix together and use in my design that seemed to work together with triggers and queues and whatever?
\nIn the end, I could remove a lot of the overhead in terms of services by simply stepping away from it and thinking of the overall structure and technologies I need rather than the different implementations<\/strong>.<\/p>\n

\n
\n
\n

Broken down into a few distinct steps<\/strong>, the project should:<\/p>\n