{"id":4164,"date":"2018-08-31T18:00:00","date_gmt":"2018-08-31T16:00:00","guid":{"rendered":"https:\/\/blog.mi.hdm-stuttgart.de\/?p=4164"},"modified":"2023-08-06T21:48:47","modified_gmt":"2023-08-06T19:48:47","slug":"tweets-by-donnie-building-a-serverless-sentiment-analysis-application-with-the-twitter-streaming-api-lambda-and-kinesis","status":"publish","type":"post","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2018\/08\/31\/tweets-by-donnie-building-a-serverless-sentiment-analysis-application-with-the-twitter-streaming-api-lambda-and-kinesis\/","title":{"rendered":"Tweets by Donnie\u200a-\u200aBuilding a serverless sentiment analysis application with the twitter streaming API, Lambda and Kinesis"},"content":{"rendered":"
\n
<\/div>\n
\n
\n
tweets-by-donnie dashboard<\/em><\/figcaption><\/figure>\n

 <\/p>\n

Thinking of Trumps tweets it’s pretty obvious that they are controversial. Trying to gain insights of how controversial his tweets really are, we created tweets-by-donnie<\/a>.<\/p>\n

\u201cIt\u2019s freezing and snowing in New York\u200a\u2014\u200awe need global warming!\u201d
\nDonald J. Trump<\/em><\/p><\/blockquote>\n

You decide if it\u2019s meant as a joke or not.<\/p>\n

But wouldn\u2019t it be nice to know whether the public is seeing this as a joke or whether it\u2019s getting upset by it? That\u2019s where our idea originated from. By measuring the emotions presented in the responses we can see what the public is thinking of Trumps posts throughout the day.<\/p>\n

<\/p>\n

To generate such insights we decided to create a cloud architecture that can deal with a stream of tweets, enrich them and finish it all up with a simple API to query the results.<\/p>\n

Easier said than done.<\/p>\n

Home is where your IDE is, right? Writing code in the AWS console wasn\u2019t a thing we felt good about. Also, it\u2019s not reproducible in any way, which is why we chose the serverless framework<\/a>. It bridges the gap from code in your IDE to the cloud. First we were overwhelmed by the technologies as these were our first steps building anything in the cloud. We never heard of AWS Cloudformation and never touched yaml files before but this seemed the way to go and it turned out to be very handy having all code and configurations checked in a repo. This way changing, recreating, or deleting code (or even the whole architecture) is a breeze. Check out our repo,<\/a> fork it and try it yourself.<\/p>\n

The serverless.yml file acts as your main description of your architecture. It can go from a single lambda function to a whole directory containing separate yaml files for different purposes like functions, resources, roles\u2026 you name it.
\nSpeaking of roles it\u2019s easy to maintain the least privilege principle with serverless. You can create a role per serverless.yml or go as far as creating a role per function.<\/p>\n

Another good thing was creating the Resources on the fly. We needed some DynamoDB tables.<\/p>\n

On a side note: DynamoDB needs some time to get used to. Deliberately select the right set of primary and sort key for your tables because you don\u2019t want to waste time scanning through big tables. In our case we had tweet id\u2019s but that\u2019s not what we\u2019re querying. We are querying for time and as our data is time sequenced, so we chose the day (yyyy-mm-dd) as our primary key and a timestamp as the sort-key. This way we can query for days and sort by timestamp or filter for a time frame of the day.<\/p>\n

You can add resources like this.<\/p>\n

<\/figure>\n

Referencing these resources in other parts of the serverless.yml is quite handy too. For example to trigger a lambda function from a new input to a table we need the stream ARN <\/a>in our event trigger.
\nWith \u2018Fn::GetAtt:[TrumpsTweetDynamoDBTable, StreamArn]\u2019 <\/strong>the Attribute is automatically resolved when deploying.<\/p>\n

<\/figure>\n

If you\u2019re ever in need of examples or documentation have a look at the following links, they were rather helpful in the learning process.<\/em><\/p>\n

\n
\n
\n

AWS CloudFormation<\/strong><\/a><\/p>\n<\/div>\n<\/div>\n

\n

AWS CloudFormation allows you to create and manage AWS infrastructure deployments predictably and repeatedly…<\/a><\/p>\n

\n
\n
\n

serverless\/examples<\/strong><\/a><\/p>\n<\/div>\n<\/div>\n

Serverless Examples\u200a\u2014\u200aA collection of boilerplates and examples of serverless architectures built with the Serverless\u2026<\/em>github.com<\/a><\/p>\n

\n
\n
\n

Now there is a catch with having our configuration files in source control\u200a\u2014\u200a secrets<\/strong>. We don\u2019t want them in any repository. So we had to think of storing secrets such as api keys and passwords elsewhere. There are multiple ways to do this in a secure manner. We chose a method where we store the secrets in a separate yaml file (serverless.env.yml) which is then referenced in the actual serverless.yml. You can reference to other files with \u2018${file(path\/to\/file)}:some.key\u2019. This way we can gitignore the serverless.env.yml containing our secrets but keep the serverless.yml checked in without accidentally committing them to a repo.<\/p>\n<\/div>\n<\/div>\n<\/div>\n

serverless.yml<\/em><\/figcaption><\/figure>\n
serverless.env.yml<\/em><\/figcaption><\/figure>\n

This method seemed feasible for this rather uncritical POC but if you are planning a bigger projects read the following.<\/p>\n

\n
\n
\n
\n
\n

Managing secrets, API keys and more with Serverless<\/strong><\/a><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

Use Lambda environment variables and AWS Parameter Store to handle configuration in your Serverless projects<\/em>serverless.com<\/a><\/div>\n<\/div>\n<\/div>\n<\/section>\n

<\/p>\n

<\/p>\n

\n
\n
\n
\n

Having deployment out of the way let\u2019s get started with the nitty gritty.<\/p>\n

Architecture<\/h2>\n
architecture (created with cloudcraft.co)<\/em><\/figcaption><\/figure>\n

Our architecture consists of three main parts.
\n1. Getting a stream of tweets to the point they can be processed by lambda
\n2. Processing the tweets (filter, enrich)
\n3. Presenting the results<\/p>\n

1. Streaming tweets to AWS Kinesis<\/h3>\n
streaming tweets to kinesis<\/em><\/figcaption><\/figure>\n

Getting the stream of tweets from twitter into AWS we need a consistently running script that puts each tweet on to the Kinesis stream. First we tried deploying a flask app to heroku. What we didn\u2019t know was that a free tier instance on heroku will be idled after 30 mins inactivity ? (no requests to the server). Finding this we decided to deploy an EC2 compute instance that we are in full control of. You can find the code for this flask server here<\/a>. Deploying it is described fairly detailed in the readme<\/a>.<\/p>\n

2. Processing tweets<\/h3>\n

To process the tweets we used three lambda functions and DynamoDB to store the results.<\/p>\n

Preprocess<\/h4>\n
<\/figure>\n

The preprocess function is triggered by incoming items on the kinesis stream. It takes 300 items of the stream and removes retweets and media only tweets. Also, it\u2019s separating tweets from Donald Trump and responses to those into two different DynamoDB tables. (This way the postprocess function is only triggered when saving tweets by D. Trump). CODE<\/a><\/p>\n

Postprocess<\/h4>\n
<\/figure>\n

Postprocess takes care of enriching trumps tweets with tone analysis generated by Watson Tone Analyzer. It\u2019s triggered by each write to the table containing only trumps tweets. CODE<\/a><\/p>\n

PeriodicReports<\/h4>\n
<\/figure>\n

Finally, we need to aggregate the public responses and analyze their tone. For this function we chose to create a report every 10 minutes. The function get\u2019s the latest tweets since it last ran, concatenates the text and runs tone analyzer on them. After it saved the report to DynamoDB we are ready to query the reports and present them. CODE<\/a><\/p>\n

3. Presenting Results<\/h3>\n

To present our results we chose to create a lambda function that acts as a public API to query the reports and the enriched tweets. As we already knew how to query DynamoDB we only had to prepare the queries and wrap them into lambda functions. In this case we had to set HTTP as a trigger event for our functions. This way we have a path for each function. CODE<\/a><\/p>\n

<\/figure>\n

Having CORS enabled for the lambda we first thought it should be working out of the box. But when we tried to call the API from our frontend we still got CORS errors. To get CORS right we had to set the header \u2018Access-Control-Allow-Origin: *\u2019<\/em>for all possible responses from our lambda function including errors.<\/p>\n

<\/figure>\n

Lastly we created a Static Website that calls the API for a given day and displays the tweet + enrichments and a graph for the reports. CODE<\/a><\/p>\n

Summary<\/h3>\n

This project was all about taking our first steps in the cloud. We learned a lot and there are things we would like to have tested e.g. to use elastic beanstalk instead of deploying an EC2 instance for our flask server or make use of more text analysis… Still, we\u2019re proud of what we created starting from zero and in this limited amount of time. It\u2019s really refreshing to create an application in the cloud. If done right we think you can definitely save some money and headaches from maintaining your own server. It just takes some time to get used to.<\/p>\n

Written by Daniel Rot\u00e4rmel, Paul Fauth-Mayer<\/p>\n<\/div>\n<\/div>\n<\/section>\n","protected":false},"excerpt":{"rendered":"

tweets-by-donnie dashboard   Thinking of Trumps tweets it’s pretty obvious that they are controversial. Trying to gain insights of how controversial his tweets really are, we created tweets-by-donnie. \u201cIt\u2019s freezing and snowing in New York\u200a\u2014\u200awe need global warming!\u201d Donald J. Trump You decide if it\u2019s meant as a joke or not. But wouldn\u2019t it be […]<\/p>\n","protected":false},"author":894,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[120,650],"tags":[83,87,203,25,202,204],"ppma_author":[767],"class_list":["post-4164","post","type-post","status-publish","format-standard","hentry","category-cloud-technologies","category-scalable-systems","tag-aws-lambda","tag-ec2","tag-kinesis","tag-nodejs","tag-serverless","tag-twitter"],"aioseo_notices":[],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":2161,"url":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2017\/03\/09\/of-apache-spark-hadoop-vagrant-virtualbox-and-ibm-bluemix-services-part-5-spark-applications-in-pia-project\/","url_meta":{"origin":4164,"position":0},"title":"Of Apache Spark, Hadoop, Vagrant, VirtualBox and IBM Bluemix Services – Part 5 – Spark applications in PIA project","author":"bh051, cz022, ds168","date":"9. March 2017","format":false,"excerpt":"The main reason for choosing Spark was a second project which we developed for the course \u201cProgramming Intelligent Applications\u201d. For this project we wanted to implement a framework which is able to monitor important events (e.g. terror, natural disasters) on the world through Twitter. To separate important tweets from others\u2026","rel":"","context":"In "Student Projects"","block_context":{"text":"Student Projects","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/category\/student-projects\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4122,"url":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2018\/08\/27\/building-a-serverless-web-service-for-music-fingerprinting\/","url_meta":{"origin":4164,"position":1},"title":"Building a Serverless Web Service For Music Fingerprinting","author":"Alexis Luengas","date":"27. August 2018","format":false,"excerpt":"Building serverless architectures is hard. At least it was to me in my first attempt to design a loosely coupled system that should, in the long term, mean a good bye to my all-time aversion towards system maintenance. Music information retrieval is also hard. It is when you attempt to\u2026","rel":"","context":"In "Cloud Technologies"","block_context":{"text":"Cloud Technologies","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/category\/scalable-systems\/cloud-technologies\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2018\/08\/Architecture-Diagram-300x190.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":12879,"url":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2021\/03\/10\/how-to-scale-real-time-tweet-delivery-architecture-at-twitter\/","url_meta":{"origin":4164,"position":2},"title":"How to Scale: Real-time Tweet Delivery Architecture at Twitter","author":"Tamara Hezel","date":"10. March 2021","format":false,"excerpt":"There is a lot to say about Twitters infrastructure, storage and design decisions. Starting as a Ruby-on-Rails website Twitter has grown significantly over the years. With 145 million monetizable daily users (Q3 2019), 500 million tweets (2014) and almost 40 billion US dollar market capitalization (Q4 2020) Twitter is clearly\u2026","rel":"","context":"In "Allgemein"","block_context":{"text":"Allgemein","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/category\/allgemein\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2021\/03\/Bildschirmfoto-2021-03-10-um-21.35.13.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2021\/03\/Bildschirmfoto-2021-03-10-um-21.35.13.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2021\/03\/Bildschirmfoto-2021-03-10-um-21.35.13.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2021\/03\/Bildschirmfoto-2021-03-10-um-21.35.13.png?resize=700%2C400&ssl=1 2x"},"classes":[]},{"id":21653,"url":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2021\/09\/17\/studidash-a-serverless-web-application\/","url_meta":{"origin":4164,"position":3},"title":"“Studidash” | A serverless web application","author":"dk119","date":"17. September 2021","format":false,"excerpt":"by Oliver Klein (ok061), Daniel Koch (dk119), Luis B\u00fchler (lb159), Micha Huhn (mh334) Abstract You are probably familiar with the HdM SB-Funktionen. After nearly four semesters we were tired of the boring design and decided to give it a more modern look with a bit more functionality then it currently\u2026","rel":"","context":"In "Allgemein"","block_context":{"text":"Allgemein","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/category\/allgemein\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2021\/09\/grafik-1.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2021\/09\/grafik-1.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2021\/09\/grafik-1.png?resize=525%2C300&ssl=1 1.5x"},"classes":[]},{"id":5635,"url":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2019\/03\/05\/a-dive-into-serverless-on-the-basis-of-aws-lambda\/","url_meta":{"origin":4164,"position":4},"title":"A Dive into Serverless on the Basis of AWS Lambda","author":"Can Kattwinkel","date":"5. March 2019","format":false,"excerpt":"Hypes help to overlook the fact that tech is often reinventing the wheel, forcing developers to update applications and architecture accordingly in painful migrations. Besides Kubernetes one of those current hypes is Serverless computing. While everyone agrees that Serverless offers some advantages it also introduces many problems. The current trend\u2026","rel":"","context":"In "Allgemein"","block_context":{"text":"Allgemein","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/category\/allgemein\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2019\/03\/warm.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2019\/03\/warm.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2019\/03\/warm.png?resize=525%2C300&ssl=1 1.5x"},"classes":[]},{"id":8478,"url":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2019\/08\/09\/production-monitoring-industry-4-0\/","url_meta":{"origin":4164,"position":5},"title":"Production Monitoring – Industry 4.0","author":"Philip Betzler","date":"9. August 2019","format":false,"excerpt":"When I was invited to a design thinking workshop of the summer school of Lucerne - University of Applied Sciences and Arts, I made my first experience with the end user interaction part of Industry 4.0. It was an awesome week with a lot of great people and made me\u2026","rel":"","context":"In "Cloud Technologies"","block_context":{"text":"Cloud Technologies","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/category\/scalable-systems\/cloud-technologies\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2019\/08\/Connection_Cut-1.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2019\/08\/Connection_Cut-1.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2019\/08\/Connection_Cut-1.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2019\/08\/Connection_Cut-1.png?resize=700%2C400&ssl=1 2x"},"classes":[]}],"jetpack_sharing_enabled":true,"authors":[{"term_id":767,"user_id":894,"is_guest":0,"slug":"dr053","display_name":"dr053","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/ade3e64639b5cb9766139afde0fd997b4c8c25fe539d5b19f5c80e67fd1f89d8?s=96&d=mm&r=g","0":null,"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/posts\/4164","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/users\/894"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/comments?post=4164"}],"version-history":[{"count":26,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/posts\/4164\/revisions"}],"predecessor-version":[{"id":25484,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/posts\/4164\/revisions\/25484"}],"wp:attachment":[{"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/media?parent=4164"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/categories?post=4164"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/tags?post=4164"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/ppma_author?post=4164"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}