{"id":25686,"date":"2023-09-04T12:40:21","date_gmt":"2023-09-04T10:40:21","guid":{"rendered":"https:\/\/blog.mi.hdm-stuttgart.de\/?p=25686"},"modified":"2023-09-04T12:43:17","modified_gmt":"2023-09-04T10:43:17","slug":"lyrics-analyzer-on-cloud-infrastructure-music-and-things-between","status":"publish","type":"post","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2023\/09\/04\/lyrics-analyzer-on-cloud-infrastructure-music-and-things-between\/","title":{"rendered":"Lyrics Analyzer &#8211; On Cloud, Infrastructure, Music, and Things Between"},"content":{"rendered":"\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p><\/p>\n<cite>After several unsuccessful attempts to weld my results together into such a whole, I realized that I should never succeed. The best that I could write would never be more than philosophical remarks; my thoughts were soon crippled if I tried to force them on in any single direction against their natural inclination. [\u2026]<br>I should have liked to produce a good book. This has not come about, but the time is past in which I could improve it.<br><em>Ludwig Wittgenstein<\/em> on his impending deadline. The deadest of lines as far as deadlines go as he passed away shortly after writing this.<\/cite><\/blockquote>\n\n\n\n<h2 class=\"wp-block-heading\">Introduction<\/h2>\n\n\n\n<p>The songs are meant to provide comic relief during the reading of this semi-coherent rambling.<br>Yes there&#8217;s an album, and yes you should listen to it before proceeding.<br>Not because it has anything to do with this text but because it&#8217;s a really good album.<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-rich is-provider-spotify wp-block-embed-spotify wp-embed-aspect-21-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe title=\"Spotify Embed: Who Knew\" style=\"border-radius: 12px\" width=\"100%\" height=\"152\" frameborder=\"0\" allowfullscreen allow=\"autoplay; clipboard-write; encrypted-media; fullscreen; picture-in-picture\" loading=\"lazy\" src=\"https:\/\/open.spotify.com\/embed\/track\/7fXPWrFoxtEGnoxu67ZjQa?utm_source=oembed\"><\/iframe>\n<\/div><\/figure>\n\n\n\n<p>Hip Hop music and me starting my journey as a programmer are connected through a peculiar link which I would like to expand on before getting into the gritty details of this project resulting from a chain of events that were set in motion more than a decade ago.<\/p>\n\n\n\n<p>I&#8217;ve been a big fan of Hip Hop music ever since discovering Eminem when I was around 10 years old.<\/p>\n\n\n\n<p>I still remember listening to his first albums on my CD player while diligently reading the lyrics to every song until I had memorized entire albums by heart &#8211; a habit I still have to this day.<\/p>\n\n\n\n<p>At the time, I was far from being a proficient English speaker, so I had to look up a lot of the words and even knew some only by their phonetics.<\/p>\n\n\n\n<p>I don&#8217;t quite know what it was, but something had sparked my interest in both the English language and lyrics in general. Some years later, after having finished school, after having discovered plentiful other artists and after having gone through the same process of obsessing over their work, I found a website (<a href=\"https:\/\/pudding.cool\/projects\/vocabulary\/index.html\" title=\"\">https:\/\/pudding.cool\/projects\/vocabulary\/index.html<\/a>) ranking rappers by their vocabulary.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/ranking.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"640\" data-attachment-id=\"25689\" data-permalink=\"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2023\/09\/04\/lyrics-analyzer-on-cloud-infrastructure-music-and-things-between\/ranking\/\" data-orig-file=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/ranking.png\" data-orig-size=\"1110,694\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"ranking\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/ranking-1024x640.png\" src=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/ranking-1024x640.png\" alt=\"\" class=\"wp-image-25689\" srcset=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/ranking-1024x640.png 1024w, https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/ranking-300x188.png 300w, https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/ranking-768x480.png 768w, https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/ranking.png 1110w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/a><\/figure>\n\n\n\n<p>Lo and behold, my favorite rapper at the time, Aesop Rock, was at the top of the list with a whooping 7,879 unique words used throughout his songs. If you know you know &#8211; no surprises there. However, some of the artists I was listening to at the time didn&#8217;t appear at all.<\/p>\n\n\n\n<p>Consequently, I did the only thing any reasonable person with a little too much time on their hands would do: I started to learn Python, the goal being to write a [script](https:\/\/github.com\/4350pChris\/LyricAnalyzer) to scrape the lyrics of artists I was listening to and run the same analysis on them.<\/p>\n\n\n\n<p>During that time I realized &#8211; I really dig programming.<\/p>\n\n\n\n<p>The project I will be talking about is my attempt at rewriting the aforementioned script in a more modern fashion, using the tools I have learned over the past few years.<\/p>\n\n\n\n<p>Who would&#8217;ve guessed that a simple 100 LoC script could be blown up to a 4,500 LoC project involving multiple cloud services?<\/p>\n\n\n\n<p>Well, I did. So that&#8217;s exactly what I did. Now let&#8217;s get to it.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><a href=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/cloc.png\"><img loading=\"lazy\" decoding=\"async\" width=\"643\" height=\"384\" data-attachment-id=\"25690\" data-permalink=\"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2023\/09\/04\/lyrics-analyzer-on-cloud-infrastructure-music-and-things-between\/cloc\/\" data-orig-file=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/cloc.png\" data-orig-size=\"643,384\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"cloc\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/cloc.png\" src=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/cloc.png\" alt=\"\" class=\"wp-image-25690\" srcset=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/cloc.png 643w, https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/cloc-300x179.png 300w\" sizes=\"auto, (max-width: 643px) 100vw, 643px\" \/><\/a><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">The Setup<\/h2>\n\n\n\n<figure class=\"wp-block-embed is-type-rich is-provider-spotify wp-block-embed-spotify wp-embed-aspect-21-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe title=\"Spotify Embed: The Set Up (feat. Havoc)\" style=\"border-radius: 12px\" width=\"100%\" height=\"152\" frameborder=\"0\" allowfullscreen allow=\"autoplay; clipboard-write; encrypted-media; fullscreen; picture-in-picture\" loading=\"lazy\" src=\"https:\/\/open.spotify.com\/embed\/track\/7lPJ5PrQnQRurUZIzMCQib?utm_source=oembed\"><\/iframe>\n<\/div><\/figure>\n\n\n\n<p>First off, let&#8217;s talk shortly about the tools I used to build this project and why I chose them.<\/p>\n\n\n\n<p>One thing to be said for the infrastructural decision making process is that on one hand I was trying to have a look at services I hadn&#8217;t used before, while on the other hand I had to stay within the limits of the free tier of AWS, as I didn&#8217;t want to spend any money on this project.<\/p>\n\n\n\n<p>Some of them I will go into more detail later on, but for now, let&#8217;s just have a quick overview.<\/p>\n\n\n\n<div class=\"wp-block-jetpack-markdown\"><h3>Lyrics<\/h3>\n<p>For the lyrics I used the <a href=\"https:\/\/docs.genius.com\/\">Genius API<\/a>, simply because this is the website I grew up reading lyrics on and they offer a free API to search for artists and songs.\nHowever, it doesn\u2019t provide any statistics or even lyrics to songs.\nWhat it does provide is a link to the actual lyrics on their website, which is why I had to scrape them from there.<\/p>\n<h3>Language<\/h3>\n<p>Initially I was torn between using <a href=\"https:\/\/golang.org\/\">Go<\/a> and <a href=\"https:\/\/www.rust-lang.org\/\">Rust<\/a> for this project, as I had been wanting to learn both for quite some time. However, I decided against it, as I wanted to focus on the cloud services and not on learning a new language.\nTherefore, I decided to go with <a href=\"https:\/\/www.typescriptlang.org\/\">TypeScript<\/a>, as I\u2019m quite familiar with it and like it a lot.\nIn terms of code structure I wanted to try out <a href=\"https:\/\/en.wikipedia.org\/wiki\/Domain-driven_design\">Domain-driven Design<\/a>, as I had never used it before and it seemed like a good fit for this project, as it tends to lend itself better to a microservices approach than other architectures.\nWhile this isn\u2019t a microservices project per se, I figured I might as well pretend and it a shot to see how it works out.<\/p>\n<h3>Hosting &#8211; K8s vs Serverless<\/h3>\n<p>Since one of the prerequisites for this project was to use some sort of cloud service, I had to choose between writing a more traditional application and deploying it to a Kubernetes cluster or using a serverless approach.\nI had already worked with Kubernetes in the past and, while I do like it a lot, I wanted to use this project as an opportunity to learn something new, as I had never written a proper serverless application before, which is why I decided to go with the latter.<\/p>\n<h3>Lambdas<\/h3>\n<p>The next step was finding a way to deploy my application to the cloud. I had seen the horrors of <s>war<\/s> manually deploying lambda functions in blog posts pretending that this was a-okay and decided I would have none of that.\nOut of the box, AWS offers a way to deploy lambda functions using CloudFormation via <a href=\"https:\/\/aws.amazon.com\/serverless\/sam\/\">SAM<\/a>, but I didn\u2019t want to lock myself in too much, plus I had heard a lot of good things about the <a href=\"https:\/\/www.serverless.com\/\">Serverless Framework<\/a> and decided to give it a shot.\nAnother option would\u2019ve been <a href=\"https:\/\/www.terraform.io\/\">Terraform<\/a>, but I had already used it in the past and wanted to try something new.<\/p>\n<h3>Website<\/h3>\n<p>For the website, I decided to stick with Serverless as well and integrate the deployment into the same pipeline as the backend.\nFor something so simple, this was surprisingly hard to do, but more on that later.<\/p>\n<h3>Message Queues<\/h3>\n<p>Since I was quite sure I wouldn\u2019t want to do all of the work in a single lambda function, I needed a way to communicate between them.\nFor this, I decided to use <a href=\"https:\/\/aws.amazon.com\/sqs\/\">SQS<\/a>, as I had already committed to using AWS and this seemed like the way to go.\nI had also considered using <a href=\"https:\/\/aws.amazon.com\/sns\/\">SNS<\/a>, but decided against it, as I didn\u2019t see the purpose at the time. This proved to be a mistake, albeit one that was quite straightforward to fix, as I will explain later.<\/p>\n<h4>Quick Note<\/h4>\n<p>Another option would\u2019ve been to invoke the lambda functions via AWS step functions. On one hand, it\u2019s kinda neat to be explicit about the state machine you\u2019re implementing, which in this case models the work of fetching the lyrics, parsing them, doing analysis, etc. and where each step corresponds to a lambda function, but on the other hand we would need to define this state machine using YAML files, which use their own kind of config I would\u2019ve needed to learn.\nAnd to be honest, something doesn\u2019t feel quite right about defining your application logic in YAML files\u2026\nSo I decided against using step functions.<\/p>\n<h3>Database<\/h3>\n<p>For the database, I decided to go with <a href=\"https:\/\/aws.amazon.com\/dynamodb\/\">DynamoDB<\/a> which is less of a full-blown database and more of a key value store.\nMostly because I wanted to try it out and it seemed like a good fit for this project as there will be no complex relations and the data model is quite simple.\nOther options like RDS I had already used in the past, and their pricing model, while still covered by the free tier, seemed less generous than that of DynamoDB.<\/p>\n<\/div>\n\n\n\n<h2 class=\"wp-block-heading\">Enter The Cloud (36 Chapters)<\/h2>\n\n\n\n<figure class=\"wp-block-embed is-type-rich is-provider-spotify wp-block-embed-spotify wp-embed-aspect-21-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe title=\"Spotify Embed: Enter The Wu-Tang (36 Chambers) [Expanded Edition]\" style=\"border-radius: 12px\" width=\"100%\" height=\"352\" frameborder=\"0\" allowfullscreen allow=\"autoplay; clipboard-write; encrypted-media; fullscreen; picture-in-picture\" loading=\"lazy\" src=\"https:\/\/open.spotify.com\/embed\/album\/3tQd5mwBtVyxCoEo4htGAV?utm_source=oembed\"><\/iframe>\n<\/div><\/figure>\n\n\n\n<div class=\"wp-block-jetpack-markdown\"><p>Now that we\u2019ve talked a little about the tools used, without further ado, let\u2019s get into the gritty details.<\/p>\n<h3>The Backend<\/h3>\n<p>For the backend, I decided to go more or less full-on DDD. This first involved a lot of reading both on the topic of DDD itself and code implementing its proposed patterns.\nI tried to follow hexagonal architecture which also proved to be not quite straightforward. All in all I would say this requires a different style of thinking about structuring your code from the one I was used to, which was both challenging but also quite fun.\nIn addition I intended to be thorough with unit testing, which worked out quite well.<\/p>\n<h4>The Domain<\/h4>\n<p>The domain I ended up with is a quite simple one, consisting of a single aggregate root, the <code class=\"\" data-line=\"\">Artist<\/code>, which contains a list of <code class=\"\" data-line=\"\">Songs<\/code> and the calculated <code class=\"\" data-line=\"\">Stats<\/code> for the artist. As I wanted to avoid ending up with a so called \u201canemic domain model\u201d, I decided to make the <code class=\"\" data-line=\"\">Artist<\/code> responsible for calculating its own stats, which is why it contains a reference to its <code class=\"\" data-line=\"\">Songs<\/code> and not the other way around. However, I achieved a decoupling of the entity and the logic by using a <code class=\"\" data-line=\"\">StatsCalculator<\/code> interface which is implemented by a class which is, in turn, injected into the <code class=\"\" data-line=\"\">Artist<\/code> by its accompanying <code class=\"\" data-line=\"\">ArtistFactory<\/code>.<\/p>\n<p>Quite a lot of abstractions already at this point for something so simple, but that seems to be the point in this case as it does allow for quite a lot of flexibility.\nAlso, I meant to go a little overboard on this, just to see what happens.<\/p>\n<h4>The Application<\/h4>\n<p>This is where it gets a little interesting. I modeled this layer around <code class=\"\" data-line=\"\">use cases<\/code>, which all contain an <code class=\"\" data-line=\"\">execute<\/code> method.\nI did this because I initially intended to use CQRS, which I ended up not doing, but I still found this unified method name to be quite nice as the class name itself already carries the semantics of the operation.\nIn general, use cases return either DTOs or nothing at all. This is because I wanted to make the layer that calls these use cases not have to bother with the domain model.<\/p>\n<p>These use cases dictate what functionality would need to be implemented by the infrastructure layer.\nThat is, anything involving any of the mentioned services like storage, the message queue or downloading lyrics.<\/p>\n<p>A thing that I needed some time to come to grips with was the fact that use cases have to be triggered in a specific order &#8211; however, they can\u2019t really invoke one another as this would lead to a single lambda function doing all the work.\nIn order to achieve the needed decoupling, I opted for firing integration events whenever a use case finishes, which in turn trigger the next lambda function in the chain.<\/p>\n<p>Here\u2019s what such an integration event looks like in practice:<\/p>\n<pre><code class=\"language-typescript\" data-line=\"\">export type FetchedSongsEvent = IntegrationEvent&lt;&#039;fetchedSongs&#039;&gt; &amp; {\n  songs: SongDto[];\n};\n<\/code><\/pre>\n<p>It extends a base type which contains the <code class=\"\" data-line=\"\">eventType<\/code> and then adds whatever data is needed for the next lambda function to do its job.<\/p>\n<p>That way I could theoretically scale each use case differently e.g. the one that does the downloading or parsing, which brings me to the next layer.<\/p>\n<h4>Infrastructure<\/h4>\n<p>Before talking about the code compromising this part of the application, let\u2019s have a look at the infrastructure I ended up using from the point of starting an analysis run for an artist.<\/p>\n<\/div>\n\n\n\n<figure class=\"wp-block-image size-full\"><a href=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/aws.png\"><img loading=\"lazy\" decoding=\"async\" width=\"557\" height=\"886\" data-attachment-id=\"25691\" data-permalink=\"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2023\/09\/04\/lyrics-analyzer-on-cloud-infrastructure-music-and-things-between\/aws\/\" data-orig-file=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/aws.png\" data-orig-size=\"557,886\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"aws\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/aws.png\" src=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/aws.png\" alt=\"\" class=\"wp-image-25691\" srcset=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/aws.png 557w, https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/aws-189x300.png 189w\" sizes=\"auto, (max-width: 557px) 100vw, 557px\" \/><\/a><\/figure>\n\n\n\n<div class=\"wp-block-jetpack-markdown\"><p>At first I was using a single queue to handle all events, as there are what AWS calls <strong>filters<\/strong> which allow selecting events according to certain fields in the message.\nIn my case there\u2019s <code class=\"\" data-line=\"\">eventType<\/code> which is a string that denotes the event that was fired and could therefore be used to select the correct event from the queue.\nNow, this led to a peculiar thing to happen (for the uninitiated, at least) &#8211; jobs would execute only sometimes.\nOther times they would simply vanish into the void. But why?<\/p>\n<\/div>\n\n\n\n<figure class=\"wp-block-embed is-type-rich is-provider-spotify wp-block-embed-spotify wp-embed-aspect-21-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe title=\"Spotify Embed: Do Not Fire!\" style=\"border-radius: 12px\" width=\"100%\" height=\"152\" frameborder=\"0\" allowfullscreen allow=\"autoplay; clipboard-write; encrypted-media; fullscreen; picture-in-picture\" loading=\"lazy\" src=\"https:\/\/open.spotify.com\/embed\/track\/1APrKVAuOsotWulrz0GyAz?utm_source=oembed\"><\/iframe>\n<\/div><\/figure>\n\n\n\n<p>Well, after <em>hours<\/em> of troubleshooting (no kidding) I finally found the section in the docs that explained why my lambdas weren&#8217;t firing. If a lambda receives an event and the filter does not match, it silently discards the received message.<\/p>\n\n\n\n<div class=\"wp-block-jetpack-markdown\"><p>This subverted my initial expectation that these messages would simply be passed over silently, staying in the queue for the correct handler to pick them up.<\/p>\n<p>So my first solution was to use a separate queue for each integration event.\nHowever, upon reflecting a little on this I came to the conclusion that this was highly unwieldy &#8211; I would have to add a queue in the <code class=\"\" data-line=\"\">serverless.yaml<\/code> every time I wanted to implement a new type of event and I didn\u2019t seem to find a nice way to have no duplicate code when it came to handling and firing events.\nI felt like a single queue <em>is<\/em> the best abstraction, as we want <em>one<\/em> message bus for the application.\nThis is what I had alluded to earlier &#8211; the solution to this is to use an SNS topic and have queues subscribe to it.\nIn contrast to SQS, SNS does not delete messages from the topic in this instance, as it does not care about subscribers not caring for a particular topic and will simply send events to all subscribers that are interested in the current message.\nWhile this technically still requires separate SQS queues for each event, I found these to lend themselves much better to the way Serverless\u2019 configuration works.<\/p>\n<p>This is called the <a href=\"https:\/\/docs.aws.amazon.com\/sns\/latest\/dg\/sns-common-scenarios.html\">Fanout Pattern<\/a> and is a quite common way to handle this problem.<\/p>\n<p>Apart from this, there isn\u2019t anything too interesting about the infrastructure layer.\nIt implements a repository to fetch the artist from DynamoDB, a service to fetch lyrics from the <a href=\"https:\/\/docs.genius.com\">Genius API<\/a>, a service to fire the aforementioned integration events using SNS and other things which mostly act as glue between the domain and the actual data structures which are stored in the database or sent via queues.<\/p>\n<h4>Presentation Layer<\/h4>\n<p>Last but not least, there\u2019s a separate layer which represents the API.\nWhile this could also be integrated into the infrastructure layer I felt like a separate folder might be suited better as this involves quite a lot of glue code, e.g. for validating input data for events or HTTP requests.\nMoreover, this is the part of the application that actually utilizes the use cases from the application layer.\nIf this part were in the infrastructure layer I feel the resulting bidirectional dependencies, i.e. the application layer using repositories from the infrastructure and the infrastructure importing use cases from the application layer, might lead to confusion or even circular dependencies.<\/p>\n<h4>Unit Testing<\/h4>\n<p>As I mentioned earlier, I wanted to be thorough with unit testing.\nI ended up with a test coverage of almost 100% (excluding the lambda handlers though):<\/p>\n<\/div>\n\n\n\n<figure class=\"wp-block-image size-full\"><a href=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/coverage.png\"><img loading=\"lazy\" decoding=\"async\" width=\"992\" height=\"530\" data-attachment-id=\"25692\" data-permalink=\"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2023\/09\/04\/lyrics-analyzer-on-cloud-infrastructure-music-and-things-between\/coverage\/\" data-orig-file=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/coverage.png\" data-orig-size=\"992,530\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"coverage\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/coverage.png\" src=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/coverage.png\" alt=\"\" class=\"wp-image-25692\" srcset=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/coverage.png 992w, https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/coverage-300x160.png 300w, https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/coverage-768x410.png 768w\" sizes=\"auto, (max-width: 992px) 100vw, 992px\" \/><\/a><\/figure>\n\n\n\n<div class=\"wp-block-jetpack-markdown\"><p>As a test framework I used <a href=\"https:\/\/github.com\/avajs\/ava\">ava<\/a> and for mocking I used <a href=\"https:\/\/github.com\/testdouble\/testdouble.js\">testdouble<\/a> both of which were new to me and both of which I found to be quite nice to work with.\nA lot better than jest, at least.\nOne thing to be said for DDD is that it lends itself <em>extremely<\/em> well to unit testing as the layers are very well separated and should be easy to test in isolation.\nIf they\u2019re not, you\u2019re probably doing something wrong.\nAnd this kind of immediate feedback regarding your architecture is something I really appreciated while working on this project.<\/p>\n<h4>Integration Testing<\/h4>\n<p>This is something I struggled with and in the end didn\u2019t implement in any meaningful way.\nOne of the barriers was that I didn\u2019t want to setup an entire infrastructure for testing purposes as I feared I might go over my free tier limits.\nAnother barrier is that integration testing is <em>hard<\/em> to do. And even harder to do right.\nWhile Serverless does offer a ridiculously hidden away and badly documented way to run some tests automatically that invoke your functions I found this approach to be lackluster at best and completely useless any other time.\nI think this may be the biggest part that\u2019s missing in this project but it seemed to be too much of a hassle to get it to work properly to warrant me going through all the hoops of either mocking out way too much of the implementation details of SQS (e.g. the message format including an MD5 hash) or having every test run set up its own infrastructure.\nOr maybe a hybrid where there is a testing infrastructure we can deploy lambdas to.<\/p>\n<p>My hunch is that this is a general problem with room to improve on not just in this particular case but in general.\nI also seemed to find that there is curiously little good information on this topic.<\/p>\n<h3>The Frontend<\/h3>\n<p>The frontend is a basic <a href=\"https:\/\/vuejs.org\">Vue<\/a> SPA which uses the HTTP API provided by the lambdas to fetch  data and display it in a basic scatter plot using <a href=\"https:\/\/www.chartjs.org\/\">ChartJS<\/a>.\nAs this is my home turf I wanted to get it over with as quickly as possible, which is why I didn\u2019t bother with making it responsive or even look too pleasing in general.\nI did add some component tests for the sake of it, but nothing too fancy.<\/p>\n<\/div>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/frontend.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"536\" data-attachment-id=\"25693\" data-permalink=\"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2023\/09\/04\/lyrics-analyzer-on-cloud-infrastructure-music-and-things-between\/frontend\/\" data-orig-file=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/frontend.png\" data-orig-size=\"1554,813\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"frontend\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/frontend-1024x536.png\" src=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/frontend-1024x536.png\" alt=\"\" class=\"wp-image-25693\" srcset=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/frontend-1024x536.png 1024w, https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/frontend-300x157.png 300w, https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/frontend-768x402.png 768w, https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/frontend-1536x804.png 1536w, https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/frontend.png 1554w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/a><\/figure>\n\n\n\n<div class=\"wp-block-jetpack-markdown\"><h3>Deployment<\/h3>\n<p>As I mentioned earlier, this project is deployed using the Serverless Framework.\nThis means all infrastructure is defined in a <code class=\"\" data-line=\"\">serverless.yaml<\/code> file, which is then used to deploy the application to AWS via the <code class=\"\" data-line=\"\">sls<\/code> CLI.\nTo be more precise, it\u2019s split up into two <code class=\"\" data-line=\"\">serverless.ts<\/code> files (yes you can use TypeScript for this), one for the backend and one for the frontend, which can be called separately or together via <a href=\"https:\/\/www.serverless.com\/framework\/docs\/guides\/compose\">Serverless Compose<\/a>.<\/p>\n<p>There\u2019s a CI\/CD pipeline running on <a href=\"https:\/\/https:\/\/github.com\/features\/actions\">GitHub Actions<\/a> which is triggered on every push to the <code class=\"\" data-line=\"\">main<\/code> branch and runs the tests, builds the application and deploys it to AWS.\nWhile this is quite nice, it\u2019s not as straightforward or easy as I had hoped.<\/p>\n<p>Serverless only supports CI\/CD if you\u2019re using a YAML file.\nWhat a bummer.<\/p>\n<p>One thing I struggled with <em>majorly<\/em> at the beginning was the fact that I had to transpile the TypeScript code before deploying it.\nAt first I figured I could just use <code class=\"\" data-line=\"\">tsc<\/code> to compile the project via the command line.\nHowever, this proved to be wrong. I mean it works in theory &#8211; but the size of the resulting code is <strong>huge<\/strong>.\nThis seems to be due to the fact that <code class=\"\" data-line=\"\">tsc<\/code> does not do any tree shaking.<\/p>\n<\/div>\n\n\n\n<figure class=\"wp-block-image size-full\"><a href=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/no-biggie.jpg\"><img loading=\"lazy\" decoding=\"async\" width=\"574\" height=\"640\" data-attachment-id=\"25694\" data-permalink=\"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2023\/09\/04\/lyrics-analyzer-on-cloud-infrastructure-music-and-things-between\/no-biggie\/\" data-orig-file=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/no-biggie.jpg\" data-orig-size=\"574,640\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"no-biggie\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/no-biggie.jpg\" src=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/no-biggie.jpg\" alt=\"\" class=\"wp-image-25694\" srcset=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/no-biggie.jpg 574w, https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/no-biggie-269x300.jpg 269w\" sizes=\"auto, (max-width: 574px) 100vw, 574px\" \/><\/a><\/figure>\n\n\n\n<div class=\"wp-block-jetpack-markdown\"><p>There\u2019s a library for anything, right? Right.\nMore than one, in fact.\nOr, in the case of Serverless, there\u2019s plugins.<\/p>\n<p>As I wanted to avoid webpack and the complexity of configuring it that usually accompanies it, I decided to go with <a href=\"https:\/\/esbuild.github.io\/\">esbuild<\/a>, for which a <a href=\"https:\/\/www.serverless.com\/plugins\/serverless-esbuild\/\">plugin for Serverless<\/a> exists.\nAfter some work I had a working setup which would transpile the code and deploy it to AWS.\nFor the backend, that is.<\/p>\n<p>The frontend I meant to deploy to S3, which seems to be the <em>AWS way<\/em> of doing things.\nSo far so good. There\u2019s a <a href=\"https:\/\/www.serverless.com\/plugins\/serverless-finch\">plugin<\/a> for that, obviously.\nOne thing I had to figure out was how to tell the frontend where the API gateway resides as this URL is different for each deployment.\nThis screams environment variables.\nBut how to inject them?\nWell, for the backend which needs a connection to DynamoDB or SNS this was quite easy, as Serverless has a built-in way of doing this.\nHowever, the frontend needs to be deployed after the backend, as it needs to know the URL of the API gateway, and it then needs to have that injected at build time.\nGuess what? There\u2019s a <a href=\"https:\/\/www.serverless.com\/plugins\/serverless-build-client\">plugin<\/a> for that which also conveniently handles the build process for you.\nThen once it\u2019s deployed to S3 you can access it via HTTP.\nBut what about HTTP<strong>S<\/strong>?\nWell, this is also a little more complicated than I had hoped.\nI had used other cloud providers like <a href=\"https:\/\/vercel.com\/\">Vercel<\/a> or <a href=\"https:\/\/www.netlify.com\/\">Netlify<\/a> in the past and they all offer a way to get a TLS certificate for your domain.\nOh, and also a domain. Which is just a subdomain of their own domain, but you take what you can get.<\/p>\n<p>With AWS, however, there is no option that makes hosting a proper website anywhere near as easy.\nThere is <a href=\"https:\/\/aws.amazon.com\/cloudfront\/\">CloudFront<\/a> which is a CDN that can be used to serve static websites, but it doesn\u2019t offer a way to get a domain assigned for your bucket, at least from what I could tell.\nAnd that\u2019s a big L for AWS in my book.\nAnd that\u2019s why I stopped at this point and figured I\u2019ll just serve it over HTTP.\nI\u2019ve had enough.<\/p>\n<\/div>\n\n\n\n<figure class=\"wp-block-embed is-type-rich is-provider-spotify wp-block-embed-spotify wp-embed-aspect-21-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe title=\"Spotify Embed: 8 Iz Enuff\" style=\"border-radius: 12px\" width=\"100%\" height=\"152\" frameborder=\"0\" allowfullscreen allow=\"autoplay; clipboard-write; encrypted-media; fullscreen; picture-in-picture\" loading=\"lazy\" src=\"https:\/\/open.spotify.com\/embed\/track\/5LZ4d69L4TK8hVyh2q7YEJ?utm_source=oembed\"><\/iframe>\n<\/div><\/figure>\n\n\n\n<div class=\"wp-block-jetpack-markdown\"><p>I would like to pause here for a moment.\nWhile this seems <em>easy enough<\/em> and there are a lot of plugins surrounding Serverless that do their job well it took me quite a while to figure out how to get all this to work.\nFar more time than I would\u2019ve spent on this if I had just used a more traditional approach.\nSlap it into a Docker image, deploy it to a Kubernetes cluster or even just a simple server, done.\nAnd while I\u2019m sure it gets easier as you get more familiar with the tools, I can\u2019t help but feel that something is a little\u2026 off.\nThis becomes more evident when looking at the development process.<\/p>\n<p>Using all these cloud services &#8211; how do you develop locally?\nWell, you don\u2019t.\nOr at least I didn\u2019t.\nWhile there exist ways to emulate these services locally, none of them worked out of the box for me.\nThen there\u2019s also limitations around some of the services you emulate, which means that you can never be sure that something which works locally will also work in production.\nOr, more likely, the other way around.\nSo you just develop locally and then you kind of deploy your code and hope for the best.\nHope, because you\u2019ll be waiting about a minute for your changes to be deployed to AWS.\nEvery. Single. Time.\nI know that there\u2019s ways to speed this up, but in comparison to the development experience I\u2019m used to where \u201cdeploying\u201d means saving the file, this is orders of magnitude slower.\nSeriously, this is the antithesis of fun.<\/p>\n<p>On the other hand, this forces you to be thorough with your unit testing for the sake of your own sanity which is a good thing in some twisted sense.<\/p>\n<p>A thing that must be said for this type of development process is that I would imagine it to be quite nice for teams as you don\u2019t have to go through the process of setting up a complete environment on your machine and you can just start to deploy lambda functions which neatly tie into the rest of the application.\nSomething I didn\u2019t get to really try out was the ability to deploy to multiple stages, e.g. a staging and a production environment, as I didn\u2019t want to exceed my free tier limits and also didn\u2019t see the need to go quite that far.<\/p>\n<p>On a side note &#8211; every lambda function that isn\u2019t invoked for a certain period of time needs some time to start before it can run.\nAnd while this shouldn\u2019t be an issue when you have users using basically all of your lambdas every now and then (psst\u2026 or you use another <a href=\"https:\/\/www.serverless.com\/plugins\/serverless-plugin-warmup\">plugin<\/a>) it\u2019s still something I found to be annoying from time to time.<\/p>\n<\/div>\n\n\n\n<h3 class=\"wp-block-heading\">Error Handling<\/h3>\n\n\n\n<figure class=\"wp-block-embed is-type-rich is-provider-spotify wp-block-embed-spotify wp-embed-aspect-21-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe title=\"Spotify Embed: Shit Can Happen\" style=\"border-radius: 12px\" width=\"100%\" height=\"152\" frameborder=\"0\" allowfullscreen allow=\"autoplay; clipboard-write; encrypted-media; fullscreen; picture-in-picture\" loading=\"lazy\" src=\"https:\/\/open.spotify.com\/embed\/track\/4k26Z0ncaGQ6JisQblsCom?utm_source=oembed\"><\/iframe>\n<\/div><\/figure>\n\n\n\n<div class=\"wp-block-jetpack-markdown\"><p>Shit happens. We can try to prevent it, but it will happen.\nAnd when it does hit the fan(out architecture) we\u2019d like to at least know about it.<\/p>\n<p>Ok sorry for that joke. Anyways.<\/p>\n<p>Since all functions run in lambdas triggered via HTTP calls or SQS subscriptions and both of these have a dedicated error handling middleware in <a href=\"https:\/\/middy.js.org\">Middy<\/a> which I\u2019m using already for various other things like input validation, setting headers, etc. we get a neat way to automatically handle errors in our lambda functions.\nThis only supposes letting these errors \u201cbubble up\u201d the call chain.\nWhich is quite convenient as that\u2019s what happens anyways when we <em>don\u2019t<\/em> handle any errors.<\/p>\n<p>There is a point to be made here about the difference between recoverable and disastrous errors.\nFor recoverable errors, we would like to return a semantically relevant error to the caller which might have been a user triggering an invalid operation.\nFor disastrous errors we probably need to abort everything we\u2019re doing and restart the application, in a more traditional setting that is.\nIn a serverless setting, we can simply let the lambda function fail and let the infrastructure handle it.\nAs every lambda only runs a use case that either fails or succeeds as a whole we don\u2019t have to concern ourselves with restarting our entire application as there is no application to restart in this sense. However, we might have to rollback or invalidate transactions if we want something akin to transactional atomicity and consistency.\nA nice thing about lambdas is that they have a built-in retry mechanism which will retry the operation a certain amount of times before giving up, giving us some lee-way to handle errors that might be recoverable after all, e.g. a network error or a throttling error.<\/p>\n<p>To be fair &#8211; I don\u2019t have any proper error handling in place at the moment.\nErrors are simply logged to CloudWatch via the middleware and that\u2019s it.\nWhile this was enough for debugging purposes during development, it\u2019s way undersized for production code which should use some of the aforementioned concepts to handle errors in a more graceful fashion.\nThe frontend doesn\u2019t respond properly to errors either, but that\u2019s okay in my opinion as my focus was on the backend and I simply didn\u2019t want to bother with these types of bells and whistles.<\/p>\n<p>Another side note on an error I only encountered later on &#8211; with DynamoDB I quickly got to the point where I was hitting the capacity limits and was throttled. This effectively means that the operation will fail.\nUsually this happened due to me refreshing the page too often, which would fetch all artists from the database.\nIncreasing the limits of course worked, but I was a little surprised to see me hitting the limits of free tier even though I was only using the application myself and there was no way for me to reduce the size of the items I was fetching any further.<\/p>\n<h3>Dependency Injection<\/h3>\n<p>For this I used <a href=\"https:\/\/github.com\/jeffijoe\/awilix\">Awilix<\/a> which I wasn\u2019t familiar with.\nI chose to use it in the simplest way possible &#8211; have a single file that imports all injectable dependencies and then use it to create a container which is then used to resolve dependencies.\nThis happens automatically in the use cases themselves and manually in the lambda handlers to fetch these use cases by using a wrapper function to put the container into the event\u2019s context, which I found to be a rather clean implementation that keeps the handlers from importing the container directly.<\/p>\n<pre><code class=\"language-typescript\" data-line=\"\">const container = createContainer&lt;Cradle&gt;({\n  injectionMode: InjectionMode.CLASSIC,\n});\ncontainer.register({\n  \/\/ Environment variables\n  geniusAccessToken: asValue(getEnv(&#039;GENIUS_ACCESS_TOKEN&#039;)),\n  artistTableName: asValue(getEnv(&#039;ARTIST_TABLE_NAME&#039;)),\n  integrationEventTopicArn: asValue(getEnv(&#039;INTEGRATION_EVENT_TOPIC_ARN&#039;)),\n  geniusBaseUrl: asValue(&#039;https:\/\/api.genius.com&#039;),\n  \/\/ Models\n  artistModel: asFunction(getArtistModel).singleton(),\n  \/\/ Mappers\n  artistMapper: asClass(ArtistMapper),\n  \/\/ Factories\n  artistFactory: asClass(ConcreteArtistFactory),\n  \/\/ Repositories\n  artistRepository: asClass(DynamooseArtistRepository),\n  \/\/ Services\n  statisticsCalculator: asClass(ConcreteStatisticsCalculator),\n  \/\/ ...\n});\n<\/code><\/pre>\n<pre><code class=\"language-typescript\" data-line=\"\">export type DependencyAwareContext&lt;C extends Context = Context&gt; = C &amp; {container: AwilixContainer&lt;Cradle&gt;};\ntype DependencyMiddleware&lt;\n  Event = unknown,\n  Result = any,\n  E = Error,\n&gt; = () =&gt; middy.MiddlewareObj&lt;Event, Result, E, DependencyAwareContext&gt;;\nexport const withDependencies: DependencyMiddleware = () =&gt; ({\n  before(request) {\n    const {context} = request;\n    context.container = container.createScope();\n  },\n});\n<\/code><\/pre>\n<pre><code class=\"language-typescript\" data-line=\"\">const handler: ValidatedEventAPIGatewayProxyEvent&lt;FromSchema&lt;typeof schema&gt;&gt; = async (event, context) =&gt; {\n  const {triggerWorkflowUseCase} = context.container.cradle;\n  await triggerWorkflowUseCase.execute(event.body.artistId);\n  return formatJSONResponse({\n    message: `Lyrics for artist ${event.body.artistId} are being analyzed`,\n  });\n};\n<\/code><\/pre>\n<p>While this works quite well for a project of this scope I\u2019d imagine it to become cumbersome as the project grows.\nAlso, this majorly hinders tree-shaking as the container imports all <em>possible<\/em> dependencies, even when they\u2019re not needed.\nI\u2019m sure there\u2019s a better way to go about this and construct a proper dependency graph that allows pruning unneeded dependencies. Maybe by having classes register themselves dynamically when being imported which I think is a possibility that Awilix offers.\nI still stuck with this though as it did work and I didn\u2019t feel like rewriting some part <em>again<\/em>.<\/p>\n<h3>Configuration<\/h3>\n<p>In general we need to distinguish between two groups here.\nThere\u2019s general configuration like the API endpoint or the name of the DynamoDB table.\nThese we can usually easily handle via environment variables.\nAnd then there\u2019s what might be called <strong>secrets<\/strong> which are configuration values we want to hide from the public eye.\nThis usually includes passwords or tokens which are not a direct result of setting up the infrastructure, which is why Serverless\u2019 configuration can\u2019t really help us here.<\/p>\n<p>In my case access to AWS infrastructure is handled by roles which Serverless manages on its own.\nWe do, however, have to define the exact permissions ourselves which requires at least some knowledge of the concepts of roles and policies in the context of AWS.\nHere\u2019s what that looks like:<\/p>\n<pre><code class=\"\" data-line=\"\">const serverlessConfiguration = {\n  \/\/ ...\n  iam: {\n    role: {\n      statements: [\n        {\n          Effect: &#039;Allow&#039;,\n          Action: [&#039;dynamodb:PutItem&#039;, &#039;dynamodb:Get*&#039;, &#039;dynamodb:Scan*&#039;, &#039;dynamodb:UpdateItem&#039;, &#039;dynamodb:DeleteItem&#039;],\n          Resource: &#039;arn:aws:dynamodb:${aws:region}:${aws:accountId}:table\/${self:service}-*-${sls:stage}&#039;,\n        },\n        \/\/ ...\n      ]\n    }\n  }\n  \/\/ ...\n}\n<\/code><\/pre>\n<p>These variables we can simply inject in our Serverless configuration file.\nThis means we only have a single configuration value, the API token for the Genius API, that needs to be injected not by Serverless itself but via some interaction on part of the developer.\nAnd here\u2019s what that looks like:<\/p>\n<pre><code class=\"\" data-line=\"\">const serverlessConfiguration = {\n  \/\/ ...\n  environment: {\n    AWS_NODEJS_CONNECTION_REUSE_ENABLED: &#039;1&#039;,\n    NODE_OPTIONS: &#039;--enable-source-maps --stack-trace-limit=1000&#039;,\n    GENIUS_ACCESS_TOKEN: &#039;${env:GENIUS_ACCESS_TOKEN}&#039;,\n    ARTIST_TABLE_NAME: &#039;${self:service}-Artist-${sls:stage}&#039;,\n    INTEGRATION_EVENT_TOPIC_ARN: {\n      Ref: &#039;IntegrationEventTopic&#039;,\n    },\n  },\n  \/\/ ...\n}\n<\/code><\/pre>\n<p>While there are cloud solutions for this, which <em>are<\/em> nice for bigger projects, they usually require more setup than plain environment variables.\nThat\u2019s why I opted against using a separate service for this and instead simply inject them as environment variables.\nFor local development there\u2019s an <code class=\"\" data-line=\"\">.env<\/code> file in the repository that has to be populated by the developer.\nFor automated deployment there\u2019s a neat feature built-in to Github Actions which injects environment variables into the CI\/CD pipeline after creating them manually via the website.<\/p>\n<\/div>\n\n\n\n<h2 class=\"wp-block-heading\">The Missing Parts<\/h2>\n\n\n\n<figure class=\"wp-block-embed is-type-rich is-provider-spotify wp-block-embed-spotify wp-embed-aspect-21-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe title=\"Spotify Embed: More Gifts\" style=\"border-radius: 12px\" width=\"100%\" height=\"152\" frameborder=\"0\" allowfullscreen allow=\"autoplay; clipboard-write; encrypted-media; fullscreen; picture-in-picture\" loading=\"lazy\" src=\"https:\/\/open.spotify.com\/embed\/track\/0YNGu4AWaO2jhByIEC4B55?utm_source=oembed\"><\/iframe>\n<\/div><\/figure>\n\n\n\n<p>While I&#8217;m quite happy with the result of this project, it&#8217;s far from being finished.<br>However, finishing was never the goal in the first place.<br>The goal was to learn, mostly about DDD as well as the cloud services I used.<br>That&#8217;s why the frontend especially is lacking &#8211; it doesn&#8217;t look good or behave anywhere near sane on mobile devices as I already know CSS rather well and I didn&#8217;t want to spend any time on that.<br>All in all, I kind of just stopped at the points where I had the feeling that I&#8217;m only implementing things I already know as opposed to learning something new.<\/p>\n\n\n\n<p>Another side note &#8211; while DDD <em>seemed<\/em> to favor a classical OOP approach at first, and I did write the project in an OOP fashion simply due to the fact that I hadn&#8217;t really done OOP in some time, I get the feeling that it would work quite well with a functional approach as well.<\/p>\n\n\n\n<p>Have a look at this picture, which tries to give an overview over the typical structure of a DDD application:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><a href=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/onion.png\"><img loading=\"lazy\" decoding=\"async\" width=\"800\" height=\"800\" data-attachment-id=\"25695\" data-permalink=\"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2023\/09\/04\/lyrics-analyzer-on-cloud-infrastructure-music-and-things-between\/onion\/\" data-orig-file=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/onion.png\" data-orig-size=\"800,800\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"onion\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/onion.png\" src=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/onion.png\" alt=\"\" class=\"wp-image-25695\" srcset=\"https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/onion.png 800w, https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/onion-300x300.png 300w, https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/onion-150x150.png 150w, https:\/\/blog.mi.hdm-stuttgart.de\/wp-content\/uploads\/2023\/09\/onion-768x768.png 768w\" sizes=\"auto, (max-width: 800px) 100vw, 800px\" \/><\/a><\/figure>\n\n\n\n<div class=\"wp-block-jetpack-markdown\"><p>Now, unfortunately I didn\u2019t find a nice image to showcase the similarity to functional programming so let me try to put it in words.\nWhen using a functional style, it\u2019s a common approach to have a core of pure functions which are then wrapped in impure functions that handle side effects.\nSeems familiar?\nThat\u2019s because it\u2019s exactly the thing DDD tries to achieve, only looked at from a different perspective.\nBy having the domain logic not concern itself with any impurity like state, HTTP calls, database connections, etc. we can focus entirely on the actual thing we\u2019re trying to solve.<\/p>\n<p>As this project is rather small in terms of domain logic, there\u2019s not a lot of code that could be considered pure.\nThere\u2019s a lot of glue code that connects the inner parts to the impure outside world &#8211; however, now that we have this glue in place, I suspect that as we increase the domain (we could for example start analyzing playlists instead of artists, etc.) we could concern ourselves more with the inner parts and less with the glue.<\/p>\n<h2>Wrap Up<\/h2>\n<p>Quite a lot of text up until here, so let\u2019s keep it short.<\/p>\n<ul>\n<li>DDD is nice, albeit a little verbose. Good for bigger applications I\u2019d guess but definitely over the top for something so simple. I\u2019ll definitely consider it in the future though and I\u2019m glad to have worked with it as it\u2019s less of a concrete pattern and more of a way of thinking about the problem at hand.<\/li>\n<li>The Serverless Framework is also nice, but far more work to setup than I had anticipated. I\u2019m also unsure how the configuration scales as the project grows. Also, I\u2019m missing some convenience features like proper typing or input validation for lambdas out of the box. It still seems to be one of the better alternatives though.<\/li>\n<li>Policies are\u2026 needed, I know. I still hate them. Fortunately Serverless abstracts a lot of this away, which is one of the biggest selling points for me.<\/li>\n<li>Coming from a more traditional development background, I found the development process to be quite cumbersome. I feel like this is an area that could be improved upon, partly in your own projects but probably the most by the cloud providers themselves.<\/li>\n<li>I want first class Typescript support and I want it yesterday.<\/li>\n<li>Unit testing is nice. I should do it more often.<\/li>\n<li>Integration testing is hard. I should do it more often.<\/li>\n<li>Writing is actually kind of fun. I should do it more often.<\/li>\n<\/ul>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>After several unsuccessful attempts to weld my results together into such a whole, I realized that I should never succeed. The best that I could write would never be more than philosophical remarks; my thoughts were soon crippled if I tried to force them on in any single direction against their natural inclination. [\u2026]I should [&hellip;]<\/p>\n","protected":false},"author":1161,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1],"tags":[],"ppma_author":[948],"class_list":["post-25686","post","type-post","status-publish","format-standard","hentry","category-allgemein"],"aioseo_notices":[],"jetpack_featured_media_url":"","jetpack-related-posts":[],"jetpack_sharing_enabled":true,"authors":[{"term_id":948,"user_id":1161,"is_guest":0,"slug":"chris-robin_ennen","display_name":"Chris-Robin Ennen","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/9ac1631ae74f0f5b963d589c29d8e4e64ee97bd2438862125bf28769c0e5dcc8?s=96&d=mm&r=g","0":null,"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/posts\/25686","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/users\/1161"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/comments?post=25686"}],"version-history":[{"count":6,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/posts\/25686\/revisions"}],"predecessor-version":[{"id":25701,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/posts\/25686\/revisions\/25701"}],"wp:attachment":[{"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/media?parent=25686"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/categories?post=25686"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/tags?post=25686"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/wp-json\/wp\/v2\/ppma_author?post=25686"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}