{"id":26422,"date":"2024-09-03T19:46:42","date_gmt":"2024-09-03T17:46:42","guid":{"rendered":"https:\/\/blog.mi.hdm-stuttgart.de\/?p=26422"},"modified":"2024-09-03T19:46:45","modified_gmt":"2024-09-03T17:46:45","slug":"how-to-not-get-rich-using-tiktok-and-aws","status":"publish","type":"post","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2024\/09\/03\/how-to-not-get-rich-using-tiktok-and-aws\/","title":{"rendered":"How to (not) get rich using TikTok and AWS"},"content":{"rendered":"\n

In our Software Development for Cloud Computing course, we were tasked with building something, anything, using the cloud. After various discarded ideas, our team decided to create a platform aimed at automatically generating short-form videos. In this blog post, we will go over our development journey, the architectural decisions we made, the technologies we chose (and why), the challenges we encountered and the (many) lessons we learned.<\/p>\n\n\n\n

Introduction<\/h2>\n\n\n\n

Short-form video content has enjoyed a massive rise in popularity in recent years, with platforms like YouTube and Instagram, inspired by TikTok, building their own versions of an algorithmically generated feed of short looping videos.<\/p>\n\n\n\n

As avid consumers of short-form videos ourselves, we chose to focus on this topic for our project. We found a common pattern amongst the videos we watched, where a text-to-speech (TTS) voice would read a reddit story on top of some kind of background video.<\/p>\n\n\n\n

What if we could automatically generate these types of videos, upload them to various platforms using their APIs, and use the creator monetization of the platforms to get rich? All we would need to do is find stories, narrate them using TTS, and generate an accompanying video. Our plan was foolproof!<\/p>\n\n\n\n

We didn\u2019t quite end up fully realizing our vision (or we\u2019d be sitting on a yacht instead of writing this blog post), but here\u2019s what we came up with:<\/p>\n\n\n\n

\"\"<\/a>
You\u2019ll understand all of this shortly.<\/figcaption><\/figure>\n\n\n\n

How does it work?<\/h2>\n\n\n\n

Let\u2019s walk through each of the steps a user needs to make in order to generate a finished video.<\/p>\n\n\n\n

Getting a story<\/h3>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n

Our initial goal was to automatically scrape stories from Reddit and use them in our videos. Unfortunately, Reddit has been fighting against this type of use<\/a> and their API is increasingly difficult for developers to use. Instead, we decided to automatically generate stories using OpenAI\u2019s ChatGPT 4o model.<\/p>\n\n\n\n

\n

After experimentation with various prompts, we started getting good results with this one:<\/p>\n\n\n\n

You’re a bot for the project TikTok Money Glitch.<\/p>\n\n\n\n

Your goal is to generate text for a TikTok video, given a prompt.<\/p>\n\n\n\n

The videos are in the style of reddit posts.<\/p>\n\n\n\n

Your goal is to generate content that is in some way interesting and would fit for a TikTok audience.<\/p>\n\n\n\n

You can either generate a Reddit post’s body, or one or multiple replies to a Reddit post.<\/p>\n\n\n\n

If you are generating a post, only generate the body of the post. Don’t include the title anywhere and don’t use Markdown.<\/p>\n\n\n\n

If you are generating replies, include the title without any formatting. Then, generate the body of the replies without any formatting.<\/p>\n\n\n\n

If your prompt is invalid, reply with I cannot generate a post from this prompt<\/p>\n<\/blockquote>\n\n\n\n

Users tell the AI what kind of story they would like, at which point the story input will get automatically filled with a story that the user can edit if they would like to make any adjustments. We\u2019re using Next.js\u2019 streaming feature to load the generated text into user\u2019s browsers as it’s being generated, similar to the UI for tools like ChatGPT.<\/p>\n\n\n\n

Background Video<\/h3>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n

The videos we aim to generate are characterized by video game footage in the background, typically featuring Minecraft Parkour or Subway Surfers. Users can upload this type of footage through a dedicated backgrounds<\/code> route, where they can manage their library of background videos – uploading new footage or deleting old clips as needed.<\/p>\n\n\n\n

Once users have added their desired videos to their library, they can access these videos during the video creation process. Here, they can select the background video they want to pair with their story.<\/p>\n\n\n\n

Preview<\/h3>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n

After a story and background video are provided, a preview is generated following a brief loading time. Behind the scenes, a Next.js API route handler on \/api\/lambda\/polly<\/code>  is triggered. This handler invokes two AWS Lambda functions: one for generating the audio and another for creating the speech marks from the provided story.<\/p>\n\n\n\n

The Polly responses are returned as Audio Stream objects. The audio stream is directly stored as a blob in an S3 bucket, while the speech marks stream is parsed into a JSON object array. Each object in the array represents one word with metadata, including start and end times, necessary for displaying subtitles.<\/p>\n\n\n\n

At this point all necessary parts have been collected: audio, speech marks, text, and background video. We use Remotion, a tool for programmatically generating video, to turn all these parts into a single video. Remotion has a feature called Remotion Player, which we use to preview the video to our users before continuing to the next step. At this point the finished video can already be viewed, although it\u2019s not actually a video yet. Instead, it\u2019s a collection of HTML-elements just being animated to look like a video.<\/p>\n\n\n\n

Rendering<\/h3>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n

If the user is happy with the preview, it can be turned into an actual encoded video. Remotion supports distributing these rendering workloads on AWS lambda using @remotion\/lambda<\/code>. Each lambda gets a configurable amount of frames to render (in our case 10), and after all the tasks have completed, the frames get merged together into one video, which is uploaded to S3.<\/p>\n\n\n\n

Publishing<\/h3>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n

Finally, in the publishing step, users have the choice of either simply downloading the video directly from the S3 bucket with a pre-signed URL, or automatically uploading it to YouTube. Here, we use a Next.js server action that lives entirely on the server side, communicates with Clerk, gets the user\u2019s access token and uploads the video directly to the signed in Google account as a YouTube short via a call to the YouTube API. This is all based on the use of an extended OAuth provider in Clerk, which initiates the necessary Scopes to upload videos.<\/p>\n\n\n\n

\"\"<\/a>
An overview of our authentication flows<\/figcaption><\/figure>\n\n\n\n

Technology Stack<\/h2>\n\n\n\n

We\u2019d like to spend some time explaining how we made our tech stack choices. We considered several factors while making these decisions like our personal experience, popularity, and fit for the project.<\/p>\n\n\n\n

Rendering Videos: Remotion (and its consequences)<\/h3>\n\n\n\n

Generating videos was the most central part of our project, and we decided to go with Remotion for it. Remotion can generate videos using templates defined in code (more specifically React). As there is not a lot of competition in this field and Remotion is free to use at our scale, it was an obvious choice. <\/p>\n\n\n\n

Because Remotion has direct support for rendering videos on AWS Lambda concurrently (which is faster and more efficient than using a VPS), this locked us in to using AWS as well.<\/p>\n\n\n\n

And due to Remotion being React-based, it was an obvious choice to use a React-based framework for the rest of our frontend as well.<\/p>\n\n\n\n

This tight integration between Remotion and the rest of our stack made development far quicker and easier, but also came with some disadvantages. If we hadn\u2019t chosen Remotion, we could\u2019ve tried out a framework based on a different library (like Vue or Svelte), but as this would\u2019ve led to us separating our video rendering and application code, we decided to stick with React. We probably would have also decided to use AWS as our cloud provider without Remotion, but we did not spend much time evaluating the alternatives due to Remotion only supporting AWS.<\/p>\n\n\n\n

Framework Choice: Next.js Ecosystem<\/h3>\n\n\n\n

The two most popular React-based frameworks are Next.js and Remix. They have a similar feature-set with both of them supporting server-side rendering, file-based routing and having integrations for SST.<\/p>\n\n\n\n

We ended up choosing Next.js ecosystem primarily because our team was already familiar with React from previous lectures and projects. This allowed us to use our existing knowledge, making our development process faster, with the benefit of letting us focus more on the actual cloud infrastructure and less on building the web app. Remotion also provides templates for integrating with Next.js which made starting out far quicker.<\/p>\n\n\n\n

Authentication: AWS Cognito vs NextAuth vs Clerk <\/h3>\n\n\n\n

As we wanted to use AWS for everything, our initial thought was to use AWS Cognito for authentication. In initial testing, we struggled with this though, and it quickly became clear that this would require significant effort on our part to get working. As authentication is not a central part of our app, we wanted to minimize the time spent working on it.<\/p>\n\n\n\n

Instead, we considered using NextAuth, as we had also used this in previous projects. In those projects we had a fair share of issues with it though, and had actually considered switching those projects to use Clerk instead. As we were starting from scratch, we decided this was the way to go.<\/p>\n\n\n\n

Clerk is an auth-as-a-service platform that has specific integrations for Next.js which made it extremely simple to set up for our use case. Later on, we needed our users to grant us access to their YouTube accounts. Clerk allowed us to integrate this into their authentication flow by changing a single setting in their admin portal, which reduced the time we had to work on the YouTube integration significantly.<\/p>\n\n\n\n

Database: DynamoDB vs AWS RDS with Drizzle ORM<\/h3>\n\n\n\n

The SST documentation has a tutorial<\/a> for integrating AWS RDS with SST while using the Drizzle ORM to talk to the database. As we wanted to try out Drizzle, we tested this as an option. Despite following the tutorial, we never actually got the integration between the database and Drizzle to work. As discussed later on, we also ran into some very significant cost issues though, due to the very high price of AWS RDS.<\/p>\n\n\n\n

As we still needed a database and didn\u2019t want to have a repeat of the cost issue, we chose DynamoDB, primarily due to its simplicity and low cost. We quickly got set up with it and had no further issues, so we stayed with it for the rest of the project.<\/p>\n\n\n\n

E2E-Testing Framework: Cypress vs Playwright<\/h3>\n\n\n\n

For E2E-Testing frameworks there are two popular options that integrate well with our stack: Cypress and Playwright<\/p>\n\n\n\n

Clerk\u2019s Cypress integration has several helper methods which can authenticate the test client by calling a method with provided credentials. Unfortunately, Playwright lacks such an integration.<\/p>\n\n\n\n

However, as a group, we had no experience using Cypress, and Playwright had also been described as a more modern alternative to it. We decided to go ahead with Playwright\u2019s weaker integration, which forced us to manually write the authentication part of our tests. We ended up finding an example<\/a> from Clerk that helped with this though and had no further struggles getting our E2E tests to work together with Clerk.<\/p>\n\n\n\n

We tried setting up our continuous integration to run our E2E-tests automatically, but this turned out to be too challenging due to the required infrastructure needed in the background for our app to work. Instead, we chose to run our E2E-tests manually to validate that our app still worked as intended when refactoring parts behind the scenes.<\/p>\n\n\n\n

Storage: AWS S3 for Video Storage<\/h3>\n\n\n\n

We needed a way to store files for various parts of the app, and as we had been using AWS for most parts of the app, S3 was the obvious choice.<\/p>\n\n\n\n

More specifically, we used S3 for three different parts of our app:<\/p>\n\n\n\n

    \n
  1. We needed to store audio generated from Polly so that it could be accessed in the browser. An automatic expiration date for speech audio is set to avoid cluttering storage, because speech audio is generated during video creation and does not need to be accessed at a later point.<\/li>\n\n\n\n
  2. Users are able to upload background videos which can be used for the generated videos. In addition to uploading the videos to S3, we also store metadata for those files in DynamoDB, with the filename (a randomly generated UUID) linking the file and metadata. This enhances user experience by sorting the way data is viewed in the web app and making files searchable.<\/li>\n\n\n\n
  3. The generated videos are uploaded to an S3 bucket by remotion, at which point they can be downloaded by the user or uploaded directly to YouTube.<\/li>\n<\/ol>\n\n\n\n

    How to get our stuff into the Cloud?<\/h2>\n\n\n\n
    \"\"<\/a>
    Our code is here (xkcd 908<\/a>, licensed under CC BY-NC 2.5<\/a>)<\/figcaption><\/figure>\n\n\n\n

    Deploying our app to the cloud manually would be quite complicated and unpleasant. Ideally, we should use an Infrastructure-as-Code (IaC) tool like Terraform or Pulumi to automate the process and make it repeatable. Since we\u2019re using Next.js, it would be ideal to deploy it in a serverless way, as this is cheaper and can be faster.<\/p>\n\n\n\n

    Vercel, the creators of Next.js, offer a service which hosts Next.js apps using AWS Lambda behind the scenes. We had good experiences with using them in the past, but as Vercel does not work properly with IaC and is designed to work with no additional configuration, using it would\u2019ve gone against the spirit of learning about the cloud in this lecture.<\/p>\n\n\n\n

    AWS offers a service called Amplify which is similar to Vercel, so we decided to test it out with our app. We ran into some compatibility issues though, and our goal was still to deploy \u201ccloser to the metal\u201d than with a service like Vercel. As Amplify is a very close copy of Vercel, this wouldn\u2019t work either.<\/p>\n\n\n\n

    Open Next is an open-source project that turns Next.js\u2019 build output into a format that can be deployed on AWS Lambda. This sounded like it would work great for our use case, so we started exploring its documentation. Their Get Started<\/em> section mentions that \u201cThe easiest way to deploy OpenNext to AWS is with SST\u201d so we looked into that.<\/p>\n\n\n\n

    We were immediately surprised by the simplicity of the example on their landing page.<\/p>\n\n\n\n

    \"\"<\/a>
    I\u2019m sold<\/figcaption><\/figure>\n\n\n\n

    After some testing with SST, we quickly made the decision to use it for this project. Let\u2019s take a look at how it works, why we made the decision to use it, and what our experience with it was like during the project.<\/p>\n\n\n\n

    Infrastructure Management with SST<\/h2>\n\n\n\n

    SST is an IaC tool, similar to Terraform, Pulumi, and CloudFormation. Infrastructure can be provisioned by creating various objects in TypeScript (called components<\/em> by SST). Unlike those tools though, the components in SST abstract the lower-level infrastructure primitives to a level that is much nicer to work with.<\/p>\n\n\n\n

    Why not Pulumi\/Terraform<\/h3>\n\n\n\n

    Pulumi and Terraform provide many different low-level primitives for infrastructure. Unfortunately, these are often too low-level to comfortably work with.<\/p>\n\n\n\n

    As an example, this is the amount of code required to create an AWS Lambda function using Pulumi:<\/p>\n\n\n\n

    import * as pulumi from "@pulumi\/pulumi";\nimport * as archive from "@pulumi\/archive";\nimport * as aws from "@pulumi\/aws";\n\nconst assumeRole = aws.iam.getPolicyDocument({\n    statements: [{\n        effect: "Allow",\n        principals: [{\n            type: "Service",\n            identifiers: ["lambda.amazonaws.com"],\n        }],\n        actions: ["sts:AssumeRole"],\n    }],\n});\nconst iamForLambda = new aws.iam.Role("iam_for_lambda", {\n    name: "iam_for_lambda",\n    assumeRolePolicy: assumeRole.then(assumeRole => assumeRole.json),\n});\nconst lambda = archive.getFile({\n    type: "zip",\n    sourceFile: "lambda.js",\n    outputPath: "lambda_function_payload.zip",\n});\nconst testLambda = new aws.lambda.Function("test_lambda", {\n    code: new pulumi.asset.FileArchive("lambda_function_payload.zip"),\n    name: "lambda_function_name",\n    role: iamForLambda.arn,\n    handler: "index.test",\n    sourceCodeHash: lambda.then(lambda => lambda.outputBase64sha256),\n    runtime: aws.lambda.Runtime.NodeJS18dX,\n    environment: {\n        variables: {\n            foo: "bar",\n        },\n    },\n});<\/code><\/pre>\n\n\n\n
    That\u2019s a lot<\/p>\n\n\n\n
    The tasks of setting IAM Permissions, creating an archive with the code, and setting it up to be run all have to be done in separate steps. And this example doesn\u2019t even contain potentially having to compile code from TypeScript to Javascript.<\/p>\n\n\n\n
    Once infrastructure has been provisioned using a regular IaC tool, actually interacting with it from code is not a great experience either. Infrastructure also can\u2019t be easily duplicated, in case multiple developers want to work on a project simultaneously. And using these primitives to deploy something like a Next.js app is difficult, even with a module that integrates with OpenNext<\/a>.<\/p>\n\n\n\n
    What makes SST so nice?<\/h3>\n\n\n\n
    SST is different. Deploying a Next.js app is as simple as using the sst.aws.Nextjs component. That infrastructure is deployed separately for every developer working with the app, and it can access other infrastructure that is linked to it automatically. Behind the scenes, SST is actually based on Pulumi which means that it\u2019s still possible to use the low-level primitives it provides, if necessary.<\/p>\n\n\n\n
    SST Components<\/h4>\n\n\n\n
    \"\"<\/a><\/figure>\n\n\n\n
    SST has built-in support for components from various cloud providers, but we only used the AWS-specific ones. Specifically, we used Bucket<\/code> (creates an S3 Bucket), Dynamo<\/code> (sets up a DynamoDB behind the scenes), Function<\/code> (creates a serverless AWS Lambda function), Secret<\/code> (synchronizes secrets across the app), and of course Nextjs<\/code> (uses Open Next to deploy a Next.js app on AWS Lambda). As mentioned, these components require much less boilerplate. For example, let\u2019s try to create an AWS Lambda function like the previous example, but using SST:<\/p>\n\n\n\n
    new sst.aws.Function("MyFunction", {\n  handler: "src\/lambda.handler"\n});<\/code><\/pre>\n\n\n\n
    That\u2019s a lot less, eh?<\/p>\n\n\n\n
    All the previous boilerplate necessary to set the correct permissions, compile TypeScript to JavaScript zip the result, and upload it is automatically handled behind the scenes. Because of the unique name generated for each component deployment, it\u2019s not possible for two developers to have conflicting deployments either.<\/p>\n\n\n\n
    The community can also make their own components, which was very helpful in the case of Remotion SST<\/a>. The Remotion documentation only explains how to manually deploy and update the AWS Lambda function responsible for rendering out videos. This can cause issues when Remotion versions are out of sync between the Lambda and App, and is also just annoying to do manually. Remotion SST can be used as a component like any other component and handles the deployment and updating of the required Lambda automatically.<\/p>\n\n\n\n
    SST Linking<\/h4>\n\n\n\n
    When two different components are intended to work together, like a Next.js app accessing an S3 bucket, they can be linked together.<\/p>\n\n\n\n
    const bucket = new sst.aws.Bucket("MyBucket");\n\nnew sst.aws.Nextjs("MyWeb", {\n\u00a0\u00a0link: [bucket],\n});<\/code><\/pre>\n\n\n\n
    Internally, SST will modify the IAM permissions of the bucket so that the Next.js app can interact with it without needing any further authentication. At the same time, the SST SDK is notified about the resource, which means that it can be imported using the name defined in the SST config.<\/p>\n\n\n\n
    import { Resource } from "sst";\n\nconsole.log(Resource.MyBucket.name);<\/code><\/pre>\n\n\n\n
    There\u2019s full type-safety for this (very nice)<\/p>\n\n\n\n
    Not having to think about authenticating different services with each other and instead things \u201cjust working\u201d made our development experience significantly better. As Remotion SST from earlier also supports linking, we no longer had to worry about setting the correct permissions for remotion either (something we struggled with previously).<\/p>\n\n\n\n
    SST Live<\/h4>\n\n\n\n
    One of the standout features of SST is SST Live. It makes it possible to test changes made to SST Components almost instantly, without having to redeploy the entire application. <\/p>\n\n\n\n
    SST Live uses IoT over WebSocket to bridge communication between a developer\u2019s local machine and the remote Lambda function. When starting the app with sst de<\/code>v, SST swaps out the standard Lambda functions with stub versions. At the same time, a WebSocket client on the developer\u2019s machine connects to their AWS accounts\u2019 IoT endpoint.<\/p>\n\n\n\n
    When a Lambda function is triggered, it sends an event with the request data. This event is then captured by the local WebSocket client, which then processes the function locally using a Node.js Worker. The result is finally sent back as an event, which the stub Lambda receives and returns as the function\u2019s response.<\/p>\n\n\n\n
    All this setup may seem complicated, but it happens automatically in the background, and has several big advantages:<\/p>\n\n\n\n