Cost-Efficient Server Structure: Merging Static and Dynamic API

While developing our guessing game, “More or Less”, we found a method to significantly reduce traffic on our serverless API, leading to cost savings and an improved content creation experience.

The Problem

In our game, players can contribute their own game modes, using the web editor.

Additionally, we develop game modes ourselves to guarantee high-quality content for trendings topics. These own created game modes are featured on the discover pages and played a lot, subsequently leading to increased costs in API usage.

So generally spoken:

Websites usually have featured content or items on the landing page that get viewed many times, leading to higher costs in the API.
This content is often created by the website owner rather than coming from community contributions.

Confronted with this scenario, we posed ourselves a challenge:

How can we reduce the dynamic API traffic and consequently, the associated costs?
How can we simplify the process for us to contribute our content, making it as effortless as possible?

To address this challenge, we need to understand static and dynamic APIs.

Static API

What is a Static API?

A Static API is an API that delivers its data in the form of static, pre-generated JSON files hosted on a content delivery network (CDN) rather than depending on real-time database queries or actions executed on a server.

Read more about static APIs and a detailed list of their pros and cons here.

If we take a look at our challenge, we consider the following.

Pro: Cost-effective content delivery

With a static API we can publish our content at zero cost (0$). There are several platforms, which offer to host around 20 000 static files per project, for zero dollars. For our guessing game “More or Less”, we decided to go with Cloudflare Pages.

Pro: Effortless content creation

Another advantage, for us is, the creation of the data itself. Using a web editor can be an easy and intuitive experience for normal users, but for us as superusers, we would love to have benefits we already know like automation, version control (git) and working in files instead of form fields.

But remember: This doesn’t have to mean that’s the right way for everyone: In your own project, think about, who is submitting the data? What tools do they like and use? Ask them in which way they would like to create the content.

Cons: But it’s (delivered) static…

On the other hand, the disadvantages are that we can not easily submit data from requests going to our API and there’s no user authentication mechanism.

Meaning a static API is good for game modes created by us but it is not the right when working with user generated game modes. For this we require a Dynamic API (the API you probably already know).

What is a Dynamic API?

A Dynamic API runs on a web server and interacts in real-time with databases or other systems to dynamically respond to user queries and actions. It supports access control (authentication and authorization), data creation and manipulation, and can adapt to varying user demands.

Pro: Works well with user data

In other words, using a dynamic API for our user generated game modes fits perfectly because users can submit and read data from the API. With the benefits of having access control we can control which data a user can modify and implement private game modes.

To keep the focus on the functional user requirements we decided to use Firebase, with Firestore, for the dynamic data.

Cons: But dynamic data costs money…

In the most serverless environments, also in Firebase, every time you access or retrieve data, there are associated monetary costs. So you have to try to keep these at a minimum.

Solution: Merging Static and Dynamic

To fix our problems we can combine both kinds, static and dynamic. We store and maintain our own created game modes with a static API and do all the user created game modes with a dynamic API.

Why should you combine (Pros)

Reduced Server Costs: The static API is completely free of platform charges. Content which comes from the static API can reduce the costs at your dynamic API a lot.
Scalability: Another benefit is the reduction of traffic to your dynamic API. If you have troubles scaling your system at spike times you can shift a lot of traffic from your dynamic API, leading to reduced server load.
Improved Performance: CDNs, where static APIs are deployed, distribute static content to edge locations worldwide, reducing the distance between users and the content they request. This results in lower latency, improved user experience, and faster loading times. Plus, it reduces the load on your dynamic API, leading to faster response times there as well.
Easier content creation: You can speed up content creation. If you use methods, which are known by the content creators, they can maintain content faster, easier and with more fun. Additionally you can automate many things without the need to modify your web editor.

When you should not combine (Cons)

Complexity: Combining static and dynamic APIs can be complex, requiring extra effort to integrate and maintain both effectively. In your project, think of how complex your application or data model is and if it’s worth going the extra mile.
Increased development costs: Developers need to invest effort in implementing the additional API, which can lead to increased development costs. Consider which causes more costs in the end, the development process or the hosting costs?

How to implement?

Create a Static API

First you have to create your own static API. This is how we did it:

1. Define a suitable data input model

Think of how you would like to store or get your data. Do you want to use a database? Do you want to fetch a certain API? Do you want to use files?

You don’t have to think of an optimal structure for your web consumption yet.

2. Generate the Data

In the next step you have to write a service which generates a static version of your data in the form of files. First, plan which endpoints you would like to offer in your API? After that you have to write a service which converts your input data into data portions which you finally write into a static file. The most common format is JSON.

You can include list files with minimal data, single view files with detailed data, pagination for your list files and so on.

You can also compress and deliver image assets at this point.

3. Serve the Data

Your generated data now has to be accessible from the internet. Host the generated files on any static file hosting platform. This could be a traditional web server, a cloud storage solution like Amazon S3, or a specialized static hosting service like Netlify, GitHub Pages, or Vercel. If you are under 20 000 files we recommend you to go with a static hosting service. They are mostly free and offer easy integration into GitHub Actions or Gitlab CI/CD. You can define a stable branch in your repo and automate the deployment of your static API. This makes the deployment of your data very easy.

4. Basic Access Control (Optional)

Sometimes it makes sense to limit your API with CORS to prevent other websites from just consuming it. Most hosting services offer this for free.

Another problem: How to add dynamic data to your static data?

In our game, we maintain high scores for both:

Games we’ve created, sourced from the static API.
User-generated games, sourced from Firebase.

How can we add dynamic highscores to the static games?

Structure your data to split the “must-be” dynamic parts?

To incorporate dynamic highscores into static game data, we followed these steps:

Identify Dynamic Data: Determine which parts of your data need to be dynamic.
Organize Your Data: Move the dynamic components into separate collections within the dynamic API.
Use Both APIs: Once separated, continue to use the static API for standard game data. For any dynamic elements, like highscores, refer to the dynamic API.

How to handle this in the frontend client?

Your frontend client has the responsibility of deciding which API to query static or dynamic. Let’s say the user wants to play a game with a certain ID. We found two strategies on how you can do it.

Static First Approach

This approach optimizes for reduced costs and improved performance. By default, always attempt to fetch from the static API first:

Attempt Static Fetch: Initially, always try fetching data from the static API.
Fallback to Dynamic: If the required game mode or data isn’t found in the static data (e.g., because it’s user-generated), fall back to querying the dynamic API.

const [game, setGame] = useState(null);

useEffect(() => {
	const loadGame = async () => {
		// First, try to get the game data from a static source
		let game = await getStaticGame(id);
		
		// If the game data wasn't found in the static source, try to get it from Firebase
		if (!game) {
			game = await getFirebaseGame(id);
		}
		
		// If the game data wasn't found in both sources, throw a 404 error
		if (!game) {
			throw Error(404);
		}
		
		setGame(game);
	}
	
	loadGame();
}, [id]);

Fetching IDs for Static and Then Deciding

Instead of directly fetching the data, another approach is to fetch identifiers or metadata first:

Fetch Metadata: Start by fetching a list of game IDs from the static API.
Decide on API Type: Once you have the static game IDs, the frontend client can easily determine which API to use for fetching detailed game data.

const [staticGameIds, setStaticGameIds] = useState([]);

// Fetch the list of game IDs from the static API
useEffect(() => {
	const fetchStaticGameIds = async () => {
		const ids = await getStaticGameIds(); 
		setStaticGameIds(ids);
	}
	
	fetchStaticGameIds();
}, []);

const [game, setGame] = useState(null);

// Fetch the game data from the static API or the Firebase API
useEffect(() => {
	const loadGame = async () => {
		let gameData;
		
		// Check if the 'id' is in the list of static game IDs
		if (staticGameIds.includes(id)) {
			gameData = await getStaticGame(id);
		} else {
			gameData = await getFirebaseGame(id);
		}

		if (!gameData) {
			throw Error(404);
		}

		setGame(gameData);
	}

	loadGame();
}, [id, staticGameIds]);

For both strategies ensure if you create your IDs that they are unique.

Why Weren’t Firebase Bundles Our Solution?

When faced with the challenge of optimizing server costs and improving content creation, many might wonder: Why didn’t we just use Firebase Bundles?

Pricing Issue Remains Unresolved: The most significant challenge we were trying to address was the high API usage costs. By transitioning to a Static API for our created game modes, we eliminate these costs altogether. With Firebase Bundles, however, the pricing issue would persist because the pricing structure is fundamentally the same.
Missing Git Advantages: One of our primary objectives was to streamline the content creation process for ourselves. Integrating data management with Git brought several benefits to the table:
- Version Control: With Git, every change is tracked. This offers an advantage in managing content. If a mistake is made or if we wish to revert to a previous version of the game mode, Git allows us to do so effortlessly.
- Automation Possibilities: Utilizing Git opened doors for automation. By integrating with CI/CD pipelines like GitHub Action, content deployment becomes easy. Automatic data generation, testing, and deployment are possible without the need for manual interventions.

Conclusion

In our journey to optimize our guessing game “More or Less”, we combined static and dynamic APIs, achieving significant cost savings and content creation efficiency. While not every project will benefit from this hybrid approach, we think we did by rethinking traditional server structures.

Now put your skills to the test: Dive into our game and guess which repository has more commits.

Our highscore was 23, are you able to beat that? 😉

Happy Guessing!