HyperRace – finding the right architecture approach for an online racing game

Introduction

You all know racing games. From the realistic, simulated ones like Formula 1 or Forza Motorsport to the playful, arcade-heavy ones like Super Mario Kart or Need For Speed.

The range is large, but somehow pretty standard. So we want to take it a little crazier. After all, who says that it always has to be cars as vehicles? Why not a turbine? Or yes – directly a whole rocket engine? Yes! And the driver sits directly on the drive train! And players shoot each other down with hilarious and mean attacks and traps. Something like this:

I know it’s not strictly a racing game – more of a flying game, but from a technical point of view we have the same parameters per player such as the position or speed of the players and other objects on the track and the collisions of these objects with each other.

Ok, now that we have our crazy game idea, we also need to somehow ensure that our players can enjoy playing together on different devices.

But how are we gonna do this? What multiplayer architecture approaches are there? What are their pros and cons? What data do we need to exchange between players? And how do we trade different latencies to give every player a smooth, zero-latency gaming experience?

Exactly these questions are answered in this blog entry. Thus, this blog entry can be seen as the first starting point into the world of online multiplayer architecture.

Requirements

To narrow things down a bit, let’s first establish a few rules for our game:

• Each race takes place on a closed race track / There is no open world
• There are 10 different racetracks and 7 different vehicles.
• A maximum of 10 players play together in a lobby or on a race track
• The graphic representation takes place exclusively on the end device of the user
• The gaming experience should take place in real time
• Players are matched to a lobby
• There is a global player ranking
• Each player can accelerate, brake and steer
• Each player can pick up items on the racetrack and use them to attack other players or place traps on the racetrack
• Each player must be authenticated in order to play
• The system should initially be able to trade around 250,000 users per hour and scale accordingly
• The system should be operated as cheaply as possible; however, we also want to offer our players a smooth, fun and immersive gaming experience

As can be seen in the requirements, in this article we will focus exclusively on the pure racing game experience – for example driving, overtaking and collision between vehicles and objects. For the time being, we neglect the use of universally popular social functions such as managing friend lists or joining friend lobbies.

Architecture approaches 

Now that we’ve defined some basic requirements, let’s first look at the most popular architectural paradigms that connect players around the world. Underlying these approaches is a common goal: to provide players with a synchronous, latency-free and fluid gaming experience in near real-time. 

Hosted Peer-to-Peer (Hosted P2P) 

In times when cloud solution is always touted as the all-purpose weapon, it is almost unbelievable that an old approach like peer-to-peer is still so widespread. But popular games like Demon’s Souls, Dark Souls, Bloodborne, GTA Online, Red Dead Online, Animal Crossing: New Horizons, Super Smash Bros. Ultimate or F1 2021 partly operate their online worlds via peer-to-peer networks ¹.

The principle of a hosted P2P network seems simple at first: One player is the host of the game, i.e. the player’s computer acts as a “server”. The host inputs, as well as the inputs of the other players, arrive at the host. The current game events (state) are calculated and sent back to the players (Img.1). 

So as a company, there is no need for a server at this point, so we gonna save these costs, which can be an enormous sum with a targeted utilization of 250,000 players. The scaling and the corresponding rush of incoming players can also be easily distributed among several host players. Unlike a dedicated server, thousands of games are not calculated on one machine, but on a selected player – the host – calculates exactly one game and transmits the results to the other players (in our example max. 10 players in a lobby, so 25,000 host players must be found with a planned load of 250,000 players).

Conversely, this also means that the other players in a lobby are dependent on the host in many ways. The host migration – which player is the best host? – is crucial. Because if the host has a miserable download or upload speed, this creates high latencies for the other players – delays and a bad gaming experience for these players are the consequences. One solution would be that the host is the player who has the lowest average ping time among all other participating players. Or only players who are geographically close to each other are assigned to a lobby. But what do we do if a player from Tokyo wants to play with a player in Berlin? In addition, the host user must provide enough free hardware resources such as CPU and RAM in order to be able to calculate the entire gameplay and display his own gameplay. But what if that’s not the case? Likewise, the host receives the results on his PC first – simply because of the short geographic and physical transmission path and the resulting lack of ping – so that this can also lead to asynchronous gameplay. This must also be prevented so that all players get a synchronous, simultaneous gaming experience. 

Another challenge in a hosted P2P network is host cheating or host hacking. How can we trust the host and make sure they aren’t cheating and manipulating gameplay? Because all player events are calculated on his PC. To solve this requires a solution such as a separate authoritative server ² .

In summary, there are the following reasons for or against a hosted peer-to-peer approach:

In particular, the problem of the trustworthiness of the host poses a major challenge for the P2P approach. Another popular approach is the “dedicated server” architecture. This approach solves the trust problem through a central authority. Let’s take a closer look.

Dedicated Server

With a “dedicated server” architecture, all game events – the inputs of the players – are sent to a central server. The server calculates the resulting state for each player and sends this state back to the players (Img. 2). The server is neither part of the game nor a separate player as in the hosted P2P approach. A dedicated server is only responsible for calculating the gameplay. Other social functions such as matchmaking are usually outsourced to other servers, since a dedicated server and all its resources are only provided for this one task – calculating the current state. Accordingly, dedicated servers usually do not have a GPU, since a graphical game display is not necessary. Instead, they have a high number of CPUs and RAM, since game lobbies and their state have to be kept and calculated during the game. 
A large number of currently very popular multiplayer games such as Fortnite, Minecraft, Apex Legends, Rocket League, Among Us, Rainbow Six Seige, Ghost of Tsushima Legends or Roblox use this principle ¹.

Compared to the hosted P2P approach, there are definitely costs associated with operating dedicated servers. Depending on the infrastructure and scaling pattern, this can result in different costs ³. Furthermore, scaling can also be a challenge if the servers are operated “on-premise”, because there must be enough servers available, they must be maintained and operated. And in addition, these servers are sometimes “dead capital” if they are not fully utilized. 

On the other hand, there are advantages that eliminate some of the disadvantages of the hosted P2P approach: When operating a dedicated server, you can count on consistently high performance, because we know about the performance of our servers and know how many users can play on one unit at the same time. This is usually not the case with hosted P2P, since the performance and networking capability can vary greatly depending on the host’s end device. This also makes it easier to estimate and plan capacities. If we operate our servers using the cloud, we can relatively easily scale horizontally (more servers) and vertically (more resources per server) depending on the corresponding demand from the players. Operation in the cloud also ensures consistent global performance, since the servers are distributed around the world and are not in a central location. This is mostly not the case in the hosted P2P approach, since a gaming lobby is dependent on the performance and location of the host, which does not need to be close to the other players.

However, one of the biggest advantages of the dedicated server approach is the reliability and security of the game state, as it is calculated on an independent, secure instance, making it very difficult for players to cheat. While the hosted P2P approach allows a host to manipulate its position on the racetrack, for example, this is made significantly more difficult with approaches such as server-side reconciliation. Because the server always knows the last position of the player. If this delivers unrealistic values ​​as input, the server can compare this with the value of the last frame and determine that there is an abnormal value and send adjustments to the player accordingly ⁴. 

In summary, there are the following reasons for or against the dedicated server approach:

Hybrid solutions & services 

In addition to the approaches just mentioned, there are also mixed approaches. It is not uncommon for a dedicated server or hosted P2P architecture to be expanded by additional, separate game services. Such services are operated on standalone servers and deal with things like matchmaking, grouping up players and picking a dedicated server or (for peer to peer games) sharing IP addresses between players so they can connect together. They handle other things such as leaderboards, player progress, unlocks, chatting and friend lists etc. In this way, areas of responsibility are clearly separated and the load is distributed to different machines (Img. 3). 

Well-known examples of such a solution are the games GTA Online, Warframe or Among us, whose players are connected to each other via a hosted P2P network, but the matchmaking runs on an own server. This model has been the subject of some criticism due to the potential for host cheating. However, the developers mostly has implemented several separate Measurement Services to prevent cheating in the game like Client-Side-Validation that includes checks for speed hacks, aimbots, and other forms of cheating and Activity- and System Monitoring that checks for suspicious activity on the player’s system like memory manipulation, injection attacks, and other forms of hacking ⁴.

Game States: Racing World & Player Data

Now that we’ve gotten to know the common architecture approaches in online gaming, we have to ask the question of what data we have to exchange in order to determine the current state of the game. But before we get into the data, what exactly is the state anyway? To shed more light on this, let’s look at the multiplayer approach of one of the most well-known game engines – the Unreal Engine 5 – here for example, a distinction is made between three different units: GameMode, GameState and PlayerState ⁵ (see Figure 5). Other game engines like Unity use a similar approach, but the implementation varies ⁶.

Let’s explain the orange part of the image above (Img. 4) using our racing game ⁷: 

The GameMode manages the global rules of a race. In our case these are for example: 

• The minimum number of players needed to start a race (min. 2)
• The maximum number of players in a race (10 players)
• The number of laps each player must complete to finish the race
• The spawn location and order of players on the racetracks
• The time of the countdown
• The allowed weapons and traps

The GameState manages global information about the current gameplay that is relevant for all players, such as: 

• The number of currently connected players
• The start and finish time of each individual player
• The current ranking (based on each player’s current position)
• The position of all players
• The location and state of all fired weapons and traps
• The position and state of each checkpoint on the track.

The PlayerState will exist for every player connected to the game on both the server and the clients. This class can be used for replicated properties that all clients, not just the owning client, are interested in like the current

• car model
• round on the race track
• position of a player  
• speed of a player 
• direction of a player
• steering angle of a player
• down force of a player
• amount of a specific collected item of a player
• status of a specific collected item (fired or not and target player)
• collisions of a player with another player or object

By continuously updating the game state and all player states based on these inputs, we can provide a seamless and engaging experience for all players involved.

In addition to identifying what data makes up the state, it is important to understand how the player state is computed taking into account different latencies so that all players feel like they are sharing a synchronous gaming experience with other players at all times. How this works is explained in the next section.

State Simulation

Almost all modern 3D online games have to face an essential challenge: How do we balance the latency of the client-server communication (or client-host player in P2P) so that each player can experience smooth, latency-free gameplay locally, but at the same time uses the current state of the server. 

Client prediction 

During a multiplayer online game, each player communicates with the server (dedicated server) or the host player (hosted P2P). This communication happens asynchronously so that the player can experience a smooth gaming experience: For example, the player sends his new position to the server. Instead of waiting until the answer comes back from the server and freezing the player’s screen for the response time (latency) of for example 300 ms (that would be synchronous), the next frame is rendered directly and the player (in our case the racing car) is located directly at the new position. This approach is known as client prediction. Assuming that the game world is deterministic, the player’s input can lead directly to the desired action locally (for exmaple a new position, an animation or a physics calculation) and this will in most cases also correspond to the state of the server ^8,9.

Server reconciliation 

Since in a racing game the communication between player and server should take place as quickly as possible, UDP is used as the network protocol in most online multiplayer games and frameworks ^10,11. In contrast to TCP, the order of the incoming packets is not guaranteed with UDP. Depending on the latency between player and server, the responses from the server can sometimes arrive at the player at different times and in a different order, so that the position of the server no longer has to match the new local position of the player (because the player is already moving in the game, for example). Thus, the player game state and the server game state are desynchronized. However, since the server game state is the trusted game state ¹², the game now has to process the new information from the server and make an adjustment, for example reset the player locally to the server’s position. However, since the player has already moved locally, this would lead to an unclean gaming experience (glitches) and breakes immersion – and we want to avoid that at all costs! So what do you do in such a case?

One solution to this problem is server reconciliation. A sequential number is added to each input sent by the player to the server to ensure the order of actions. This input is stored both on the player and on the server. If the player now receives a response from the server that matches the player state, the associated number can be removed locally and the next number follows in order. However, if the server’s answers follow in a different order, the player now knows locally that an answer to a specific number is still missing. However, since the player knows the answer to the actions before and after the missing answer, he can simulate the current state, which is then validated again by the server afterwards. This means that the server and player always synchronize with a slight delay in comparison to the next frame. However, this also ensures that the gameplay is represented graphically fluently ⁸.

Interpolation

Now that the simulated state between two states (the result of the server-side reconciliation) also graphically leads to a smooth transition movement, we use the concept of interpolation. Interpolation algorithms are part of every common game engine ^14,15.  There is linear interpolation to calculate states between two positions and spherical interpolation to calculate states between two degrees of rotation. This algorithm always requires two states (for example two positions on the race track) in order to calculate the positions in between and then display them in a fluid movement. In our case, these are the two positions that have been successfully validated and transmitted by the server before and after a certain input from the player. Based on these positions, the intermediate positions are now calculated, which are required for the execution of the graphic representation ¹³.

The combination of client-side prediction, server-side reconciliation and interpolation can help ensure a smooth and responsive online multiplayer game experience. By using these techniques, game developers can minimize network latency issues and keep players synchronized across multiple devices and network connections. To make these three approaches easier to understand, I will highly refer to a Live-Demo at this point.

Conclusion

With this article we took the first step into the world of online-multiplayer-architecture. We discussed architectural approaches for multiplayer games, got to know the game and player state and understood which data we have to exchange between our players. Finally, we looked at concepts that compensate the latencies between player and server by simulating an intermediate state.

When choosing a suitable architecture for an online game, it has been shown that P2P, dedicated servers or hybrid approaches are technical solutions that present developers with different challenges. As it turned out, hybrid solutions in particular are preferred in order to keep costs as low as possible while still being able to offer players an immersive and fun gaming experience through server-based, separate services such as match-making or activity monitoring. P2P solutions are ideal for small, unknown games with little to no budget, while games with predictable, stable sources of income have more money for their own server infrastructures. However, by using free tiers, smaller games with a few players at start can also take advantage of Dedicated Cloud Servers to launch their game in the first place ^2,16.

In order to make a final decision regarding the architecture of our racing game, further research is required, such as a detailed cost comparison, comparison of different frameworks or cloud providers and implementation details.

Sources