KUBERNETES + AGONES

Trying Agones for Multiplayer Game in Kubernetes

There’s always something to learn, even when the project is not finished.

Agones logo.

My latest project is an attempt to learn Agones, a Helm chart for multiplayer game developed by Google. Despite I did not finish the project in time, I learn a few lessons with it.

Background

Agones is a framework installed in Kuberenetes cluster for creating and managing multiplayer games. Agones, in theory, allows us to spawn a new game server after matchmaking process is completed (meaning, the room is full and the game is ready to begin).

Why am I interested in Agones? Because I’ve been thinking for the past few months on how game server works and the scalability aspect of it. So, when I found Agones, I’m interested to see how it works.

The idea is Agones can allocate a new game server for each game. It’s not like one game server filled with many rooms (unlike Socket.io). Rather, one game server is dedicated to one room. At least that’s the concept.

Learning Process

Developing The Game

In order to try Agones, I developed a tic-tac-toe game. I store the code in three repositories:

• Frontend code, which contains the user interface for the game.
• Game server code, which contains the backend logic during the game.
• Backend-database code, which contains a simple server to connect to MongoDB.

Now, you might wonder, why I need 3 services. The reason is that during testing, I found out that due to Webpack, the frontend cannot use Mongoose library, so I need to create a service dedicated to access MongoDB.

Architecture diagram for planned tic-tac-toe game.

This is the frontend code. The frontend is programmed with Next.js and using awesome template from Theodorus Clarence. The development took quite a while because I’m not used to React hooks, but in the end, I make it work.

GitHub - iamdejan/agones-tic-tac-toe-fe

This is the game server code. The game server is programmed with Node.js and Typescript. The connection between frontend and game server uses Socket.io. The reason why I use Socket.io is because it’s easier, compared to Websocket (or even handling TCP directly).

GitHub - iamdejan/agones-tic-tac-toe-be

This is the backend-database code. It’s build with Express.js and Typescript. As explained, this service only acts as a proxy to database. There are few endpoints for creating the game and/or retrieving game server URL by game ID.

GitHub - iamdejan/agones-tic-tac-toe-be-db: Agones Tic-Tac-Toe - BE Proxy from FE to DB

Since I rarely code Tic-Tac-Toe (especially the frontend part took quite a while), those 3 repositories took me about one month probably.

Provisioning The Infrastructure

Now, after developing the infrastructure, I need to provision Kubernetes cluster, as well as installing Agones. I use Google Kubernetes Engine simply because I have free credits left.

You can see the code here (sorry for the lack of README file):

GitHub - iamdejan/agones-tic-tac-toe-be-infra: Agones Tic-Tac-Toe Infra

Note that I created my own module to provision the GKE from Terraform, which you can see here:

GitHub - iamdejan/terraform-k8s-module: Terraform module to provision a Kubernetes cluster in GCP.

However, during provisioning step, I face two difficulties:

1. If I add ip_allocation_policy during provisioning, Terraform will show error. Turns out, when we first creating the cluster, GKE insists on creating and allocating the secondary subnets (subnets used for Kubernetes pods and services) themself.
2. After provisioning Kubernetes cluster, if I didn’t add ip_allocation_policy , then Terraform will “reset” (delete then re-create) the Kubernetes cluster. We don’t want that every time we apply the Terraform configuration. The solution is to add ip_allocation_policy after provisioning the cluster. Yes, not ideal, but I do not do research on fix this problem, especially when provisioning Kubernetes cluster at Jakarta (asia-southeast2).

The solution is written in the README file of terraform-k8s-module repo.

Trying Agones

Now, onto the main part, which is trying Agones.

My first attempt is to use the image I pushed to Google Container Registry. I create the game server by modifying the tutorial from here to create the game server. Here’s the YAML file:

apiVersion: "agones.dev/v1"
kind: GameServer
metadata:
  generateName: "tic-tac-toe-"
spec:
  health:
    disabled: true
  ports:
  - name: default
    portPolicy: Dynamic
    containerPort: 4000
    protocol: TCP
  template:
    spec:
      containers:
      - name: agones-tic-tac-toe
        image: [the image version of the game server]
        resources:
          requests:
            memory: 128Mi
            cpu: 40m
          limits:
            memory: 128Mi
            cpu: 40m

Then, per tutorial, I run kubectl create -f game-server.yaml. Game server is created, and then the status is changed to Scheduled. But, I cannot access it. What happened?

Game server’s state is stuck at Scheduled after one minute.

Turns out that I need to set the server’s state as Ready, otherwise the server cannot receive traffic. How fool I am, I said to myself in my head, not reading the tutorial completely and missing the required state to accept connections.

Screenshot from Agones tutorial that states that the game server needs to be put into Ready state in order to receive connection.

After reading the tutorial, I can set the game server’s state to Ready by sending POST request to http://localhost:${AGONES_SDK_HTTP_PORT}/ready. You can see the change here.

Then, as written in the tutorial, I tried to test the connection from my local laptop by using nc -u {IP} {PORT} command. But, it stuck. Nothing happened. I tried to debug the connection with nc -vz command, but this is what I found on my local.

Failed to connect to the game server after the state is changed to Ready.

Turns out that I need to open port 7000–8000 in Kubernetes nodes. That’s interesting, since I usually rely on Service or Ingress to access Kubernetes pods. I never open Kubernetes node’s ports directly.

In order to open the node of the port, you can add a firewall rule (either in Firewall page if you’re using default network, or in the VPC page if you provision Kubernetes cluster in VPC) to open port 7000–8000 for either TCP (which is what I need, since I’m using Socket.io), UDP (this is what you probably need if you’re following Agones tutorial). In order to target Kubernetes nodes, you need to use gke-node network tag. Ideally, I would like to use Terraform, but since this is uncharted territory for me, I choose to directly add the rule in the console.

I add firewall rule for Agones connecttion in custom VPC.

Then, I tried again. But, it’s still timeout. It seems that I need to delete the game server after adding the firewall rule to Kubernetes node. So I delete and re-create the game server, and the connection works!

The new game server is ready.

My local computer is successfully connected to game server in Agones.

Now, it’s time to play the game. I run the game in my local computer, and I use Google Chrome and Brave browsers to play. Ideally, I also need to put the frontend in Kubernetes cluster, but time does not permit me (see The Unfinished Part section below).

Here is the screenshot when I tried to play the game:

Home page of the tic-tac-toe game.

The game creator is in waiting room after clicking “Create a game” button. The game ID then needs to be input by second player.

Second player inputs the game ID given by first player (the game’s creator).

Game is started after second player clicks “Join a game” button, and we can play the game.

The page after one player wins, either X or O.

The Unfinished Part

There are several things that I need to improve for this project:

1. At the moment, the IP address and the port is hard-coded in the frontend, because I create the game server manually from Cloud Shell. Ideally, not only I put the frontend inside Kubernetes cluster, but I also need to create the game server in the code, not hard-coding the IP address and port. I think this can be improved later, once I moved to another cloud provider and getting another free credit :)
2. One logic that can be improved is that I provision the game server when a player creates a game room. I think the more optimized version is to provision only after the room is full.
3. One thing that I want to try but the time doesn’t permit (my GCP free trial period is almost up) is that I want to use GameServerAllocator from Agones, so that I don’t need to create game server, but rather use what’s already allocated for me. This is related to point 1.

Conclusion

Despite being an unfinished project, I learn a few things:

1. Understand the tools needed first, before trying to implement custom game. I spent weeks developing Tic-Tac-Toe game, only to realize that Agones is not as easy as I think, because there are many things I need to setup, from Kubernetes cluster, game server (including its’ states), and firewall rule(s).
2. Properly read the tutorial. Just because you assume you know how it works, that doesn’t guarantee that you really know. Slowly and properly read the tutorial.
3. Related to number one, try the example given by Agones. If the example didn’t work, you know that either you misunderstand the guide, or simply the guide is not good enough. If following the examples didn’t work, at least you save some time before developing a custom game. This is also related to point 2.
4. For startups, if resource permits (money, head-counts, etc.), I think it’s better to use Google Cloud Game Servers, because they already manage the server allocation for you. Google Cloud Game Servers internally use Agones. However, if you already have Kubernetes in other cloud providers, I can see the appeal to use Agones directly.

That being said, the most important conclusion is that do not be afraid to try. You rarely get everything right the first try.