Debugging Go in Kubernetes with Delve and Tilt

The idea of this post is to introduce an easy way to debug Go applications running in Kubernetes.

Let’s introduce briefly the tools we are going to use before we get down to the point.

Why a debugger?

Who has not ever found that their program does not work due to some unknown reason? In front of this situation, surprisingly (or not) what most people use for debugging is Println. I admit to having abused it countless times…

https://github.com/MariaLetta/free-gophers-pack

Usually using Println it is a bad idea because it is inefficient since you need to add all the variables manually at the risk of leaving uncovered cases unprinted, apart from leaving the code messy.

Debugger to the rescue!

To solve this we will use Delve, a Golang debugger. Explaining how it works is outside the scope of this post but there are plenty of good tutorials out there. Some of the advantages of using a debugger are:

• Inspect all variables at the debug point.
• Visibility of the call stack.
• Update variables on the fly.

Fair point… but what about apps running on a Kubernetes cluster?

Why Tilt?

Tilt is an open-source microservice development environment for devs deploying their apps to Kubernetes.

The main goal is to ease the developer experience by helping with local continuous development and deployment of apps to local Kubernetes clusters. It does this by monitoring the source code and automatically building and pushing the deployments.

Another useful feature is the UI provided to monitor the deployment process, with information on all your deployments and logs.

Let’s use the best of both worlds, Delve and Tilt!

From now on I will assume that you have Delve and Tilt installed as well as a local Kubernetes cluster running, I will use Minikube for it.

Down to the point!

Let me explain, again, at a high level what we are trying to achieve so that we have a clear mental model. The idea is to launch our application through a debug server and expose it so that we can connect remotely from our terminal or IDE to debug it as if we were running our application from our machine.

First of all I have created a simple app to be our guinea pig:

Once we have the application we need to be able to deploy the app to the cluster (ignore please the step where the image is built, as it is done in the Tiltfile that we are going to see next):

With all this, we only have to write our Tiltfile, a file that Tilt will run on startup. It is written in Starlark, a dialect of Python. To know more about it, you can check the official docs.

Let’s take a look at the file to then explain what we are really doing:

Although the comments already explain what is being done on each step, I will try to break it down for a better understanding.

First, we create a couple of variables and we allow Tilt to deploy to our cluster with the allow_k8s_contexts. function.

Next, we load the restart_process extension with the docker_build_with_restart function.

After, we build the binary locally to be used in the Dockerfile. Notice that in the Dockerfile we also install the Delve debugger.

Then, we call the docker_build_with_restart function that basically is a wrapper of docker_build to build the image with the addition that it knows how to restart the process at the end of a live update.

Finally, we create the Kubernetes deployment from the yaml file k8s_yaml(deployment.yaml) and we configure it with a port_forward. Broadly this will direct all the traffic from a local port, we have used the same as the Delve server port 50100 but it could be different, to the pod port exposed by the Delver server itself. So, this will allow us to connect locally to our debugger running in the cluster.

How to run it

Now that we have all the pieces, let’s see how we can run Tilt with the debugger server to then connect it to it.

In order to run the application I created a simple make target in a Makefile like the following one:

.PHONY: tilt-up
tilt-up:
   eval $$(minikube docker-env) && tilt up; tilt down

This will start Tilt which will therefore run the Tiltfile. After this command is run you will have your app running in your cluster exposing the debugger server port 50100. Taking a look at the pod logs you should see something like this:

sample-app │ API server listening at: [::]:50100
sample-app │ 2022–10–02T20:51:33Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)

Now that we have the app ready, let’s connect to it.

dlv connect localhost:50100 --api-version=2

If you followed all the steps in order, you should be able to connect to the debugger without any problem.

BONUS: I’m going to leave as a task for you to configure the debugger of your preferred IDE to connect remotely to our debug server instead of accessing it manually through the terminal.

Closing

I hope this post has been useful and I have been able to convince you to at least try a debugger next time 😊

The fact of running the debugger in a Kubernetes cluster adds a little complexity but being honest working with Tilt removes most of the toil.

Any suggestions or improvements are welcome, please share it.

Happy coding!