Build your own DSL with Go & HCL

DSL stands for Domain Specific Language

The main goal of tools such as Kubernetes, Docker compose, Jenkins CI, Gitlab CI, or Ansible, among others, is that their behavior can be customized by configuration. They‘re like a black box that depending on the input (YAML/JSON descriptors), will generate different output.

Other tools which belong to HashiCorp, such as Terraform, Vault, or Nomad use HCL instead of YAML and JSON. HCL stands for HashiCorp Configuration Language and It’s more powerful than JSON or YAML because It provides mechanisms. to implement dynamic configuration files as we will see it later. Some of the advantages of HCL are enum below:

• It is easy to read (and to write) even for non-technical people.
• Implementing a custom DSL, as we will do, is not rocket science because parsing, validation, and plenty of things are provided out of the box.
• A bunch of existing functions are already provided.
• A well-known corporation like HashiCorp is under this project and great products such Terraform, Nomad and Vault use HCL. That implies that the Ops community is used to working with this syntax.

On the other hand, to implement a custom DSL based on HCL you will be required to have knowledge of Go. But I assume that If you’re reading this article you have the required knowledge.

How does HCL look?

The three columns in the figure-1 represent the same information. This is an example of K8s configuration.

Figure 1 — The same K8s configuration in HCL, JSON. and YAML

After looking at figure 1, we can establish that the best syntax is the one provided by YAML because this file is the shortest one. But… keep in mind that the K8s is a dynamic tool but its configuration is static. What I mean is that the configuration of Kubernetes needs to be written in a very specific format and there’s no margin to create dynamic configurations.

HCL by example

I thought that a good showcase would be a DSL used to define the Continuous Integration (CI) or Continuous Deployment pipelines. Let’s see the following HCL script.

example.hcl

Figure 2 — Example of usage of our DSL

Even though you haven’t seen this DSL before, you suppose the purpose of this script, right? The pipeline contains a couple of jobs: The first one seems to run a Python script and the second one seems to run a Python script and later It sends a notification via Slack. There are certain parts in an HCL script that we must identify. The intention of figure 3 is to help us to identify these parts.

Figure 3¡Identification of parts in a HCL script

• Vars: The variables are included in the HCL context. The HCL context is used to evaluate the value of other variables and the value of the attributes in a block.
• Blocks: A block is composed of a name (required), labels (optional), attributes(optional), and subblocks (optional).
• Attributes: There are optional and required attributes and they’re always defined inside of a block.
• (Custom) function: A function enables us to assign the result of a programmatic operation to a variable. Although there are already several functions provided by default and we learn how to implement new ones.

In regard to the DSL of our example, we can state the following:

• There are 3 blocks: job , python and slack
• The block job can contain subblocks of type python and slack .
• The block job has two labels: The first label is used to define the name of the job and the second one to add a brief description.
• The block python has only one label that is the name of the python script to be executed.
• The block python has an optional attribute root_dir .
• The block slack has 2 attributes: channel and message .

Coding

First of all, we need to define the HCL schemas that match our desired DSL. We will follow a bottom-up approach. That means that we will start with the blocks that don’t contain other subblocks. For each block, we need to define the block ID (the name of the block), an array with the labels, and the BodySchema (the attributes and the blocks that can contain)

dsl/schema.go

Source code

Figure 4 — Schema definition for the blocks

The HCL definitions are used to parser the input file to internal HCL structures (generic structures) and They will be used to validate the parsed files.

Don’t forget about the random(10) function that appears in the example above. The implementation of this function is shown in figure 5.

dsl/functions.go

Source code

Figure 5— Implementation of a custom function, random(int)

Visit this link If you want to see more implementations of custom functions. I created a few ones when I built Orion.

We just defined the schemas (HCL structs), but we should model our own DSL into a more friendly structure. Moreover, we want to execute the content in the scripts and not only parse them. Looking at the following code in figure 6, you will realize that all the structs implement a method with the following signature Run(*hcl.context) error. The executions of the scripts will follow a top-down approach. The reason why there’s an interface Step is that we need to keep the order of the steps in the block job (It can contain python and slack`)

dsl/model.go

Source code

Figure 6 — Model implementation for the blocks

We are almost done, we just need to implement the code that will convert the HCL structure into our implementation.

dsl/decoder.go

Source code

Figure 7— Decoder for the root file

Figure 8 — Decoder for blocks job, python and slack

To run the code, we can implement a basic main function as It’s shown in the figure 9.

main.go

Source code

Figure 9 — main function

To run the example you just need to pass the example file.

go run main.go

Figure 10 — Output of

There are several tools that take advantage of HCL but the point is that you can create your own DSL and build great things like Terraform or Nomad.

I also encourage you to have a look at Orion. It’s a personal project that I created and I learned a lot about HCL.

The full code of this repository can be downloaded from Github

GitHub - ivancorrales/hcl-by-example