Creating an OS using Rust: [Part-1] Creating “freestanding” executable

This is one of my hobby projects to build a fully functioning operating system using Rust. Especially considering the numerous interesting features that Rust provides out-of-the-box, like pattern matching, error handling, string formatting, fearless concurrency, and the ownership system (for memory safety), to name a few.

As part of “Creating an OS” series, I’ll be releasing a series of blog posts as I continue to work on this project. This is the first of many such blogs which describes the first step into building an OS: Creating a “freestyle” Rust executable.

Let’s get started.

Prerequisite

The basic assumption in these blog posts is that the reader is already well accustomed to the Rust Programming Language, and understands the Rust syntax and concepts like cargo, rustc, rustup, traits, structs, etc.

Bare Metal

In order to create our own operating system kernel, we need to first write a Rust executable that does not link to the Rust standard library. What this achieves is the ability to run Rust code on the bare metal without an underlying operating system.

Let’s first understand what bare metal means.

Bare machine (or bare metal) refers to a computer executing instructions directly on logic hardware without an intervening operating system.

Bare-metal programming requires more effort to work properly and is more complex because the services provided by the operating system, and used by the applications, have to be re-implemented to serve the needs. These services can be:

• System boot (mandatory)
• Memory management: Storing location of the code and the data regarding the hardware resources and peripherals (mandatory)
• Interruptions handling (if any)
• Task scheduling
• Peripherals management (if any)
• Error management, if wanted or needed

What this means is that we can’t use the operating system goodies like threads, files, heap memory, the network, random numbers, standard output, or any other features requiring OS abstractions or specific hardware. And this makes sense, considering we’re trying to write our own OS and our own drivers.

As a result, we will have to let go of most of the Rust standard library, but there are still a lot of Rust features that we can use. For example, we can use iterators, closures, pattern matching, option and result, string formatting, and of course the Rust ownership system. These features make it possible to write an OS kernel in a very expressive, high-level way without worrying about undefined behavior or memory safety.

In short, we will create an executable that can be run without an underlying operating system. Such an executable is often called a “freestanding” or “bare metal” executable.

Disabling the Rust Standard Library

First, we need to create a new cargo application project.

cargo new corrode-os --bin

This will set up the Rust directory structure containing the below files.

• Cargo.toml contains the crate configuration, like the crate name, the author, the semantic version number, and dependencies.
• src/main.rs file contains the root module of our crate and our main function.

Note: The — bin flag specifies that we want to create an executable binary.

Next, we will disable the standard library using the no_std attribute.

As we can see from the red curly line, Rust is already showing error for println command now. Let’s try to build the application and see what happens.

We can see that the println macro is failing, which is expected since it is part of the standard library, which is no longer included. As a result, we can no longer print things, which makes sense since println writes to standard output, which is a special file descriptor provided by the operating system.

Additionally, the Rust compiler has also highlighted two more errors:

• #[panic_handler] function, and
• a language item.

Panic Implementation

The panic_handler attribute defines the function that the compiler should invoke when a program panics (i.e. crashes) unexpectedly. The Rust standard library provides its own panic handler function, but since we can’t use the standard library anymore, we need to define our own panic implementation.

The PanicInfo parameter contains the file and line where the panic happened and the optional panic message. The panic function should never return, so it is marked as a diverging function by returning the “never” type “!”.

At this point, we can’t do much in the panic function, so we just loop indefinitely.

“eh_personality” language item

The eh_personality language item is a function used by the compiler’s failure mechanisms. It’s named eh_personality because of the complexity of stack unwinding on various platforms.

Stack Unwinding

Stacks are last-in, first-out (LIFO) data structures, in other words, the last item pushed (inserted) on the stack is the first item popped (removed) from it.

A call stack is a stack data structure that stores information about the active functions. The call stack is also known as an execution stack, control stack, function stack, or run-time stack. The main reason for having a call stack is to keep track of the point to which each active function should return control when it completes executing. Here, the active functions are those which have been called but have not yet completed execution by returning.

When a function terminates, execution goes to the address stored when the function was called, and the stack for the called function is freed. So, a function normally returns to the function that called it, with each function freeing its automatic variables as it completes.

If a function terminates due to a thrown exception instead of normal return call, the program still frees memory from the stack. However, instead of stopping at the first return address on the stack, it continues freeing the stack until it reaches a return address that resides in a try block. Execution control then passes to the exception handlers at the end of the try block rather than to the first statement following the function call. This process is called unwinding the stack.

By default, Rust uses unwinding to run the destructors of all live stack variables in case of a panic. This ensures that all used memory is freed and allows the parent thread to catch the panic and continue execution.

Unwinding, however, is a complicated process and requires some OS-specific libraries (e.g., libunwind on Linux or structured exception handling on Windows), so we don’t want to use it for our operating system.

Rust provides an option to abort on panic which disables the generation of unwinding symbol information and thus considerably reduces binary size. We’ll add the below lines in Cargo.toml to disable unwinding.

This sets the panic strategy to abort for both the dev profile (used for cargo build) and the release profile (used for cargo build — release). Now the eh_personality language item should no longer be required.

Let’s try and compile the application again.

We get a new error this time. Our program is missing the start language item, which defines the entry point to an application.

“Start” language item

One might think that the main function is the first function called when you run a program. However, most languages have a runtime system, which is responsible for things such as garbage collection (e.g., in Java) or software threads (e.g., goroutines in Go). This runtime needs to be called before main, since it needs to initialize itself.

In a typical Rust binary that links the standard library, execution starts in a C runtime library called crt0 (“C runtime zero”), which sets up the environment for a C application. This includes creating a stack and placing the arguments in the right registers.

The C runtime then invokes the entry point of the Rust runtime, which is marked by the start language item. Rust only has a very minimal runtime, which takes care of some small things such as setting up stack overflow guards or printing a backtrace on panic. The runtime then finally calls the main function.

Our freestanding executable does not have access to the Rust runtime and crt0, so we need to define our own entry point. Implementing the start language item wouldn’t help, since it would still require crt0. Instead, we need to overwrite the crt0 entry point directly.

To tell the Rust compiler that we don’t want to use the normal entry point chain, we add the #![no_main] attribute. Also, we will remove the main function, since it won’t make sense without an underlying runtime that calls it.

Next, we will overwrite the operating system entry point with our own _start function.

The #[no_mangle] attribute disables the name mangling to ensure that the Rust compiler really outputs a function with the name _start. The attribute is required because we need to tell the name of the entry point function to the linker in the subsequent steps.

Name Mangling

In compiler construction, name mangling (also called name decoration) is a technique used to solve various problems caused by the need to resolve unique names for programming entities in many modern programming languages.

The need for name mangling arises where the language allows different entities to be named with the same identifier as long as they occupy a different namespace (typically defined by a module, class, or explicit namespace directive) or have different signatures (such as in function overloading). It is required in these use cases because each signature might require different, specialized calling convention in the machine code.

We have also marked the function as extern “C” to tell the compiler that it should use the C calling convention for this function (instead of the Rust calling convention). The reason for naming the function _start is that this is the default entry point name for most systems.

The ! return type means that the function is diverging, i.e., not allowed to ever return. This is required because the entry point is not called by any function but invoked directly by the operating system or bootloader.

So instead of returning, the entry point should e.g., invoke the exit system call of the operating system. In our case, shutting down the machine could be a reasonable action, since there’s nothing left to do if a freestanding binary returns. For now, we fulfill the requirement by looping endlessly.

Let’s try compiling the application once more to see if this works.

This time, we got a new linker error.

Linker Errors

The linker is a program that translates the generated code into an executable. Since the executable format differs between Linux, Windows, and macOS, each system has its own linker that throws a different error.

The fundamental cause of the errors is the same: the default configuration of the linker assumes that our program depends on the C runtime, which it does not.

To solve the errors, we need to tell the linker that it should not include the C runtime. We can do this via below two approaches.

• By passing a certain set of arguments to the linker (not recommended, since the arguments are OS-specific and need to be configured and handled separately for every operating system)
• By building an executable for a bare metal target (recommended)

Building executable for bare metal target

By default, Rust tries to build an executable that is able to run in your current system environment. For example, if you’re using Windows on x86_64, Rust tries to build an .exe Windows executable that uses x86_64 instructions. This environment is called your “host” system.

We can check the host system information by running the below command.

rustc --version --verbose

We see that the host triple is x86_64-pc-windows-msvc, which includes the CPU architecture (x86_64), the vendor (pc), the operating system (Windows), and the ABI (msvc).

Application Binary Interface

In computer software, an application binary interface (ABI) is an interface between two binary program modules. Often, one of these modules is a library or operating system facility, and the other is a program that is being run by a user.

An ABI defines how data structures or computational routines are accessed in machine code, which is a low-level, hardware-dependent format.

In contrast, an application programming interface (API) defines this access in source code, which is a relatively high-level, hardware-independent, often human-readable format.

By compiling for the host information, the Rust compiler and the linker assume that there is an underlying operating system such as Linux or Windows that uses the C runtime by default, which causes the linker errors. So, to avoid the linker errors, we can compile for a different environment with no underlying operating system.

An example of such a bare metal environment is the thumbv7em-none-eabihf target host, which describes an embedded ARM system.

To be able to compile for this target, we need to first add it in rustup.

This downloads a copy of the standard (and core) library for the system. Now we can build our freestanding executable for this target.

And voila! The build finished successfully. By passing a — target argument we cross compile our executable for a bare metal target system. Since the target system has no operating system, the linker does not try to link the C runtime and our build succeeds without any linker errors.

Summary

Finally, we’ve been able to create a Rust “freestanding” executable that can be triggered on bare metal.

For next steps, I’m currently working on turning the freestanding binary into a minimal operating system kernel. This includes creating a custom target, combining our executable with a bootloader, and learning how to print something to the screen.

I’ll cover the detailed implementation steps in the next post.