Under the hood of a Gleam test-framework

If you were to build a test-framework in your favourite language, how would it look? What features would you add? How would you implement the test execution? Should tests be able to share data or setup code?

These are all questions I have been thinking about lately as I have been busy implementing a small test-framework for Gleam.

Gleam is the programming language I enjoy spending my time with most recently. I have mostly been using it to do web stuff, so far so I wanted to build something different.

Even if you have little to no experience with Gleam the article should still be digestible. The language is quite simple (in the Rich Hickey way1) and the tool-chain makes Gleam easy to use.

So far, the user satisfaction has been through the roof, 100% happy users!

Yes, as you probably guessed, I am the only user so far. Maybe this article can inspire someone else to try it and give me some feedback though!

Show me the code

This is a snippet from one of my other hobby projects. It is a test suite used to verify that resolving a path using a root path resolves correctly.

pub fn parse_path_segments_suite() {
  Suite("When parsing a path into segments", [
    Test("an empty path has no segments", fn() {
      segments("")
      |> expect.to_be_equal([])
    }),

    Test("the root path has no segments", fn() {
      segments("/")
      |> expect.to_be_equal([])
    }),

    Test("a single segment path is split", fn() {
      segments("/posts")
      |> expect.to_be_equal(["posts"])
    }),

    Test("nested segments are split in order", fn() {
      segments("/posts/hello-world")
      |> expect.to_be_equal(["posts", "hello-world"])
    })
  ])
}

fn segments(raw: String) -> List(String) {
  request.parse_path(raw, None).segments // <-- this is what is tested
}

Running this when everything is working produces the following output:

$ gleam test
  Resolving versions
   Compiled in 0.07s
    Running paper_test.main
  Discovered 1 suite(s).
  Analysed suites: No problems found.
  Running 1 suites...
  Suite When parsing a path into segments completed successfully with 4 test(s)
     Test an empty path has no segments completed successfully
     Test the root path has no segments completed successfully
     Test a single segment path is split completed successfully
     Test nested segments are split in order completed successfully
  All 4 test(s) passed!

If we break the implementation, it fails loudly:

$ gleam test
  Resolving versions
   Compiled in 0.07s
    Running paper_test.main
  Discovered 1 suite(s).
  Analysed suites: No problems found.
  Running 1 suites...
  Suite When parsing a path into segments completed with 4 failure(s)
     Test an empty path has no segments failed with:
       Expected [""] to equal [].
       Actual:  [""]
       Expected:  []

     Test the root path has no segments failed with:
       Expected ["", ""] to equal [].
       Actual:  ["", ""]
       Expected:  []

     Test a single segment path is split failed with:
       Expected ["", "posts"] to equal ["posts"].
       Actual:  ["", "posts"]
       Expected:  ["posts"]

     Test nested segments are split in order failed with:
       Expected ["", "posts", "hello-world"] to equal ["posts", "hello-world"].
       Actual:  ["", "posts", "hello-world"]
       Expected:  ["posts", "hello-world"]
  4 of 4 test(s) failed.

In the console, the actual value will be printed in bold and red. Whereas the expected value will be in bold and green.

But why?

There is already a test-framework in Gleam. The gleeunit is even part of the official tool-chain, and when you scaffold a new project you get a placeholder test2 as well. Could I not just use that?

Sure. But I wanted to build something small where I could try out OTP (the concurrency features in Gleam and Erlang) and I wanted a test-framework that has matchers for common asserts and not only using Gleam's built in assert like gleeunit does.

There is absolutely nothing wrong with gleeunit, or with not using matchers. You can write good tests in either, it is just that I prefer matchers.

What makes a good test?

Writing good tests is quite hard, I think. When you have all the context loaded in your head and are writing the test it is a lot easier to spot a failing detail and immediately know what's wrong. Come back to the same test two years later (heck, even two weeks) and it is a lot harder!

I would say that a good test is 50/50 between reading the test code and reading the test failure message.

The test code should clearly communicate the intent3 of the test. The situation / context the test operates in, what the test does, and finally what the test expects to happen.

In Garanti this is communicated using the suite name, test name and matcher. The code is there as well, to give you the finer details. But ideally, the other things should describe what is being tested fairly well.

The other 50% then, well that is how well the test failure helps you to understand why something is wrong. Here I think clearly presenting the expected and actual values, and reducing that to what matters are very important.

Matchers are quite important for this I think. Instead of letting the test do the comparison and fail using a boolean or something, the matchers make the comparison and can then decide on how to reduce failure details.

The test then does not have to contain extra code for this and the failures get consistent throughout the code base.

These features are not novel to Garanti, and I do not claim them to be. I will not dwell on the theory of testing any longer, this was just a bit of a background that shaped the solution.

Under the hood

Let us open the hood of Garanti and have a look at how it is implemented.

The value of values

At the centre is this little type. This is what every test needs to return. Yes, return. Not panicking4.

pub type AssertionResult {
  Pass
  Fail(summary: String, expectations: List(Expectation))
  Timeout
}

That type has not been in any of the previous examples though. Let us zoom out. There are two types that you have seen, the Suite and the Test, they look as follows.

pub type Suite {
  Suite(name: String, tests: List(Test))
}

pub type Test {
  Test(name: String, run: fn() -> AssertionResult)
}

So, a suite is something with a name and a list of tests. And a test is just a name and an anonymous function that returns a result.

You have now seen 3 out of the 4 public types you would interact with when using Garanti to write tests.

Those matchers I have been mentioning are nothing more than a function that compares two values and produces an AssertionResult. Look:

pub fn to_be_equal(actual: a, expected: a) -> garanti.AssertionResult {
  case actual == expected {
    True -> garanti.Pass
    False ->
      garanti.Fail(...) // Keep reading for more details!
  }
}

A test knows nothing about how it is run, or how passes and failures are reported to the user. Their only responsibility is to produce this AssertionResult, they do not have to use any of Garanti's built-in matchers, they can bring their own, or even produce it on the fly if they want:

/// An always passing assert.
pub fn volkswagen_assert() -> garanti.AssertionResult {
  garanti.Pass
}

By using values like this I believe that it will be fairly easy to extend Garanti to support features like:

• Skipping tests.
• Retrying tests.
• Data-driven tests.
• Report results differently.

Without having to change matchers, runners, or anything else.

Running a test

Garanti is made up of three packages. garanti is the core, containing the types above, the built-in expect matchers and some shared code for reporting (that we will get to eventually). This package is not target specific, it contains only Gleam code.

garanti_erlang and garanti_javascript are the two runners, one per Gleam's target. Each one has a discovery mechanism to find the tests and a runner that, well, runs them.

Having two runners comes with both pros and cons.

Erlang target

Remember when I said that I wanted to build something using Gleam/Erlang OTP features? We will soon get to that part, but first, let us look at FFI, reflection and some LLM-assisted parts!

A suite is not registered. Like in many other languages, it is picked up by a runtime. So when you write gleam test, whatever tests you have should run. When you use the Erlang target in your Gleam project the compiler is kind enough to compile and load all modules under test/.

The runner, however, does need to be registered, similar to how gleeunit works.

In the example included in the Garanti repository the runner is registered like this:

// example/test/example_test.gleam

import garanti
import garanti_erlang/runner

pub fn main() -> Nil {
  runner.run(garanti.Debug)
}

The runner will, as a first thing, use the discovery module to call discover_all_suites() which will return a list of Suite.

The discovery code is full of FFI (Foreign Function Interface) calling Erlang code. FFI is Gleam's way of interacting with its target runtime, either Erlang or JavaScript.

1. Find all loaded test modules.
2. For each loaded module, find all exported functions with the _suite suffix.
3. For each such function, execute it and collect the returned Suite.

🤖 As I wrote in the README.md, I want to build this thing myself. BUT when it comes to writing Erlang, I folded and shelled out to an LLM for help.

Executing the Suite function during discovery (3) can fail. What can fail during discovery? Pretty much anything!

pub fn bad_suite() {
  // The test author might do some setup code here. This can be
  // convenient to do expensive computation that is shared between
  // tests.
  //
  // Do not worry! Gleam is an immutable language so the tests
  // cannot affect each other!

  // The code can however fail...
  panic as "This is rigged!"

  Suite("When the suite fails during discovery", [
    Test("this test will not run", fn() {
      expect.to_be_equal(True, True)
    })
  })
}

When such a thing inevitably happens, the failure will be wrapped in a Suite containing a single Test on the fly. This way, test discovery errors will be reported back to the user in the same way as any other test failure. After all, a Suite is just a value.

  ...
  [...] // A list of Suite
  |> list.map(fn(export) {
    case discovery_ffi.apply_suite(module_name, export.name) {
      Ok(suite) -> suite
      Error(reason) -> failed_suite(module_name, export.name, reason)
    }
  })
}

fn failed_suite(
  module_name: String,
  function_name: String,
  reason: String,
) -> Suite {
  Suite(module_name <> "." <> function_name, [
    Test(function_name, fn() { garanti.Fail(reason, []) }),
  ])
}

Messages everywhere!

Gleam has actors, these can run isolated and concurrently, and only interact by sending messages. Gleam is standing on the shoulders of Erlang here, and Erlang has a well proven history of building rock-solid systems using OTP.

First, an actor that has the responsibility to collect test results and to wait for all suites to run is started. This is the console_reporter.

Then the runner starts all suites. Each suite runs as a suite actor where it immediately fans out and spawns new Erlang processes for each test to run in. These test processes are unlinked to the suite, meaning that if a test crashes, it does not bring the suite down.

The individual test is being executed by the executor which is another unlinked process that runs the test function using a simple FFI call. The FFI is an Erlang function that runs the code in an Erlang try/catch, so that a crashing test does not print an Erlang stack-trace but exits normally.

The test process monitors the unlinked process executing the test. If it fails, the test process gets back an ExecutionFailure. If the test times out, it gets an ExecutionTimeout instead. Currently, Garanti is running the tests with a firm hand, only allowing for 1 second5 (wall-clock time) until timing out.

The suite actor keeps track of when all tests have been run. Each test execution results in a TestComplete message being sent to the suite actor itself, where the results are being collected. Once all tests have completed, the suite actor sends a SuiteComplete to the reporting actor.

That was a lot of details, actors, processes, messages, linked/unlinked and what not.

Yes, but if you look at the linked code you can see that each module is pretty small and that each module has a specific responsibility.

Here is my attempt6 to illustrate this!

                          console_reporter
                  ↑                          ↑
                  | SuiteComplete            | SuiteComplete
                suite A                    suite B
                  | spawns                   | spawns
           ┌─────────────┐             ┌─────────────┐
           ↓             ↓             ↓             ↓
       test proc.    test proc.    test proc     test proc
          Pass         Fail          Timeout      panics

• The console_reporter receives SuiteComplete or SuiteCancelled and prints the result to the console.
• The suite actor sends SuiteComplete or SuiteCancelled to the console_reporter based on the test process result.
• The test process converts its result into a TestComplete message and sends that to the suite actor.

Actors and processes are the way to do asynchronous or parallel execution in Gleam (the only way, I believe, when you target Erlang). That is why the suite actor runs the test process in a separate process. That way the suite is kicked off and when the test process returns a result, the suite will send itself a message. When that message is handled and the tally over the number of tests executed matches the expected number of tests, it will send a message to the console_reporter.

When you are used to imperative programming with async / await or futures with callbacks, this is a bit confusing at first. But when you get the messages aligned, it is a beautiful pattern!

JavaScript target

When targeting JavaScript the exact same Suite and Tests are run. The discovery and execution are however completely different.

Initially, I did not consider the JavaScript target something7 that Garanti should support. I wanted to use OTP, and the project I used to dog-food this in was only targeting Erlang. Well, until it wasn't. Then I decided to add a simple JavaScript runner as well.

The JavaScript discovery is inspired by gleeunit. It works similar to the Erlang way. It traverses the compiled test files, finds any exported function with the _suite suffix and runs it to collect the suites.

Unlike Erlang, JavaScript runs everything single-threaded. It basically maps over the list of suites, which maps the list of tests to a result that is passed to the JavaScript reporter. There is some support to convert Promises to callbacks, but I have not investigated further if this would allow for concurrent execution.

The upside is that I can now test Gleam projects that target JavaScript, and the resulting runner package (garanti_javascript) became quite small.

The flipside is that there is no concurrency, no test isolation (try/catch is there, but no real isolation), no timeout. There is no way to preempt a synchronous call that loops forever.

TestResult to ANSI colours

The tests all return an AssertionResult. The variant used for failures is Fail(summary: String, expectations: List(Expectation)).

Each matcher has the responsibility to encode the failure using the variants of Expectation to provide semantic details on what was expected. The type looks like this:

pub type Expectation {
  /// The expected value described as a string.
  Expected(String)
  /// The actual value described as a string.
  Actual(String)
  /// The value NOT expected described as a string.
  NotExpected(String)
  /// When something is missing from an expected volume.
  Missing(String)
  /// When something is extra for (not expected to be part of) an expected volume.
  Extra(String)
}

Here is how it is used, filling out the details from the omitted failure handling from the previous example of the to_be_equal matcher function.

garanti.Fail(
  string.concat([
    "Expected ",
    string.inspect(actual),
    " to equal ",
    string.inspect(expected),
    ".",
  ]),
  [
    garanti.Actual(string.inspect(actual)),
    garanti.Expected(string.inspect(expected)),
  ],
)

Remember how the reporter got the entire suite's test results as List(TestResult)? That list is transformed into a List(report.Message). And a Message is, again, just a value.

pub type Message {
  Message(level: Level, tokens: List(Token))
}

pub type Token {
  Plain(text: String)
  Enriched(text: String, effects: List(Effect))
  Indent
  Block(text: String)
  NewLine
}

pub type Effect {
  Positive
  Negative
  Important
  Name
  Bold
  Secondary
}

Notice how there still are no colours anywhere among those types? These are still only semantic values, that can contain effects to better describe them.

The console is the only part of Garanti that knows about printing to the console, using ANSI colours, \n and a whole lot of string concatenations.

Here is the pipeline:

    List(TestResult)   Pass, Fail, ...
            |
            ↓
describer transforms into
            |
            ↓
     List(Message)     Name, Positive/Negative, ...
            |
            ↓
  console transforms into
            |
            ↓
         String        "\u{001b}[32m" <> "..."

Presenting results is yet another "everything is a value" reasoning!

Composable

I am actually quite happy with the design of this little test-framework of mine!

Each part is cohesive, only responsible for one thing, and unaware of how other parts of the program operate. Using values for everything from Suite, Test, TestResult, to the Positive effect of an Enriched text has made it simple to change along the way.

My favourite part is the TestResult though, making it very easy to write your own matchers and have them well integrated in the rest of the test running and reporting.

Are two runners worth it?

It is still early days for Garanti, and having two runners is complicating the usage. The consumer must select a runtime, even if their project has no FFI and they really do not care. This is unfortunate.

Maybe I should kill my darlings (actors) and turn a blind eye to concurrency and timeouts? Define a single FFI function that is then implemented in both Erlang and JavaScript. I did have my fun with the OTP already, I don't know. If you have any good thoughts, please, let me know.

The fine print

This is my second time using actors in Gleam. I have no previous experience of Erlang or OTP. I have used actors in Akka, a Scala library though, but that is slightly different. Anyhow, I might have missed something or made mistakes here, so please do not take me as an authority on OTP! And if you spot a mistake, or an improvement, do not be afraid to tell me. Either open an issue at GitHub, or email me, please.

Next step?

As I said, I am dog-fooding this in another project of mine. This helps to spot missing features, provide better messages etc. Time will tell if Garanti proves to be something to hold on to, but so far I am happy with it, and I built it for me after all.

The next features I am thinking about:

• Combining multiple matchers into a single AssertionResult to do things like asserting that a value is less than 100 AND greater than 0 in the same assertion.
• Data-driven tests where you can list inputs to run over the test, e.g. 2, 4, 6 is even.
• Collect and display failed tests at the end of the report.

I hope this has inspired you to try out Garanti, Gleam, OTP, or to write your very own test-framework! Whenever I get feedback from a reader my day gets instantly better, so don't be shy, drop me an email with your thoughts.

If you want to give Garanti a spin, just head over to github.com/eliasson/garanti!

Thank you for reading!