Tests and suites

Overview

TestBalloon has a DSL-based API with two core functions: testSuite and test.

• Tests are functions which either succeed or fail (throw). Tests contain your assertions. Code inside a test can suspend.

• Test suites structure your tests. They can nest across multiple levels. Code inside a test suite registers tests, test suites, and fixtures.

• You create top-level test suites as properties with a by testSuite delegation. TestBalloon will find them.

Tests and test suites accept strings as names.

Note

You have now learned TestBalloon's DSL API. The rest on this page is plain Kotlin.

But please familiarize yourself with Green code and blue code and TestBalloon's golden rule.

val ExampleTests by testSuite { // (1)!
    test("string length") { // (2)!
        assertEquals(8, "Test me!".length) // (3)!
    }

    testSuite("integer operations") { // (4)!
        test("max") {
            assertEquals(5, max(5, 3))
        }

        test("min") {
            delay(10.milliseconds) // (5)!
            assertEquals(3, min(5, 3))
        }
    }
}

1. Registers a top-level test suite. TestBalloon will automatically use the property's fully qualified name unless you provide an explicit name.
2. Registers a test.
3. An assertion from kotlin-test.
4. Registers a nested test suite.
5. A suspend function call.

Custom functions for tests and test suites

You can define your own types of tests and test suites, like this test variant with an iterations parameter:

fun TestSuite.test(
    name: String,
    iterations: Int,
    action: suspend TestExecutionScope.() -> Unit
) = test(name) {
    for (iteration in 1..iterations) {
        action()
    }
}

Tip

While the above code creates a custom test function, you can do the same with a test suite.

Find more details under Configuration.

Parameterized tests and test suites

In a test suite, you can use all Kotlin constructs (variable scopes, conditions, loops) to create tests and child test suites dynamically.

val ParameterizedTests by testSuite {
    val testCases = mapOf(
        "one" to 3,
        "two" to 3,
        "three" to 5
    )

    for ((string, expectedLength) in testCases) {
        test("length of '$string' is $expectedLength") {
            assertEquals(expectedLength, string.length)
        }
    }
}

Parameterization works across test suites:

val UserTest by testSuite {
    for (invalidUserName in listOf("", "a", "+", "+foo")) {
        testSuite("User name '$invalidUserName'") {
            for (role in User.Role.entries) {
                test("is invalid with role '$role'") {
                    assertFailsWith<IllegalArgumentException> {
                        User(invalidUserName, role)
                    }
                }
            }
        }
    }
}

Using value sources (generators)

Value sources help cover test edge cases and/or random samples without repetitive boilerplate code:

testSuite("Accepted counts") {
    val samples = buildList {
        addAll(listOf(0, 1, Int.MAX_VALUE)) // (1)!
        val randomSampleSource = Random(42) // (2)!
        repeat(20) { add(randomSampleSource.nextInt(0, Int.MAX_VALUE)) }
    }

    for (count in samples) {
        test("count $count is accepted") {
            service.updateTransactionCount(count) // should not throw
        }
    }
}

1. Edge cases.
2. Generating repeatable pseudo-random values with a seed.

Green code and blue code

Info

For a DSL-based framework, it is paramount to be aware of the effects of closures and variable capturing.

Runtime phases

TestBalloon has two primary runtime phases:

graph LR
    A("`**Test registration**<br/>(always sequential)`") --> B("`**Test execution**<br/>(sequential or concurrent)`")

The test registration phase is part of TestBalloon's setup: It creates the test element hierarchy, registering test suites, tests and fixtures, and configuring test elements. At the end, it knows exactly what to run. (1)

1. TestBalloon is fast to register tests. It can register and configure 1.7 million tests in 7 seconds on a decent Laptop. It also knows shortcuts if only parts of the test element hierarchy have been selected.

Info

The test registration phase always completes before the test execution phase starts.

The test execution phase is where the action is. While the default is to run tests sequentially, concurrent execution can be configured at any level of the test element hierarchy.

Blue code (registration phase)

In TestBalloon, all code outside the lambdas of tests, fixtures and execution wrappers is registration-phase code, or blue code:

val ParameterizedTests by testSuite {
    val testCases = mapOf(
        "one" to 3,
        "two" to 3,
        "three" to 5
    )

    for ((string, expectedLength) in testCases) {
        test("length of '$string' is $expectedLength") {
            assertEquals(expectedLength, string.length)
        }
    }
}

Green code (execution phase)

Code inside the lambdas of tests, fixtures and execution wrappers is execution-phase code, or green code:

val ParameterizedTests by testSuite {
    val testCases = mapOf(
        "one" to 3,
        "two" to 3,
        "three" to 5
    )

    for ((string, expectedLength) in testCases) {
        test("length of '$string' is $expectedLength") {
            assertEquals(expectedLength, string.length)
        }
    }
}

TestBalloon's golden rule

With power comes some responsibility. The golden rule is:

Danger

Never leak mutable state from blue code into green code.

Why is this important?

❌ This breaks

testSuite("Broken calculator test suite") {
    val calculator = Calculator()
    val operands = listOf(7, 23, 15)
    var sum = 0 // (1)

    for (operand in operands) {
        sum += operand // (2)
        test("add $operand, expect $sum") {
            calculator.add(operand)
            assertEquals(sum, calculator.result) // (3)
        }
    }
}

1. sum is mutable state.
2. sum is mutated in blue code.
3. sum leaks into green code.

The above code will produce failing tests:

Why? All the additions to sum occurred in blue code before any green code would start. So green code inside tests can only see the latest blue-code state of sum, which is 45.

✅ This works

testSuite("Healthy calculator test suite") {
    val calculator = Calculator()
    val operandsAndSums = buildList { // (1)
        val operands = listOf(7, 23, 15)
        var sum = 0
        for (operand in operands) {
            sum += operand
            add(operand to sum)
        }
    }

    for ((operand, sum) in operandsAndSums) {
        test("add $operand, expect $sum") {
            calculator.add(operand) // (2)
            assertEquals(sum, calculator.result) // (3)
        }
    }
}

1. Blue code contains immutable state only.
2. Well, not entirely: The calculator is mutable. This works, because it is not mutated in blue code, but only in green code.
3. Using immutable state originating in blue code is safe.

Here, all operands and expected sum values were created as immutable state in blue code. This is always safe.

Info

State leaks with closures, which run later or concurrently, can happen anywhere (flows, sequences, coroutines, anything lazy). For this reason, YouTrack issue KT-15514 suggests to make the compiler emit a warning in such cases.