Effective testing

Using TestBalloon's powers, how can we test better with less effort? This section offers guidance for typical scenarios.

Expressive names

Make tests communicate their purpose:

test("User '$user' must be signed in") {
    // ...
}

Write once, test many

Let immutable state describe and drive your test cases:

val ParameterizedTests by testSuite {
    val testCases = mapOf(
        "one" to 3,
        "two" to 3,
        "three" to 5
    )

    for ((string, expectedLength) in testCases) {
        test("length of '$string' is $expectedLength") {
            assertEquals(expectedLength, string.length)
        }
    }
}

Cover edge cases and samples

Test edge cases and/or random samples with value sources (generators):

testSuite("Accepted counts") {
    val samples = buildList {
        addAll(listOf(0, 1, Int.MAX_VALUE)) // (1)!
        val randomSampleSource = Random(42) // (2)!
        repeat(20) { add(randomSampleSource.nextInt(0, Int.MAX_VALUE)) }
    }

    for (count in samples) {
        test("count $count is accepted") {
            service.updateTransactionCount(count) // should not throw
        }
    }
}

1. Edge cases.
2. Generating repeatable pseudo-random values with a seed.

One test per sample documents which tests were actually run:

Supply fresh state to multiple tests

Suppose a number of tests needs fresh state like this:

val service = Service().apply {
    signIn(userName = "siobhan", password = "ask") // (1)!
}

1. Note that signIn can be a suspending function.

To conveniently provide each test with that fresh state, available as a context via this, define a custom DSL function:

testSuite("Multiple tests with fresh state") {
    @TestRegistering // (2)!
    fun test(name: String, action: suspend Service.() -> Unit) = // (3)!
        this.test(name) {
            val service = Service().apply {
                signIn(userName = "siobhan", password = "ask") // (1)!
            }
            service.action() // (4)!
            // cleanup...
        }

    test("deposit") {
        deposit(Amount(20.0)) // (5)!
        assertEquals(Amount(40.99), accountBalance()) // (6)!
    }

    test("withdraw") {
        withdraw(Amount(20.0))
        assertEquals(Amount(0.99), accountBalance())
    }
}

1. signIn can be a suspending function.
2. Tell the IDE plugin to put test run gutters at the function's call sites.
3. Redefine the test() function locally to use a service as an extension receiver.
4. Provide the context to each test action.
5. deposit() is a method of the Service extension receiver.
6. accountBalance() is a method of the Service extension receiver.

Tip

In this case, tests are fully isolated from each other. They are ideal candidates for concurrent execution.

Use shared state across multiple tests

To conveniently share state among tests, use a fixture and define a custom DSL function providing it as a context:

testSuite("Multiple tests sharing state") {
    class Context { // (1)!
        val service = Service().apply {
            signIn(userName = "siobhan", password = "ask")
        }
        var expectedTransactionCount = 0 // (2)!
    }

    val context = testFixture { Context() }

    @TestRegistering // (3)!
    fun test(name: String, action: suspend Context.() -> Unit) = // (4)!
        this.test(name) { context().action() } // (5)!

    test("deposit") {
        service.deposit(Amount(20.0))
        assertEquals(Amount(40.99), service.accountBalance())
        assertEquals(++expectedTransactionCount, service.transactionCount())
    }

    test("withdraw") {
        service.withdraw(Amount(20.0))
        assertEquals(Amount(20.99), service.accountBalance())
        assertEquals(++expectedTransactionCount, service.transactionCount())
    }
}

1. Use a local class to define a test-specific context.
2. We can use mutable state here. This is green code which exists exclusively at test execution time, preserving TestBalloon's golden rule.
3. Tell the IDE plugin to put test run gutters at the function's call sites.
4. Redefine the test function locally to use the test-specific context.
5. Provide the fixture as a context to each test action.

Tip

Writing tests that build on each other is easy, because, by default, TestBalloon runs tests in the order they appear in the source. Just make sure that you don't configure concurrent execution for them.

Make tests run fast

…if all tests avoid non-local mutable state

If you have a module where all tests only mutate local state(1), you can speed up test execution greatly by running them concurrently. To do so, put this declaration anywhere in your test module:

1. Ascertain that tests do not share mutable state among each other and do not access global mutable state.

class ConcurrentTestSession :
    TestSession(
        defaultCompartment = { TestCompartment.Concurrent } // (1)!
    )

1. For technical reasons, a compartment assignment must be done lazily.

…if most tests avoid non-local mutable state

1. Configure the module's test session for concurrency:

   class ConcurrentTestSession :
       TestSession(
           defaultCompartment = { TestCompartment.Concurrent } // (1)!
       )
   

   1. For technical reasons, a compartment assignment must be done lazily.

2. Put top-level test suites, whose test's access non-local mutable state, in the predefined Sequential compartment:

   val TestsSharingMutableState by testSuite(
       compartment = { TestCompartment.Sequential } // (1)!
   ) {
       // ...
   }
   

   1. For technical reasons, a compartment assignment must be done lazily.

TestBalloon will now execute tests in the Sequential compartment sequentially, and also isolate them from all concurrent tests.

…if only some tests can run concurrently

Put top-level test suites, whose test's can run concurrently, in the predefined Concurrent compartment:

val ConcurrentTests by testSuite(
    compartment = { TestCompartment.Concurrent } // (1)!
) {
    // ...
}

1. For technical reasons, a compartment assignment must be done lazily.

TestBalloon will now execute most tests sequentially (by default), and isolate them from those in the Concurrent compartment, which run concurrently.

A UI test with Jetpack Compose

TestBalloon does not bundle Compose dependencies, but it does provide a testWithJUnit4Rule() function. With that, you can create a custom DSL function:

@TestRegistering
@OptIn(TestBalloonExperimentalApi::class) // required for testWithJUnit4Rule
fun TestSuite.composeTest(
    name: String,
    composeTestRule: ComposeContentTestRule = createComposeRule(),
    action: suspend ComposeTestContext<ComposeContentTestRule>.() -> Unit
) = testWithJUnit4Rule(name, composeTestRule) {
    ComposeTestContext(composeTestRule).action()
}

class ComposeTestContext<Rule>(val composeTestRule: Rule)

Having done this, you can use Jetpack Compose tests inside TestBalloon via composeTestRule, as shown in the Google documentation:

val JetpackComposeTests by testSuite {
    composeTest("click") {
        composeTestRule.setContent {
            ComposableUnderTest()
        }

        composeTestRule.onNodeWithText("Button").performClick()
        composeTestRule.onNodeWithText("Success").assertExists()
    }
}

For a complete example, see Jetpack Compose UI test.

A UI test with Compose Multiplatform

Compose Multiplatform provides an experimental runComposeUiTest() API. To use it with TestBalloon, create a custom DSL function like this:(1)

1. Using the Compose Multiplatform test API requires an opt-in directive like @file:OptIn(ExperimentalTestApi::class).

@TestRegistering
fun TestSuite.composeTest(name: String, action: suspend ComposeUiTest.() -> Unit) = test(name) {
    @OptIn(TestBalloonExperimentalApi::class) // required for TestBalloon's testTimeout
    runComposeUiTest(
        runTestContext = coroutineContext.minusKey(CoroutineExceptionHandler.Key),
        testTimeout = testTimeout ?: 60.seconds
    ) {
        action()
    }
}

With that, you can use Compose Multiplatform tests inside TestBalloon as shown in the JetBrains documentation.(1)

1. Using the Compose Multiplatform test API requires an opt-in directive like @file:OptIn(ExperimentalTestApi::class).

val ComposeMultiplatformTests by testSuite {
    composeTest("click") {
        setContent {
            ComposableUnderTest()
        }

        onNodeWithText("Button").performClick()
        onNodeWithText("Success").assertExists()
    }
}

Handling flaky tests

One way of handling flaky tests is to repeat them until they succeed.

Create a TestConfig extension:

fun TestConfig.repeatOnFailure(maxRepetitions: Int) = aroundEachTest { action ->
    var lastException: Throwable? = null
    repeat(maxRepetitions) {
        try {
            action()
            return@aroundEachTest
        } catch (exception: Throwable) {
            lastException = exception
            // suppress as long as we try repeatedly
        }
    }
    throw lastException!!
}

Use it like this:

val FlakyTests by testSuite {
    testSuite("not controlled") {
        test("would succeed after 3 failures") {
            doSomethingFlaky()
        }
    }

    testSuite("under control", testConfig = TestConfig.repeatOnFailure(5)) {
        test("succeeds after 3 failures") {
            doSomethingFlaky()
        }

        test("always fails") {
            throw Error("always failing")
        }
    }
}

The outcome:

Conditional tag-based testing

TestBalloon provides the option of using environment variables to control test execution on all Kotlin targets.(1)

1. JS browsers and Android (emulated or physical) devices do not natively support environment variables. TestBalloon provides a (simulated) environment for those. For Android device tests, you need to set them via instrumentation arguments. For JS browsers, you need to declare them as browser-safe.

If you define tags(1) and a TestConfig extension like this,

1. These are your tags, literally, in plain Kotlin, instead of some complex pre-defined tag regime.

enum class MyTag {
    CI,
    SimulatedCI,
    Release;

    fun value() =
        testPlatform.environment("TEST_TAGS")?.split(',')?.last { it == name }

    fun exists() = value() != null
}

fun TestConfig.onlyIfTagged(vararg tags: MyTag) =
    if (tags.any { it.exists() }) this else disable()

…you can use a TEST_TAGS environment variable to conditionally run tests and suites at any level of the test element hierarchy:

val ConditionalTests by testSuite(
    testConfig = TestConfig.onlyIfTagged(MyTag.CI, MyTag.SimulatedCI)
) {
    // ...
}