Skip to content

Effective testing

Using TestBalloon's powers, how can we test better with less effort? This section offers guidance for typical scenarios.

Expressive names🔗

Make tests communicate their purpose:

test("User '$user' must be signed in") {
    // ...
}

Write once, test many🔗

Let immutable state describe and drive your test cases:

val ParameterizedTests by testSuite {
    val testCases = mapOf(
        "one" to 3,
        "two" to 3,
        "three" to 5
    )

    for ((string, expectedLength) in testCases) {
        test("length of '$string' is $expectedLength") {
            assertEquals(expectedLength, string.length)
        }
    }
}

Cover edge cases and samples🔗

Test edge cases and/or random samples with value sources (generators):

testSuite("Accepted counts") {
    val samples = buildList {
        addAll(listOf(0, 1, Int.MAX_VALUE)) // (1)!
        val randomSampleSource = Random(42) // (2)!
        repeat(20) { add(randomSampleSource.nextInt(0, Int.MAX_VALUE)) }
    }

    for (count in samples) {
        test("count $count is accepted") {
            service.updateTransactionCount(count) // should not throw
        }
    }
}
  1. Edge cases.
  2. Generating repeatable pseudo-random values with a seed.

One test per sample documents which tests were actually run:

Generative test results Generative test results

Supply fresh state to multiple tests🔗

To conveniently provide each test with fresh state, available as a context via this, use a fixture and provide its value as a context for each test:

testSuite("Multiple tests with fresh state") {
    testFixture {
        Service().apply {
            signIn(userName = "siobhan", password = "ask") // (1)!
        } // (2)!
    } closeWith {
        signOut() // (3)!
    } asParameterForEach {
        test("deposit") { service ->
            service.deposit(Amount(20.0))
            assertEquals(Amount(40.99), service.accountBalance())
        }

        test("withdraw") { service ->
            service.withdraw(Amount(20.0))
            assertEquals(Amount(0.99), service.accountBalance())
        }
    }
}
  1. signIn can be a suspending function.
  2. If you want to provide multiple values, use an object expression inside the fixture and asContextForEach.
  3. signOut may also suspend.

Tip

In this case, tests are fully isolated from each other, and don't need a TestScope. They are ideal candidates for concurrent execution.

Use shared state across multiple tests🔗

To conveniently share state among tests, use a fixture's value as a shared context for all tests:

testSuite("Multiple tests sharing state") {
    testFixture {
        object {
            val service = Service().apply {
                signIn(userName = "siobhan", password = "ask")
            }
            var expectedCount = 0 // (1)!
        }
    } closeWith {
        service.signOut()
    } asContextForAll {
        test("deposit") {
            service.deposit(Amount(20.0))
            assertEquals(Amount(40.99), service.accountBalance())
            assertEquals(++expectedCount, service.transactionCount())
        }

        test("withdraw") {
            service.withdraw(Amount(20.0))
            assertEquals(Amount(20.99), service.accountBalance())
            assertEquals(++expectedCount, service.transactionCount())
        }
    }
}
  1. We can use mutable state here. This is green code which exists exclusively at test execution time, preserving TestBalloon's golden rule.

Tip

Writing tests that build on each other is easy, because, by default, TestBalloon runs tests in the order they appear in the source. Just make sure that you don't configure concurrent execution for them.

Make tests run fast🔗

…if all tests avoid non-local mutable state🔗

If you have a module where all tests only mutate local state(1) and don't need a TestScope, you can speed up test execution greatly by running them concurrently. To do so, put this declaration anywhere in your test module:

  1. Ascertain that tests do not share mutable state among each other and do not access global mutable state.
class ConcurrentTestSession :
    TestSession(
        defaultCompartment = { TestCompartment.Concurrent } // (1)!
    )
  1. For technical reasons, a compartment assignment must be done lazily.

…if most tests avoid non-local mutable state🔗

  1. Configure the module's test session for concurrency:

    class ConcurrentTestSession :
    TestSession(
        defaultCompartment = { TestCompartment.Concurrent } // (1)!
    )
    1. For technical reasons, a compartment assignment must be done lazily.

  2. Put top-level test suites, whose test's access non-local mutable state or need a TestScope, in the predefined Sequential compartment:

    val TestsSharingMutableState by testSuite(
    compartment = { TestCompartment.Sequential } // (1)!
) {
    // ...
}
    1. For technical reasons, a compartment assignment must be done lazily.

TestBalloon will now execute tests in the Sequential compartment sequentially, and also isolate them from all concurrent tests.

…if only some tests can run concurrently🔗

Put top-level test suites, whose test's can run concurrently and don't need a TestScope, in the predefined Concurrent compartment:

val ConcurrentTests by testSuite(
    compartment = { TestCompartment.Concurrent } // (1)!
) {
    // ...
}
  1. For technical reasons, a compartment assignment must be done lazily.

TestBalloon will now execute most tests sequentially (by default), and isolate them from those in the Concurrent compartment, where they run concurrently.

A UI test with Jetpack Compose🔗

TestBalloon does not bundle Compose dependencies, but it does provide a JUnit4RulesContext to create test-level fixtures supporting JUnit 4 rules.

With it, you can use Jetpack Compose tests inside TestBalloon via composeTestRule, as shown in the Google documentation:

val JetpackComposeWithTestBalloon by testSuite {
    testFixture {
        object : JUnit4RulesContext() { // (1)!
            val composeTestRule = rule(createComposeRule()) // (2)!
            val myCustomRule = rule(myCustomRule())
        }
    } asContextForEach {
        test("click") {
            composeTestRule.setContent {
                ComposableUnderTest()
            }

            composeTestRule.onNodeWithText("Button").performClick()
            composeTestRule.onNodeWithText("Success").assertExists()
        }
    }
}
  1. Deriving a fixture value from JUnit4RulesContext enables support for JUnit 4 rules.
  2. Instead of annotations, use the rule() function to register a TestRule.

See complete code in this Jetpack Compose test example.

A UI test with Compose Multiplatform🔗

Compose Multiplatform provides an experimental runComposeUiTest() API. To use it with TestBalloon, create a custom DSL function like this:(1)

  1. Using the Compose Multiplatform test API requires an opt-in directive like @file:OptIn(ExperimentalTestApi::class).
@TestRegistering
fun TestSuiteScope.composeTest(name: String, action: suspend ComposeUiTest.() -> Unit) = test(name) {
    @OptIn(TestBalloonExperimentalApi::class) // required for TestBalloon's testTimeout
    runComposeUiTest(
        runTestContext = coroutineContext.minusKey(CoroutineExceptionHandler.Key),
        testTimeout = testTimeout ?: 60.seconds
    ) {
        action()
    }
}

With that, you can use Compose Multiplatform tests inside TestBalloon as shown in the JetBrains documentation.(1)

  1. Using the Compose Multiplatform test API requires an opt-in directive like @file:OptIn(ExperimentalTestApi::class).
val ComposeMultiplatformTests by testSuite {
    composeTest("click") {
        setContent {
            ComposableUnderTest()
        }

        onNodeWithText("Button").performClick()
        onNodeWithText("Success").assertExists()
    }
}

Handling flaky tests🔗

One way of handling flaky tests is to repeat them until they succeed.

Create a TestConfig extension:

fun TestConfig.repeatOnFailure(maxRepetitions: Int) = aroundEachTest { action ->
    var lastException: Throwable? = null
    repeat(maxRepetitions) {
        try {
            action()
            return@aroundEachTest
        } catch (exception: Throwable) {
            lastException = exception
            // suppress as long as we try repeatedly
        }
    }
    throw lastException!!
}

Use it like this:

val FlakyTests by testSuite {
    testSuite("not controlled") {
        test("would succeed after 3 failures") {
            doSomethingFlaky()
        }
    }

    testSuite("under control", testConfig = TestConfig.repeatOnFailure(5)) {
        test("succeeds after 3 failures") {
            doSomethingFlaky()
        }

        test("always fails") {
            throw Error("always failing")
        }
    }
}

The outcome:

Flaky test results Flaky test results

Conditional tag-based testing🔗

TestBalloon provides the option of using environment variables to control test execution on all Kotlin targets.(1)

  1. JS browsers and Android (emulated or physical) devices do not natively support environment variables. TestBalloon provides a (simulated) environment for those. For Android device-side tests, you need to set them via instrumentation arguments. For JS browsers, you need to declare them as browser-safe.

If you define tags(1) and a TestConfig extension like this,

  1. These are your tags, literally, in plain Kotlin, instead of some complex pre-defined tag regime.
enum class MyTag {
    CI,
    SimulatedCI,
    Release;

    fun value() =
        testPlatform.environment("TEST_TAGS")?.split(',')?.last { it == name }

    fun exists() = value() != null
}

fun TestConfig.onlyIfTagged(vararg tags: MyTag) =
    if (tags.any { it.exists() }) this else disable()

…you can use a TEST_TAGS environment variable to conditionally run tests and suites at any level of the test element hierarchy:

val ConditionalTests by testSuite(
    testConfig = TestConfig.onlyIfTagged(MyTag.CI, MyTag.SimulatedCI)
) {
    // ...
}

Use a temporary directory (and keep it on failures)🔗

Some tests require a temporary directory. Let's suppose you want to keep it for inspection if there were test failures – but not on CI. Let's also require the directory name to identify the test by default.

Create a test fixture for the directory, deleting it at the end of its lifecycle (its test or test suite):

fun TestSuiteScope.temporaryDirectoryFixture(
    prefix: String = "${testSuiteInScope.testElementPath}-" // (1)!
) = testFixture {
    Files.createTempDirectory(Path("build/tmp"), prefix) // (2)!
} closeWith { testsSucceeded ->
    if (testsSucceeded || testPlatform.environment("CI") != null) {
        deleteRecursively()
    } else {
        println("Temporary directory: file://${toAbsolutePath()}") // (3)!
    }
}
  1. Prefixes the directory with a name identifying the test suite.
  2. The directory (a Path) is the fixture's value.
  3. Show the path on failures for easy inspection.

If you need a directory per test, use it like this:

testSuite("temporary directory per test") {
    temporaryDirectoryFixture().asParameterForEach {
        test("one") { directory ->
            (directory / "my-result1.txt").writeText("one")
        }

        test("two") { directory ->
            (directory / "my-result2.txt").writeText("two")
        }
    }
}

If you need a directory per test suite, use it like this:(1)

  1. You can also use the others variants like asParameterForAll.
testSuite("temporary directory per suite (invokable val)") {
    val directory = temporaryDirectoryFixture()

    test("one") {
        (directory() / "my-result1.txt").writeText("one")
    }

    test("two") {
        (directory() / "my-result2.txt").writeText("two")
    }
}