Filtering ScalaTest tests by tags in sbt

ScalaTest lets you tagging your tests with arbitrary tags and then filtering by tags when executing tests.

It’s not immediately obvious though, how to make use of this in sbt, at least I couldn’t find an answer on the very first page in Google or StackOverflow, and I remember myself looking for a complete solution for quite a while, so here it is.

For example, we want to test Kubernetes integration and some of our tests depend on Kubernetes instance up and running. Such tests also can’t be executed in parallel.

As defined in ScalaTest documentation[1], we create a tag and use it to tag our tests:

package.scala

object myapp {
  object K8STest extends Tag("myapp.k8s")
}

KubernetesServiceSpec.scala

class KubernetesServiceSpec extends AsyncFuncSpec {
  it("should connect to kubernetes", K8STest) { 1
    ...
  }
}

test is tagged with K8STest tag

Selecting only tests with certain tags in ScalaTest is possible by passing additional arguments to its test runner[2]. So immediately we are able to include or exclude certain tests when using sbt's testOnly task:

> testOnly * -- -n myapp.k8s

> testOnly * -- -l myapp.k8s

This, obviuosly, is not convenient. Call it a whim, but I’d like to stick to the test task which would execute isolated tests, and it would be great to have a dedicated task to execute tests that depend on environment.

Luckily, the first whim is easy to satisfy, as sbt allows us to specify arguments to pass to test runner[3] when executing test task. There are many ways how the second goal can be achieved in sbt, my preference is to declare a separate configuration for this.

Here’s the complete solution:

build.sbt

lazy val K8STest = config("k8s") extend Test 1

lazy val myapp = (project in file(".")).
  configs(K8STest). 2
  settings(
    name := "myapp",
    ...
    testOptions in Test := Seq(Tests.Argument("-l", "myapp.k8s")), 3

    inConfig(K8STest)(Defaults.testTasks), 4
    testOptions in K8STest := Seq(Tests.Argument("-n", "myapp.k8s")), 5
    parallelExecution in K8STest := false, 6
  )

declare new configuration k8s that extends from Test configuration. Extending means, that this configuration will delegate to Test configuration when missing definitions are requested
add new configuration to the project
exclude k8s-tagged tests when executing test task in Test configuration
(re)define test tasks (test, testOnly, etc.) for the K8STest configuration. Note, that for other tasks, such as compile, K8STest will still delegate to Test. If we didn’t do this, test would be delegated to Test and hence entire test runner configuration would be reused
include only k8s-tagged tests when executing test task in K8STest configuration
any other configuration options can be adjusted as needed

We now have 2 tasks with short and self-explanatory names, and also it is easy to fine-tune execution of each task without affecting the other:

> test

> k8s:test

We can take this as far as imagination goes: adding all:test for convenience, adding global setup and cleanup functions for each type of tests[4]…

Make sure to look through the whole Testing section in sbt documentation[5] for more ideas and interesting examples.