Reference documentation for Skippy version 0.0.19.

New to Skippy? The best way to get started are the introductory tutorials:

Table of Contents

Overview

What is it?

Skippy is a Test Impact Analysis & Predictive Test Selection framework for the JVM. It cuts down on unnecessary testing and flakiness without compromising the integrity of your builds. You can run it from the command line, your favorite IDE and continuous integration server. Skippy supports Gradle, Maven, JUnit 4 and JUnit 5.

Skippy is specifically designed to prevent regressions in your codebase. It supports all types of tests where the tests and the code under test run in the same JVM. It is best suited for deterministic tests, even those prone to occasional flakiness. It provides the most value for test suites that are either slow or flaky (regardless of whether the test suite contains unit, integration, or functional tests).

What is it not?

Skippy is not designed for tests that assert the overall health of a system. Don’t use it for tests you want to fail in response to misbehaving services, infrastructure issues, etc.

Highlights

  • Support for Gradle & Maven
  • Support for JUnit 4 & JUnit 5
  • Lightweight: Use it from the command line, your favorite IDE and CI server
  • Non-invasive: Use it for a single test, your entire suite and anything in-between
  • Free of lock-in: You can go back to a “run everything” approach at any time
  • Open Source under Apache 2 License

Compatibility

Compatibility matrix for Skippys’s integration with Java, JUnit, Gradle, Maven and JaCoCo. Versions not mentioned here might work, but there is no guarantee.

Java

Skippy Java
all versions 17 or later

JUnit 4 / JUnit 5

Skippy JUnit 4 JUnit 5
≥ 0.0.18 4.10 or later 5.0 or later
≥ 0.0.8 4.10 or later 5.9 or later
≤ 0.0.7 ❌ 5.9 or later

Gradle / Maven

Skippy Gradle Maven
≥ 0.0.9 7.3 or later
8.0 or later		 3.2.3 or later
≤ 0.0.8 7.3 or later
8.0 or later		 ❌

JaCoCo

Skippy JaCoCo
≥ 0.0.9 0.8.7 or greater
≤ 0.0.8 0.8.9

Skippy’s Test Impact Analysis

The following section describe Skippy’s plugins for Gradle and Maven. The build plugins implement Skippy’s Test Impact Analysis.

Gradle

The skippy-gradle sub-project contains the Skippy plugin for Gradle.

Install

Release versions are available from Gradle’s Plugin Portal using the plugins DSL:

plugins {
    id("io.skippy") version '0.0.19'
}

Release versions are also available from Maven Central using the legacy plugin application:

buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath 'io.skippy:skippy-gradle:0.0.19'
    }
}

apply plugin: io.skippy.gradle.SkippyPlugin

Snapshots are available from s01.oss.sonatype.org using the legacy plugin application:

buildscript {
    repositories {
        mavenCentral()
        maven { url = 'https://s01.oss.sonatype.org/content/repositories/snapshots/' }
    }
    dependencies {
        classpath 'io.skippy:skippy-gradle:0.0.20-SNAPSHOT'
    }
}

apply plugin: io.skippy.gradle.SkippyPlugin

Tasks

The plugin adds the skippyClean and skippyAnalyze tasks to your project:

./gradlew tasks --group=skippy

Output:

Skippy tasks
------------
skippyClean
skippyAnalyze

skippyAnalyze Task

skippyAnalyze performs a Test Impact Analysis that stores a bunch of files in the .skippy folder. The generated files are consumed by Skippy’s testing libraries to make skip-or-execute predictions. skippyAnalyze is not meant to be invoked directly: It runs automatically whenever a Test task is executed.

skippyClean Task

skippyClean empties the skippy directory:

./gradlew skippyClean

It is the Gradle counterpart to rm -rf .skippy.

Maven

The skippy-maven sub-project contains the Skippy plugin for Maven.

Install

Release versions are available from Maven Central:

<project>
  ...
  <build>
    <plugins>
      <plugin>
        <groupId>io.skippy</groupId>
        <artifactId>skippy-maven</artifactId>
        <version>0.0.19</version>
        <executions>
          <execution>
            <goals>
              <goal>buildFinished</goal>
              <goal>buildStarted</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
  ...
</project>

Snapshots are available from s01.oss.sonatype.org:

<project>
  ...
  <pluginRepositories>
    <pluginRepository>
      <id>oss.sonatype</id>
      <url>https://s01.oss.sonatype.org/content/repositories/snapshots/</url>
    </pluginRepository>
  </pluginRepositories>
  ...
  <build>
    <plugins>
      <plugin>
        <groupId>io.skippy</groupId>
        <artifactId>skippy-maven</artifactId>
        <version>0.0.20-SNAPSHOT</version>
        <executions>
          <execution>
            <goals>
              <goal>buildFinished</goal>
              <goal>buildStarted</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
  ...
</project>

Since Skippy internally depends on JaCoCo, the JaCoCo plugin has to be added as well:

<project>
  ...
  <build>
    <plugins>
      ...
      <plugin>
        <groupId>org.jacoco</groupId>
        <artifactId>jacoco-maven-plugin</artifactId>
        <version>0.8.11</version>
        <executions>
          <execution>
            <goals>
              <goal>prepare-agent</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
  ...
</project>

Goals

The plugin adds the

  • skippy:buildStarted,
  • skippy:buildFinished and
  • skippy:clean goals to your project.

skippy:buildStarted & skippy:buildFinished

skippy:buildStarted and skippy:buildFinished perform a Test Impact Analysis that stores a bunch of files in the .skippy folder. The generated files are consumed by Skippy’s testing libraries to make skip-or-execute predictions. Both goals are not meant to be invoked directly: They run automatically when you execute your tests.

skippy:clean

skippy:clean empties the skippy directory:

mvn skippy:clean

It is the Maven counterpart to rm -rf .skippy.

Skippy’s Predictive Test Selection

Skippy’s JUnit libraries implement Skippy’s Predictive Test Selection algorithm. The algorithm makes skip-or-execute predictions based on

  • the local state of the project and
  • the impact data in the .skippy folder that was generated by Skippy’s Test Impact Analysis.

JUnit 4

The skippy-junit4 sub-project contains the Skippy library for JUnit 4.

Install

Releases are available in Maven Central.

Gradle:

repositories {
    mavenCentral()
}

dependencies {
    testImplementation 'io.skippy:skippy-junit4:0.0.19'
}

Maven:

<dependency>
    <groupId>io.skippy</groupId>
    <artifactId>skippy-junit4</artifactId>
    <version>0.0.19</version>
    <scope>test</scope>
</dependency>

Snapshots are available in s01.oss.sonatype.org.

Gradle:

repositories {
    maven { url = 'https://s01.oss.sonatype.org/content/repositories/snapshots/' }
}

dependencies {
    testImplementation 'io.skippy:skippy-junit4:0.0.20-SNAPSHOT'
}

Maven:

<project>
  
  <repositories>
    <repository>
      <id>s01-oss-sonatype-org</id>
      <url>https://s01.oss.sonatype.org/content/repositories/snapshots/</url>
    </repository>
  </repositories>
  
  <dependencies>
    <dependency>
        <groupId>io.skippy</groupId>
        <artifactId>skippy-junit4</artifactId>
        <version>0.0.20-SNAPSHOT</version>
        <scope>test</scope>
    </dependency>
  </dependencies>
  
</project>

Enable Predictive Test Selection

Add the Skippy class rule to your test:

import io.skippy.junit4.Skippy;

public class FooTest {

    @ClassRule
    public static TestRule skippyRule = Skippy.predictWithSkippy();

    @Test
    public void testFoo() {
        ...
    }

}

Run Your Tests

The Skippy class rule contains Skippy’s Predictive Test Selection algorithm. The algorithm makes skip-or-execute predictions based on

  • the state of the project and
  • the impact data in the .skippy folder that was generated by Skippy’s Test Impact Analysis.

JUnit 5

The skippy-junit5 sub-project contains the Skippy library for JUnit 5.

Install

Releases are available in Maven Central.

Gradle:

repositories {
    mavenCentral()
}

dependencies {
    testImplementation 'io.skippy:skippy-junit5:0.0.19'
}

Maven:

<dependency>
    <groupId>io.skippy</groupId>
    <artifactId>skippy-junit5</artifactId>
    <version>0.0.19</version>
    <scope>test</scope>
</dependency>

Snapshots are available in s01.oss.sonatype.org.

Gradle:

repositories {
    maven { url = 'https://s01.oss.sonatype.org/content/repositories/snapshots/' }
}

dependencies {
    testImplementation 'io.skippy:skippy-junit5:0.0.20-SNAPSHOT'
}

Maven:

<project>
  
  <repositories>
    <repository>
      <id>s01-oss-sonatype-org</id>
      <url>https://s01.oss.sonatype.org/content/repositories/snapshots/</url>
    </repository>
  </repositories>
  
  <dependencies>
    <dependency>
        <groupId>io.skippy</groupId>
        <artifactId>skippy-junit5</artifactId>
        <version>0.0.20-SNAPSHOT</version>
        <scope>test</scope>
    </dependency>
  </dependencies>
  
</project>

Enable Predictive Test Selection

You have two options to enable predictive test selection for you JUnit 5 tests:

  • Annotation Based Enablement
  • Automatic Enablement

Annotation Based Enablement

Annotate your tests with @PredictWithSkippy to enable predictive test selection:

import io.skippy.junit5.PredictWithSkippy;

@PredictWithSkippy
public class FooTest {

    @Test
    void testFoo() {
        assertEquals("hello", Foo.hello());
    }
}

The annotation based approach allows you to enable predictive test selection for a sub-set of your tests.

Automatic Enablement

Automatic enablement is based on JUnit 5’s Automatic Extension Registration. Add a file named org.junit.jupiter.api.extension.Extension in src/test/resources/META-INF/services (adjust according to the location of your test resources). The file must have the following content:

io.skippy.junit5.SkipOrExecuteCondition
io.skippy.junit5.CoverageFileCallbacks

Start your tests with the JVM argument -Djunit.jupiter.extensions.autodetection.enabled=true to automatically enable predictive test selection for all tests.

Gradle example:

test {
  jvmArgs += "-Djunit.jupiter.extensions.autodetection.enabled=true"
}

Maven example:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
        <systemPropertyVariables>
            <junit.jupiter.extensions.autodetection.enabled>true</junit.jupiter.extensions.autodetection.enabled>
        </systemPropertyVariables>
    </configuration>
</plugin>

Run Your Tests

The Skippy extension contains Skippy’s Predictive Test Selection algorithm. The algorithm makes skip-or-execute predictions based on

  • the local state of the project and
  • the impact data in the .skippy folder that was generated by Skippy’s Test Impact Analysis.

Coverage for skipped tests

Skippy supports the creation of code coverage reports that include coverage for skipped tests. This feature can be enabled using the setting coverageForSkippedTests which defaults to false.

When enabled, Skippy stores an execution data files for each test that is executed:

./gradlew test

LeftPadderTest > testPadLeft() PASSED
RightPadderTest > testPadRight() PASSED

The execution data files are stored in the .skippy folder:

ls -l .skippy                   

A4C4B20204923316E28BF16B20811F3C.exec
D16D8B65CE4D3DDCC1217644BA0C1DFF.exec

The executionId property in the test-impact-analysis.json file connects the dots between the execution of a test and the corresponding execution data file:

{
  "classes": {
    "0": {
      "name": "com.example.LeftPadderTest",
      ...
      "hash": "D6954047"
    },
    "1": {
      "name": "com.example.RightPadderTest",
      ...
    }
  },
  "tests": [
    {
      "class": 0,
      ...
      "executionId": "A4C4B20204923316E28BF16B20811F3C"
    },
    {
      "class": 1,
      ...
      "executionId": "D16D8B65CE4D3DDCC1217644BA0C1DFF"
    }
  ]
}

At the end a build, Skippy will generate a file named skippy.exec within the build directory. It is typically found under build/ for Gradle projects or target/ for Maven projects:

ls -l build

skippy.exec

skippy.exec aggregates the execution data for all tests that have been skipped in the current build. It can be merged with JaCoCo’s standard execution data file to generate a combined report with coverage for executed and skipped tests:

This feature is particularly useful for corporate CI pipelines with strict code coverage requirements.

Gradle

To enable this feature in Gradle, add the following to your build.gradle file:

skippy {
  coverageForSkippedTests = true
}

Maven

To enable this feature in Maven, add the following to your pom.xml file:

  <plugin>
    <groupId>io.skippy</groupId>
    <artifactId>skippy-maven</artifactId>
    ...
    <configuration>
      <coverageForSkippedTests>true</coverageForSkippedTests>
    </configuration>
    ...
  </plugin>

Customize how Skippy reads and writes data

Skippy’s SkippyRepositoryExtension allows projects to customize the way Skippy reads and writes data. Skippy’s default behavior:

  • It stores and retrieves all data in / from the .skippy folder.
  • It only retains the latest Test Impact Analysis.
  • It only retains the JaCoCo execution data files that are referenced by the latest Test Impact Analysis.

The default settings are designed for small projects that do not require code coverage reports, and thus, do not store JaCoCo execution data files. Those projects will typically disable the coverageForSkippedTests setting. While these defaults support projects of any size and allow for the storage of JaCoCo data files for experimental purposes, they are not recommended for large projects or those needing to store such files long-term. Doing so could significantly increase the size of a project’s history in a version control system.

Large projects aiming to store and permanently retain Test Impact Analysis instances and JaCoCo execution data files can register a custom SkippyRepositoryExtension. This extension enables the storage of these artifacts outside the project’s repository using systems such as databases, network file systems, or blob storage solutions like AWS S3.

Example for a custom SkippyRepositoryExtension: FileSystemBackedRepositoryExtension

Gradle

To enable this feature in Gradle, add the following to your build.gradle file:

buildscript {    
    dependencies {
        // provide your extension to Skippy's Gradle plugin
        classpath 'com.example:my-repository-extension:1.0.0'
    }
}

skippy {
    // register your extension
    repository = 'com.example.MyRepositoryExtension'
}

dependencies {
    // provide your extension to your tests
    testImplementation 'com.example:my-repository-extension:1.0.0'
}

Your extension must be provided as dependency to the build script and your tests.

Maven

To enable this feature in Maven, add the following to your pom.xml file:

  ...
  <dependencies>
    <!-- provide your extension to your tests -->
    <dependency>
      <groupId>com.example</groupId>
      <artifactId>my-repository-extension</artifactId>
      <version>1.0.0</version>
      <scope>test</scope>
    </dependency>
  </dependencies>
  ...
  <plugin>
    <groupId>io.skippy</groupId>
    <artifactId>skippy-maven</artifactId>
    <dependencies>
      <!-- provide your extension to Skippy's Maven plugin -->
      <dependency>
        <groupId>com.example</groupId>
        <artifactId>my-repository-extension</artifactId>
        <version>1.0.0</version>
      </dependency>
    </dependencies>
    <configuration>
      <!-- register your extension -->
      <repository>com.example.MyRepositoryExtension</repository>
    </configuration>
  </plugin>
  ...

Your extension must be provided as dependency to Skippy’s Maven plugin and your tests.

Skippy in your CI pipeline

Skippy supports execution in CI environments out of the box. It has actually been designed for this purpose. The only thing you need to do is to add the .skippy folder to version control. This will automatically enable Skippy’s Predictive Test Selection when your pipeline runs.

Roadmap

  • Support for reasoning based on changes in the classpath (e.g., updated 3rd party dependencies)
  • Support for reasoning based on changes in resources (e.g., a change in application.properties)