Documentation
Reference documentation for Skippy version 0.0.23
.
New to Skippy? The best way to get started are the introductory tutorials:
Table of Contents
- Overview
- Compatibility
- Skippy’s Test Impact Analysis
- Skippy’s Predictive Test Selection
- Coverage for skipped tests
- Customize how Skippy reads and writes data
- Skippy in your CI pipeline
- Roadmap
Overview
What is it?
Skippy is a Test Impact Analysis & Predictive Test Selection framework for the JVM. It cuts down on unnecessary testing and flakiness without compromising the integrity of your builds. You can run it from the command line, your favorite IDE and continuous integration server. Skippy supports Gradle, Maven, JUnit 4 and JUnit 5.
Skippy is specifically designed to prevent regressions in your codebase. It supports all types of tests where the tests and the code under test run in the same JVM. It is best suited for deterministic tests, even those prone to occasional flakiness. It provides the most value for test suites that are either slow or flaky (regardless of whether the test suite contains unit, integration, or functional tests).
What is it not?
Skippy is not designed for tests that assert the overall health of a system. Don’t use it for tests you want to fail in response to misbehaving services, infrastructure issues, etc.
Highlights
- Support for Gradle & Maven
- Support for JUnit 4 & JUnit 5
- Lightweight: Use it from the command line, your favorite IDE and CI server
- Non-invasive: Use it for a single test, your entire suite and anything in-between
- Free of lock-in: You can go back to a “run everything” approach at any time
- Open Source under Apache 2 License
Compatibility
Compatibility matrix for Skippys’s integration with Java, JUnit, Gradle, Maven and JaCoCo. Versions not mentioned here might work, but there is no guarantee.
Java
Skippy | Java |
---|---|
all versions | 17 or later |
JUnit 4 / JUnit 5
Skippy | JUnit 4 | JUnit 5 |
---|---|---|
≥ 0.0.18 | 4.10 or later | 5.0 or later |
≥ 0.0.8 | 4.10 or later | 5.9 or later |
≤ 0.0.7 | ❌ | 5.9 or later |
Gradle / Maven
Skippy | Gradle | Maven |
---|---|---|
≥ 0.0.9 | 7.3 or later 8.0 or later |
3.2.3 or later |
≤ 0.0.8 | 7.3 or later 8.0 or later |
❌ |
JaCoCo
Skippy | JaCoCo |
---|---|
≥ 0.0.9 | 0.8.7 or greater |
≤ 0.0.8 | 0.8.9 |
Skippy’s Test Impact Analysis
The following section describe Skippy’s plugins for Gradle and Maven. The build plugins implement Skippy’s Test Impact Analysis.
Gradle
The skippy-gradle sub-project contains the Skippy plugin for Gradle.
Install
Release versions are available from Gradle’s Plugin Portal using the plugins DSL:
plugins {
id("io.skippy") version '0.0.23'
}
Release versions are also available from Maven Central using the legacy plugin application:
buildscript {
repositories {
mavenCentral()
}
dependencies {
classpath 'io.skippy:skippy-gradle:0.0.23'
}
}
apply plugin: io.skippy.gradle.SkippyPlugin
Snapshots are available from s01.oss.sonatype.org
using the
legacy plugin application:
buildscript {
repositories {
mavenCentral()
maven { url = 'https://s01.oss.sonatype.org/content/repositories/snapshots/' }
}
dependencies {
classpath 'io.skippy:skippy-gradle:0.0.24-SNAPSHOT'
}
}
apply plugin: io.skippy.gradle.SkippyPlugin
Tasks
The plugin adds the skippyClean
and skippyAnalyze
tasks to your project:
./gradlew tasks --group=skippy
Output:
Skippy tasks
------------
skippyClean
skippyAnalyze
skippyAnalyze Task
skippyAnalyze
performs a Test Impact Analysis that stores a bunch of files in the .skippy folder.
The generated files are consumed by Skippy’s testing libraries
to make skip-or-execute predictions. skippyAnalyze
is not meant to be invoked directly: It runs
automatically whenever a Test task is executed.
skippyClean Task
skippyClean
empties the skippy directory:
./gradlew skippyClean
It is the Gradle counterpart to rm -rf .skippy
.
Maven
The skippy-maven sub-project contains the Skippy plugin for Maven.
Install
Release versions are available from Maven Central:
<project>
...
<build>
<plugins>
<plugin>
<groupId>io.skippy</groupId>
<artifactId>skippy-maven</artifactId>
<version>0.0.23</version>
<executions>
<execution>
<goals>
<goal>buildFinished</goal>
<goal>buildStarted</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
...
</project>
Snapshots are available from s01.oss.sonatype.org
:
<project>
...
<pluginRepositories>
<pluginRepository>
<id>oss.sonatype</id>
<url>https://s01.oss.sonatype.org/content/repositories/snapshots/</url>
</pluginRepository>
</pluginRepositories>
...
<build>
<plugins>
<plugin>
<groupId>io.skippy</groupId>
<artifactId>skippy-maven</artifactId>
<version>0.0.24-SNAPSHOT</version>
<executions>
<execution>
<goals>
<goal>buildFinished</goal>
<goal>buildStarted</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
...
</project>
Since Skippy internally depends on JaCoCo, the JaCoCo plugin has to be added as well:
<project>
...
<build>
<plugins>
...
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.11</version>
<executions>
<execution>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
...
</project>
Goals
The plugin adds the
skippy:buildStarted
,skippy:buildFinished
andskippy:clean
goals to your project.
skippy:buildStarted & skippy:buildFinished
skippy:buildStarted
and skippy:buildFinished
perform a Test Impact Analysis that stores a bunch of files
in the .skippy folder. The generated files are consumed by Skippy’s testing libraries
to make skip-or-execute predictions. Both goals are not meant to be invoked directly: They run automatically
when you execute your tests.
skippy:clean
skippy:clean
empties the skippy directory:
mvn skippy:clean
It is the Maven counterpart to rm -rf .skippy
.
Skippy’s Predictive Test Selection
Skippy’s JUnit libraries implement Skippy’s Predictive Test Selection algorithm. The algorithm makes skip-or-execute predictions based on
- the local state of the project and
- the impact data in the .skippy folder that was generated by Skippy’s Test Impact Analysis.
JUnit 4
The skippy-junit4 sub-project contains the Skippy library for JUnit 4.
Install Skippy (JUnit 4)
Releases are available in Maven Central.
Gradle:
repositories {
mavenCentral()
}
dependencies {
testImplementation 'io.skippy:skippy-junit4:0.0.23'
}
Maven:
<dependency>
<groupId>io.skippy</groupId>
<artifactId>skippy-junit4</artifactId>
<version>0.0.23</version>
<scope>test</scope>
</dependency>
Snapshots are available in s01.oss.sonatype.org
.
Gradle:
repositories {
maven { url = 'https://s01.oss.sonatype.org/content/repositories/snapshots/' }
}
dependencies {
testImplementation 'io.skippy:skippy-junit4:0.0.24-SNAPSHOT'
}
Maven:
<project>
<repositories>
<repository>
<id>s01-oss-sonatype-org</id>
<url>https://s01.oss.sonatype.org/content/repositories/snapshots/</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>io.skippy</groupId>
<artifactId>skippy-junit4</artifactId>
<version>0.0.24-SNAPSHOT</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
Enable Predictive Test Selection (JUnit 4)
Add the Skippy class rule to your test:
import io.skippy.junit4.Skippy;
public class FooTest {
@ClassRule
public static TestRule skippyRule = Skippy.predictWithSkippy();
@Test
public void testFoo() {
...
}
}
Run Your Tests
The Skippy class rule contains Skippy’s Predictive Test Selection algorithm. The algorithm makes skip-or-execute predictions based on
- the state of the project and
- the impact data in the .skippy folder that was generated by Skippy’s Test Impact Analysis.
JUnit 5
The skippy-junit5 sub-project contains the Skippy library for JUnit 5.
Install Skippy (JUnit 5)
Releases are available in Maven Central.
Gradle:
repositories {
mavenCentral()
}
dependencies {
testImplementation 'io.skippy:skippy-junit5:0.0.23'
}
Maven:
<dependency>
<groupId>io.skippy</groupId>
<artifactId>skippy-junit5</artifactId>
<version>0.0.23</version>
<scope>test</scope>
</dependency>
Snapshots are available in s01.oss.sonatype.org
.
Gradle:
repositories {
maven { url = 'https://s01.oss.sonatype.org/content/repositories/snapshots/' }
}
dependencies {
testImplementation 'io.skippy:skippy-junit5:0.0.24-SNAPSHOT'
}
Maven:
<project>
<repositories>
<repository>
<id>s01-oss-sonatype-org</id>
<url>https://s01.oss.sonatype.org/content/repositories/snapshots/</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>io.skippy</groupId>
<artifactId>skippy-junit5</artifactId>
<version>0.0.24-SNAPSHOT</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
Enable Predictive Test Selection (JUnit 5)
You have two options to enable predictive test selection for you JUnit 5 tests:
- Annotation Based Enablement
- Automatic Enablement
Annotation Based Enablement
Annotate your tests with @PredictWithSkippy
to enable predictive test selection:
import io.skippy.junit5.PredictWithSkippy;
@PredictWithSkippy
public class FooTest {
@Test
void testFoo() {
assertEquals("hello", Foo.hello());
}
}
The annotation based approach allows you to enable predictive test selection for a sub-set of your tests.
Automatic Enablement
Automatic enablement is based on JUnit 5’s Automatic Extension Registration.
Add a file named org.junit.jupiter.api.extension.Extension
in
src/test/resources/META-INF/services
(adjust according to the location of your test resources). The file must have
the following content:
io.skippy.junit5.SkipOrExecuteCondition
io.skippy.junit5.CoverageFileCallbacks
Start your tests with the JVM argument -Djunit.jupiter.extensions.autodetection.enabled=true
to automatically
enable predictive test selection for all tests.
Gradle example:
test {
jvmArgs += "-Djunit.jupiter.extensions.autodetection.enabled=true"
}
Maven example:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<systemPropertyVariables>
<junit.jupiter.extensions.autodetection.enabled>true</junit.jupiter.extensions.autodetection.enabled>
</systemPropertyVariables>
</configuration>
</plugin>
Run Your Tests
The Skippy extension contains Skippy’s Predictive Test Selection algorithm. The algorithm makes skip-or-execute predictions based on
- the local state of the project and
- the impact data in the .skippy folder that was generated by Skippy’s Test Impact Analysis.
Unsupported Features (JUnit 5)
The following JUnit 5 features are currently not supported:
- Parallel Execution: Test cases that run concurrently within the same JVM prevent Skippy to capture accurate coverage data. This will result in incorrect skip-or-execute predictions. You have to disable parallel execution if you want to use Skippy.
Opt-Out Mechanism
With Automatic Enablement, Skippy is enabled for all tests without source code changes.
However, if there are specific tests you don’t want Skippy to optimize, use the @AlwaysRun
annotation:
import io.skippy.core.AlwaysRun;
@AlwaysRun
public class FooTest {
@Test
void testThatYouNeverWantToSkip() {
// this test will always run
}
}
Any test marked with @AlwaysRun
, or inheriting from a superclass or interface annotated with it, will always run and bypass Skippy’s skip-or-execute logic.
This is also useful for @Nested
tests (even if you use Annotation Based Enablement):
@PredictWithSkippy
public class NestedTestsTest {
@Nested
class FooTest {
@Test
void testSomething() {
// this test can be skipped by Skippy
}
}
@AlwaysRun
@Nested
class BarTest {
@Test
void testSomething() {
// this test will always run
}
}
}
Custom PredictionModifier
Skippy’s PredictionModifier
extension point controls this behavior. The default implementation DefaultPredictionModifier
bypasses Skippy’s skip-or-execute logic for tests annotated with @AlwaysRun
.
You can register your own PredictionModifier
in the build configuration to override the default behavior.
This allows you to selectively disable Skippy based on criteria such as:
- Class name
- Package name
- Any other information available from a
Class<?>
object
Gradle
Add the following to your build.gradle
file to register a custom PredictionModifier
in Gradle:
skippy {
predictionModifier = 'com.example.MyPredictionModifier'
}
dependencies {
testImplementation 'com.example:my-prediction-modifier:1.0.0'
}
Maven
Add the following to your pom.xml
file to register a custom PredictionModifier
in Maven:
...
<dependencies>
<!-- provide your extension to your tests -->
<dependency>
<groupId>com.example</groupId>
<artifactId>my-prediction-modifier</artifactId>
<version>1.0.0</version>
<scope>test</scope>
</dependency>
</dependencies>
...
<plugin>
<groupId>io.skippy</groupId>
<artifactId>skippy-maven</artifactId>
<dependencies>
<configuration>
<!-- register your extension -->
<predictionModifier>com.example.MyPredictionModifier</predictionModifier>
</configuration>
</plugin>
...
Coverage for skipped tests
Skippy supports the creation of code coverage reports that include coverage for skipped tests. This feature can be
enabled using the setting coverageForSkippedTests
which defaults to false
.
When enabled, Skippy stores an execution data files for each test that is executed:
./gradlew test
LeftPadderTest > testPadLeft() PASSED
RightPadderTest > testPadRight() PASSED
The execution data files are stored in the .skippy folder:
ls -l .skippy
A4C4B20204923316E28BF16B20811F3C.exec
D16D8B65CE4D3DDCC1217644BA0C1DFF.exec
The executionId
property in the test-impact-analysis.json file connects the dots between the execution of a test and
the corresponding execution data file:
{
"classes": {
"0": {
"name": "com.example.LeftPadderTest",
...
"hash": "D6954047"
},
"1": {
"name": "com.example.RightPadderTest",
...
}
},
"tests": [
{
"class": 0,
...
"executionId": "A4C4B20204923316E28BF16B20811F3C"
},
{
"class": 1,
...
"executionId": "D16D8B65CE4D3DDCC1217644BA0C1DFF"
}
]
}
At the end a build, Skippy will generate a file named skippy.exec within the build directory. It is typically found under build/ for Gradle projects or target/ for Maven projects:
ls -l build
skippy.exec
skippy.exec aggregates the execution data for all tests that have been skipped in the current build. It can be merged with JaCoCo’s standard execution data file to generate a combined report with coverage for executed and skipped tests:
This feature is particularly useful for corporate CI pipelines with strict code coverage requirements.
Gradle
To enable this feature in Gradle, add the following to your build.gradle
file:
skippy {
coverageForSkippedTests = true
}
Maven
To enable this feature in Maven, add the following to your pom.xml
file:
<plugin>
<groupId>io.skippy</groupId>
<artifactId>skippy-maven</artifactId>
...
<configuration>
<coverageForSkippedTests>true</coverageForSkippedTests>
</configuration>
...
</plugin>
Customize how Skippy reads and writes data
Skippy’s SkippyRepositoryExtension allows projects to customize the way Skippy reads and writes data. Skippy’s default behavior:
- It stores and retrieves all data in / from the .skippy folder.
- It only retains the latest Test Impact Analysis.
- It only retains the JaCoCo execution data files that are referenced by the latest Test Impact Analysis.
The default settings are designed for small projects that do not require code coverage reports, and thus, do not store
JaCoCo execution data files. Those projects will typically disable the coverageForSkippedTests
setting. While these
defaults support projects of any size and allow for the storage of JaCoCo data files for experimental purposes, they are
not recommended for large projects or those needing to store such files long-term. Doing so could significantly increase
the size of a project’s history in a version control system.
Large projects aiming to store and permanently retain Test Impact Analysis instances and JaCoCo execution data files can
register a custom SkippyRepositoryExtension
. This extension enables the storage of these artifacts outside the
project’s repository using systems such as databases, network file systems, or blob storage solutions like AWS S3.
Example for a custom SkippyRepositoryExtension
: FileSystemBackedRepositoryExtension
Gradle
To enable this feature in Gradle, add the following to your build.gradle
file:
buildscript {
dependencies {
// provide your extension to Skippy's Gradle plugin
classpath 'com.example:my-repository-extension:1.0.0'
}
}
skippy {
// register your extension
repository = 'com.example.MyRepositoryExtension'
}
dependencies {
// provide your extension to your tests
testImplementation 'com.example:my-repository-extension:1.0.0'
}
Your extension must be provided as dependency to the build script and your tests.
Maven
To enable this feature in Maven, add the following to your pom.xml
file:
...
<dependencies>
<!-- provide your extension to your tests -->
<dependency>
<groupId>com.example</groupId>
<artifactId>my-repository-extension</artifactId>
<version>1.0.0</version>
<scope>test</scope>
</dependency>
</dependencies>
...
<plugin>
<groupId>io.skippy</groupId>
<artifactId>skippy-maven</artifactId>
<dependencies>
<!-- provide your extension to Skippy's Maven plugin -->
<dependency>
<groupId>com.example</groupId>
<artifactId>my-repository-extension</artifactId>
<version>1.0.0</version>
</dependency>
</dependencies>
<configuration>
<!-- register your extension -->
<repository>com.example.MyRepositoryExtension</repository>
</configuration>
</plugin>
...
Your extension must be provided as dependency to Skippy’s Maven plugin and your tests.
Skippy in your CI pipeline
Skippy supports execution in CI environments out of the box. It has actually been designed for this purpose. The only thing you need to do is to add the .skippy folder to version control. This will automatically enable Skippy’s Predictive Test Selection when your pipeline runs.
Roadmap
- Support for reasoning based on changes in the classpath (e.g., updated 3rd party dependencies)
- Support for reasoning based on changes in resources (e.g., a change in
application.properties
)