Documentation for Skippy version 0.0.24.

Overview

A quick tour how to use Skippy with Maven and JUnit 5.

What You Need

  • About 15 minutes
  • Your favorite text editor or IDE
  • Java 17 or later
  • Maven 3.2.3+

Table Of Contents

Setting Up Your Environment

Begin by cloning the skippy-tutorials repository:

git clone git@github.com:skippy-io/skippy-tutorials.git

Then, move into the tutorial directory:

cd skippy-tutorials/getting-started-with-skippy-and-junit5/

Exploring the Codebase

Let’s take a quick look at the codebase.

The pom.xml File

pom.xml declares a dependency to skippy-junit5:

<dependency>
    <groupId>io.skippy</groupId>
    <artifactId>skippy-junit5</artifactId>
    <version>0.0.24</version>
    <scope>test</scope>
</dependency>

It also adds the Skippy plugin:

<plugin>
    <groupId>io.skippy</groupId>
    <artifactId>skippy-maven</artifactId>
    <version>0.0.24</version>
    <executions>
        <execution>
            <goals>
                <goal>buildFinished</goal>
                <goal>buildStarted</goal>
            </goals>
        </execution>
    </executions>
</plugin>

Since Skippy internally depends on JaCoCo, the JaCoCo plugin has to be added as well:

<plugin>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <version>0.8.11</version>
    <executions>
        <execution>
            <goals>
                <goal>prepare-agent</goal>
            </goals>
        </execution>
    </executions>
</plugin>

src/main/java

The main source set contains three classes:

com
└─ example
   ├─ LeftPadder.java
   ├─ RightPadder.java
   └─ StringUtils.java

StringUtils is a utility class that provides methods for padding strings:

class StringUtils {

    static String padLeft(String input, int size) {
        // method logic
    }

    static String padRight(String input, int size) {
        // method logic
    }
}

LeftPadder and RightPadder utilize StringUtils for their functionality:

class LeftPadder {

    static String padLeft(String input, int size) {
        return StringUtils.padLeft(input, size);
    }

}

class RightPadder {

    static String padRight(String input, int size) {
        return StringUtils.padRight(input, size);
    }

}

src/test/java

The test source set contains three tests and one class that stores constants:

com
└─ example
   ├─ LeftPadderTest.java
   ├─ RightPadderTest.java
   └─ TestConstants.java

LeftPadderTest and RightPadderTest are unit tests for their respective classes.

Both tests are annotated with @PredictWithSkippy to enable Skippy’s predictive test selection:

import io.skippy.junit5.PredictWithSkippy;

@PredictWithSkippy
public class LeftPadderTest {

    @Test
    void testPadLeft() {
        var input = TestConstants.HELLO;
        assertEquals(" hello", LeftPadder.padLeft(input, 6));
    }

}

TestConstants declares a string constant:

class TestConstants {
    static final String HELLO = "hello";
}

Run The Tests

Run the tests:

mvn test

Output:

[INFO] Running com.example.LeftPadderTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

[INFO] Running com.example.RightPadderTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

As you would expect, all tests are executed. But there is more than meets the eye: Skippy performs a Test Impact Analysis and stores the result as bunch of files in the .skippy folder:

ls -l .skippy

test-impact-analysis.json
predictions.log
...

test-impact-analysis.json contains

  • a mapping between tests and the classes they cover and
  • a snapshot of all class files in the project.

predictions.log gives you insights into Skippy’s skip-or-execute decisions:

cat .skippy/predictions.log

...,com.example.LeftPadderTest,EXECUTE,TEST_IMPACT_ANALYSIS_NOT_FOUND
...,com.example.RightPadderTest,EXECUTE,TEST_IMPACT_ANALYSIS_NOT_FOUND

Skippy executed both tests because it did not find prior impact data.

Commit the Skippy Folder

Add the .skippy folder to Git:

git add .skippy && git commit -m 'add skippy folder'

This allows us to revert back to this state of the repository throughout the rest tutorial.

Re-Run The Tests

Re-run the tests:

mvn test

Output:

[INFO] Running com.example.LeftPadderTest
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1

[INFO] Running com.example.RightPadderTest
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1

Skippy detects that nothing has changed and skips both tests:

cat .skippy/predictions.log

...,com.example.LeftPadderTest,SKIP,NO_CHANGE
...,com.example.RightPadderTest,SKIP,NO_CHANGE

Testing After Modifications

When changes are made, Skippy reassesses which tests to run based on it’s bytecode based change detection. Reasoning based on the bytecode is powerful: It allows Skippy to distinguish relevant changes (e.g., new or updated instructions) from irrelevant ones (e.g., a change in a LineNumberTable attribute due to the addition of a line break somewhere in the source file).

Let’s perform some experiments.

Experiment 1

Add a comment to StringUtils:

/**
 * New class comment.
 */
class StringUtils {
        
    ...
}

Re-run the tests:

mvn test

Skippy detects that the newly added comment can not break any of the existing tests. LeftPadderTest and RightPadderTest will be skipped:

[INFO] Running com.example.LeftPadderTest
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1

[INFO] Running com.example.RightPadderTest
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1

Experiment 2

Undo the changes from the previous experiment:

git stash

Comment out the first three lines of StringUtils#padLeft:

class StringUtils {
    
    static String padLeft(String input, int size) {
//        if (input.length() < size) {
//            return padLeft(" " + input, size);
//        }
        return input;
    }

    static String padRight(String input, int size) {
        if (input.length() < size) {
            return padRight(input + " ", size);
        }
        return input;
    }
    
}

Re-run the tests:

mvn test

Skippy detects the change and runs the tests again:

[INFO] Running com.example.LeftPadderTest
[ERROR] com.example.LeftPadderTest.testPadLeft <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: < hello> but was: <hello>       

[INFO] Running com.example.RightPadderTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

Content of .skippy/predictions.log:

...,com.example.LeftPadderTest,EXECUTE,BYTECODE_CHANGE_IN_COVERED_CLASS,"covered class: com.example.StringUtils"
...,com.example.RightPadderTest,EXECUTE,BYTECODE_CHANGE_IN_COVERED_CLASS,"covered class: com.example.StringUtils"

At this point in time, Skippy executes a test if the covered class contains a significant bytecode change (e.g., new or updated instructions). The test itself may or may not depend on this change. In the above example, RightPadderTest could be skipped as well.

While I plan to implement more granular change detection in the future, I currently apply the 80/20 rule: My focus is robustness and simplicity while providing a significant reduction in useless testing for applications that contain large quantities of source files and tests.

Experiment 3

Undo the changes from the previous experiment:

git stash

Now, let’s see what happens if you change the expected value in LeftPadderTest from " hello" to " HELLO":

@PredictWithSkippy
public class LeftPadderTest {

    @Test
    void testPadLeft() {
        var input = TestConstants.HELLO;
        // assertEquals(" hello", LeftPadder.padLeft(input, 6));
        assertEquals(" HELLO", LeftPadder.padLeft(input, 6));
    }

}

Re-run the tests:

mvn test

Skippy detects the change and runs LeftPadderTest again:

[INFO] Running com.example.LeftPadderTest
[ERROR] com.example.LeftPadderTest.testPadLeft <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: < HELLO> but was: < hello>

[INFO] Running com.example.RightPadderTest
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1

Content of .skippy/predictions.log:

...,com.example.LeftPadderTest,EXECUTE,BYTECODE_CHANGE_IN_TEST
...,com.example.RightPadderTest,SKIP,NO_CHANGE

Experiment 4

Undo the changes from the previous experiment:

git stash

Lastly, let’s see what happens if you change the value of the constant in TestConstants:

class TestConstants {

    // static final String HELLO = "hello";
    static final String HELLO = "bonjour";

}

Re-run the tests:

mvn test

Skippy detects the change and runs both tests:

[INFO] Running com.example.LeftPadderTest
[ERROR] com.example.LeftPadderTest.testPadLeft <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: < hello> but was: <bonjour>

[INFO] Running com.example.RightPadderTest
[ERROR] com.example.RightPadderTest.testPadRight <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <hello > but was: <bonjour>

Content of .skippy/predictions.log:

...,com.example.LeftPadderTest,EXECUTE,BYTECODE_CHANGE_IN_TEST
...,com.example.RightPadderTest,EXECUTE,BYTECODE_CHANGE_IN_TEST

Congratulations - You’ve completed the tutorial! By now, you should have a solid idea how Skippy works.