Documentation for Skippy version 0.0.24.

Overview

A quick tour how to use Skippy with Gradle and JUnit 4.

What You Need

  • About 15 minutes
  • Your favorite text editor or IDE
  • Java 17 or later
  • Gradle 7.3+

Table Of Contents

Setting Up Your Environment

Begin by cloning the skippy-tutorials repository:

git clone git@github.com:skippy-io/skippy-tutorials.git

Then, move into the tutorial directory:

cd skippy-tutorials/getting-started-with-skippy-and-junit4/

Exploring the Codebase

Let’s take a quick look at the codebase.

The build.gradle File

build.gradle applies the io.skippy plugin and adds a dependency to skippy-junit4:

plugins {
    id 'io.skippy' version '0.0.24'
}

dependencies {
    testImplementation 'io.skippy:skippy-junit4:0.0.24'
}

src/main/java

The main source set contains three classes:

com
└─ example
   ├─ LeftPadder.java
   ├─ RightPadder.java
   └─ StringUtils.java

StringUtils is a utility class that provides methods for padding strings:

class StringUtils {

    static String padLeft(String input, int size) {
        // method logic
    }

    static String padRight(String input, int size) {
        // method logic
    }
}

LeftPadder and RightPadder utilize StringUtils for their functionality:

class LeftPadder {

    static String padLeft(String input, int size) {
        return StringUtils.padLeft(input, size);
    }

}

class RightPadder {

    static String padRight(String input, int size) {
        return StringUtils.padRight(input, size);
    }

}

src/test/java

The test source set contains three tests and one class that stores constants:

com
└─ example
   ├─ LeftPadderTest.java
   ├─ RightPadderTest.java
   └─ TestConstants.java

LeftPadderTest and RightPadderTest are unit tests for their respective classes.

Both tests use the Skippy.predictWithSkippy() rule to enable Skippy’s predictive test selection:

import org.junit.*;
import org.junit.rules.TestRule;

import io.skippy.junit4.Skippy;

public class LeftPadderTest {

    @Rule
    public TestRule skippyRule = Skippy.predictWithSkippy();

    @Test
    public void testPadLeft() {
        var input = TestConstants.HELLO;
        assertEquals(" hello", LeftPadder.padLeft(input, 6));
    }

}

TestConstants declares a string constant:

class TestConstants {
    static final String HELLO = "hello";
}

Run The Tests

Run the tests:

./gradlew test --rerun

Output:

LeftPadderTest > testPadLeft() PASSED
RightPadderTest > testPadRight() PASSED

As you would expect, all tests are executed. But there is more than meets the eye: Skippy performs a Test Impact Analysis and stores the result as bunch of files in the .skippy folder:

ls -l .skippy

test-impact-analysis.json
predictions.log
...

test-impact-analysis.json contains

  • a mapping between tests and the classes they cover and
  • a snapshot of all class files in the project.

predictions.log gives you insights into Skippy’s skip-or-execute decisions:

cat .skippy/predictions.log

...,com.example.LeftPadderTest,EXECUTE,TEST_IMPACT_ANALYSIS_NOT_FOUND
...,com.example.RightPadderTest,EXECUTE,TEST_IMPACT_ANALYSIS_NOT_FOUND

Skippy executed both tests because it did not find prior impact data.

Commit the Skippy Folder

Add the .skippy folder to Git:

git add .skippy && git commit -m 'add skippy folder'

This allows us to revert back to this state of the repository throughout the rest tutorial.

Re-Run The Tests

Re-run the tests:

./gradlew test --rerun

Output:

LeftPadderTest > testPadLeft() SKIPPED
RightPadderTest > testPadRight() SKIPPED

Skippy detects that nothing has changed and skips both tests:

cat .skippy/predictions.log

...,com.example.LeftPadderTest,SKIP,NO_CHANGE
...,com.example.RightPadderTest,SKIP,NO_CHANGE

Testing After Modifications

When changes are made, Skippy reassesses which tests to run based on it’s bytecode based change detection. Reasoning based on the bytecode is powerful: It allows Skippy to distinguish relevant changes (e.g., new or updated instructions) from irrelevant ones (e.g., a change in a LineNumberTable attribute due to the addition of a line break somewhere in the source file).

Let’s perform some experiments.

Experiment 1

Add a comment to StringUtils:

/**
 * New class comment.
 */
class StringUtils {
        
    ...
}

Re-run the tests:

./gradlew test --rerun

Skippy detects that the newly added comment can not break any of the existing tests. LeftPadderTest and RightPadderTest will be skipped:

LeftPadderTest > testPadLeft() SKIPPED
RightPadderTest > testPadRight() SKIPPED

Experiment 2

Undo the changes from the previous experiment:

git stash

Comment out the first three lines of StringUtils#padLeft:

class StringUtils {
    
    static String padLeft(String input, int size) {
//        if (input.length() < size) {
//            return padLeft(" " + input, size);
//        }
        return input;
    }

    static String padRight(String input, int size) {
        if (input.length() < size) {
            return padRight(input + " ", size);
        }
        return input;
    }
    
}

Re-run the tests:

./gradlew test --rerun

Skippy detects the change and runs the tests again:

LeftPadderTest > testPadLeft() FAILED
    org.opentest4j.AssertionFailedError: expected: < hello> but was: <hello>

RightPadderTest > testPadRight() PASSED

Content of .skippy/predictions.log:

...,com.example.LeftPadderTest,EXECUTE,BYTECODE_CHANGE_IN_COVERED_CLASS,"covered class: com.example.StringUtils"
...,com.example.RightPadderTest,EXECUTE,BYTECODE_CHANGE_IN_COVERED_CLASS,"covered class: com.example.StringUtils"

At this point in time, Skippy executes a test if the covered class contains a significant bytecode change (e.g., new or updated instructions). The test itself may or may not depend on this change. In the above example, RightPadderTest could be skipped as well.

While I plan to implement more granular change detection in the future, I currently apply the 80/20 rule: My focus is robustness and simplicity while providing a significant reduction in useless testing for applications that contain large quantities of source files and tests.

Experiment 3

Undo the changes from the previous experiment:

git stash

Now, let’s see what happens if you change the expected value in LeftPadderTest from " hello" to " HELLO":

public class LeftPadderTest {

    @Rule
    public TestRule skippyRule = Skippy.predictWithSkippy();

    @Test
    void testPadLeft() {
        var input = TestConstants.HELLO;
        // assertEquals(" hello", LeftPadder.padLeft(input, 6));
        assertEquals(" HELLO", LeftPadder.padLeft(input, 6));
    }

}

Re-run the tests:

./gradlew test --rerun

Skippy detects the change and runs LeftPadderTest again:

LeftPadderTest > testPadLeft() FAILED
    org.opentest4j.AssertionFailedError: expected: < HELLO> but was: < hello>
    
RightPadderTest > testPadRight() SKIPPED

Content of .skippy/predictions.log:

...,com.example.LeftPadderTest,EXECUTE,BYTECODE_CHANGE_IN_TEST
...,com.example.RightPadderTest,SKIP,NO_CHANGE

Experiment 4

Undo the changes from the previous experiment:

git stash

Lastly, let’s see what happens if you change the value of the constant in TestConstants:

class TestConstants {

    // static final String HELLO = "hello";
    static final String HELLO = "bonjour";

}

Re-run the tests:

./gradlew test --rerun

Skippy detects the change and runs both tests:

LeftPadderTest > testPadLeft() FAILED
    org.opentest4j.AssertionFailedError: expected: < hello> but was: <bonjour>
       
RightPadderTest > testPadRight() FAILED
    org.opentest4j.AssertionFailedError: expected: <hello > but was: <bonjour>

Content of .skippy/predictions.log:

...,com.example.LeftPadderTest,EXECUTE,BYTECODE_CHANGE_IN_TEST
...,com.example.RightPadderTest,EXECUTE,BYTECODE_CHANGE_IN_TEST

Congratulations - You’ve completed the tutorial! By now, you should have a solid idea how Skippy works.