Overview

gitflow-oss-java-slim is a minimal, opinionated java Continuous Integration framework which uses Gradle and Github Actions to support a simplified version of the Gitflow development model.

It is primarily targeted towards Open source Java developers and organizations that need to manage the complexity of building, versioning and releasing binary code artifacts.

Is this framework for me?

In order to make the best use of your time (and mine), reflect for a moment.

If you:

• Develop and manage multiple Java libraires on a daily basis.
• Do not have dedicated CI infrastructure servers or cloud services.
• Find yourself copy/pasting boilerplate publication configuration code for each code base that you manage.
• Struggle to assign and manage development/release versions for your artifacts.
• Publish and make use of both internal and external artifacts regularly.
• Share artifact dependencies amongst your projects, and need to know when a particular project's internal dependencies are lagging behind.
• Do all of the above using internal and external Maven distribution repositories.

If at least 4 of the 7 points above apply to you, this framework may be of meaningful use.

Otherwise you can stop here (and perhaps share some insight on how you tackle the 7 problems above).

Build System Overview

In a nutshell, this build system brings together a few concepts which are common across many code-bases within an organization, scoped down to the problem of managing Java code as shared Maven artifacts. It's possible to implement functionality to perform more advanced tasks like Docker image builds or build status reporting, but let's keep the scope short and simple for now.

The diagram above illustrates the flow of release management when you work on a code base using this Gradle plugin/Github action combination. Depending on the type of release artifact being built by Github actions, the artifacts themselves will be routed to different target Maven repositories, if any.

The advantage of this approach is that you can have multiple code bases compiled under a single organizational build configuration.

Release artifact versions are defined as:

• SNAPSHOT: A version that you're experimenting or prototyping on, and SHOULD only be consumed by other development team members who understand that this code can break and change very quickly (and thus break their builds as well).
• MILESTONE: A version that you've tested to a satisfactory level, and should be consumed by others, with the knowledge that minor changes MAY break their build.
• RELEASE: A version that you've tested thoroughly and can be consumed by other developers with a high expectation of stability and functionality.

Note: These intermediate version types are meant to serve as a complement to Semantic Versioning conventions, which are highly recommended by this project, and help your project manage the way in which software versions break downstream consumer builds of your shared artifacts.

Consider a Gradle project with Kotlin syntax defining an artifact called, my-library:

group = "io.vacco.mylibrary"
version = "0.1.0"

During development, the following are examples of intermediate artifact type versions added by this framework:

Notice that the MILESTONE version convention also appends a timestamp component in the form of YYYYMMDDHHMM, so as to give the intermediate version a unique number. The reason for this is that most Maven repositories only allow re-reploying SNAPSHOT versions, and MAY reject versions which already exist inside the repository.

Some further improvement and customization options for this naming convention may include code or configuration level overrides to allow for simple number counters or some other form of uniqueness convention. In practice, timestamps seem to be a good solution to track information about a particular artifact MILESTONE version.

At this point you may be asking yourself: "why is there a distinction between a SNAPSHOT and a MILESTONE version?" This is explained in the following section, where the relationship between a Git development branch name and the artifact version type it produces is established.

Gitflow Branching Strategy

To define which branches produce which kinds of artifacts, this framework offers a minimal set of definitions and rules which sit somewhere in the middle of the Gitflow Branching Model and the GitHub Flow Branching Model.

In the figure above there are four types of branches, and only three of them produce artifacts:

feature/XYZ branches

These branches can be used to extensively commit and test a new feature. They produce SNAPSHOT artifacts, which are always re-deployable by Maven. These branches have the highest commit frequency, and they SHOULD get squash-merged (and then quickly discarded) into the next type of branch.

The develop branch

This is a single branch where many feature/XYZ branches converge into to produce MILESTONE artifacts, and contains something which a team is starting to get confident about releasing, but still requires more testing. The develop branch SHOULD get merged straight into master/main so that they are always even (0 commits ahead/0 commits behind).

This gives you certainty that MILESTONE artifacts originated from a well-know Git commit hash.

The master/main branch

This branch MUST be the only branch from which RELEASE artifacts are derived, but does NOT itself produce such RELEASE artifacts. Instead, it acts as a PRERELEASE build stage where a Pull request from develop into master/main can get reviewed one last time. For example:

• Make sure that documentation is up to date.
• Perform release gating (i.e. make sure the right dependencies are getting imported).
• Determine official release version major, minor and patch number increments etc.).

tags/X.Y.Z

Tags derived from the master/main branch freeze a commit in the master/main branch timeline, and produce RELEASE artifacts which only contain only major, minor and patch version numbers for the artifact.

Note: while it MAY seem that this model imposes excessive burden on branch management, it's also possible to discard the develop branch entirely, and only work with feature/xyz branches which merge directly into master/main since the only important point where RELEASE artifacts are created are tags: tag/X.Y.Z.

One applicable use case for dropping the use of the DEVELOP branch could be where a single developer is working on multiple code bases, so it's no necessary to impose the level of control that the develop branch offers.

With these concepts, we can now define how to configure Gradle and Github Actions to implement this framework.

Github action parameters

In this framework both Github Actions and Gradle work together to provide development methods for different circumstances. Namely:

• Un-managed builds - These are the builds that you perform in your local development machine, i.e. gradle clean build. They produce SNAPSHOT artifacts that you can deploy to your local Maven repository.
• Managed builds - These are builds running under Github Actions, and any produced artifacts get routed to a target Maven repository.

Here is the minimal amount of configuration required under .github/workflows/main.yml:

name: Gradle Build
on: {push: {tags: null}}
jobs:
  build:
    runs-on: ubuntu-18.04
    steps:
      - uses: actions/checkout@v2
      - uses: vaccovecrana/gitflow-oss-java-slim@0.9.8
        with:
          orgConfig: https://my-bucket.s3.us-east-2.amazonaws.com/my-org-config.json
        env:
          SONATYPE_USER: ${{secrets.MY_ORGS_SONATYPE_USER}}
          SONATYPE_PASSWORD: ${{secrets.MY_ORGS_SONATYPE_PASSWORD}}
          MAVEN_SIGNING_PRV: ${{secrets.MY_ORGS_MAVEN_SIGNING_KEY}}

There are two key parameters that are needed by this framework:

orgConfig is the core piece of this framework which defines common build conventions which will drive the build process for a wide set of projects in your organization. It is defined as a JSON document and it MAY be validated with a JSON schema provided by this framework. It gets sourced from a URL that users in your organization can securely access.

How you implement said access is up to you.

In the example above, I am providing a URL pointing to a publicly accessible S3 bucket location, since I am sure that no sensitive information is contained inside the document. Another option could be to store your organization's common build configuration in a Github Gist. And as a final example, you could also lock an HTTP server containing the JSON document behind a private VPN, which would impose an extra step on your developers when they need to work with your organization's code bases.

Notice that the Github action's env property defines a set of keys/values that the organization's common build configuration MAY require to contact the Maven repositories defined in the configuration, along with credentials to sign artifacts when building RELEASE versions of a code base. Names for these keys and values are not mandated by this framework. They can be whatever keys/values you want them to. However you must make sure that the variable references inside the Org configuration document reference the right variable names.

Warning: you SHOULD be mindful with the secret values that you are sharing with this framework. In general, you SHOULD follow security practices which grant the minimal amount of credentials that will grant this action with the privileges to execute and deploy your build. A good set of sane recommendations can be found in the Configure AWS Credentials Github Action.

The syntax and purpose of the Organization's common build configuration is explained in the next section.

Org Config Specification

Having defined the underlying concepts needed for this framework, we can now define an Organization's common build configuration.

It is a JSON document which consists of a few sections backed by a JSON schema

{
  "orgId": "my-org-id",
  "internalRepo": {...},
  "snapshotsRepo": {...},
  "releasesRepo": {...},
  "publishing": {...},
  "devConfig": {...}
}

orgId is a simple String identifier for your organization (usually a Github username), and is necessary to store temporary files while working with local code bases.

The devConfig attribute specifies default versions of optional tools that you could use in your code bases. In the current version of this framework, the set of recommended tools are PMD and j8spec. Note that these are completely optional and you are not obligated to use them along with this framework.

The remaining three *Repo attributes define the three components we saw in the architecture diagram in previous chapters. They establish where your Maven repositories (if any) are located and what credentials they require.

These, along with the publishing attribute will be discussed in the next section.

Maven Publication Repositories

Maven repository definitions

"internalRepo": {
  "id": "GithubPackages",
  "url": "https://maven.pkg.github.com/my-github-org/common-packages/",
  "usernameEnvProperty": "MY_ORGS_CI_USER",
  "passwordEnvProperty": "MY_ORGS_CI_TOKEN"
}

These configuration blocks should be pretty self-explanatory:

• internalRepo refers to a Maven repository where you intend to store artifacts for private use inside your organization. Note that this repository MUST allow for both SNAPSHOT and RELEASE artifacts in the same location for practical purposes. With enough community interest, this could change in the future.

• snapshotsRepo and releasesRepo refer to remote Maven repositories which intend to store publicly accessible SNAPSHOT and RELEASE artifacts respectively.

Each configuration block can source an access username password by reading values from Environment variable names which you designate, or by reading direct username and password values at runtime inside the Org config file itself. These variable names and values SHOULD be stored as Github secrets.

Note: if you publish libraries to Sonatype's OSS repositories (Maven Central), you would configure your Sonatype deployment username and password and Github secrets.

Direct credentials input SHOULD NOT be used for CI builds, and is provided only for local development scenarios where you checkout a source code tree and work on it in your laptop or workstation. Specific details on this are discussed in the next section.

Lastly, to determine exactly which repository definitions/combinations should you be using depends on your particular development practices. For example, if you develop pure OSS components, it may not make much sense to define an internalRepo block inside the Org config file, since Sonatype's staging and release repositories (Maven Central) may be enough to fit your use case.

On the other hand, if you also develop software for commercial purposes, then it may make sense to align your internal development code bases to source artifacts from a private Maven repository (such as Nexus, Artifactory, Strongbox, Github Packages, etc.).

Publication Metadata

The publishing attribute configures OSS publication metadata that is required when you publish artifacts to Sonatype's OSS repositories (i.e. developer ids, contact and license information that you would include in a classic Maven project's pom.xml file).

"publishing": {
  "id": "an Org id (for example yoyodyne)",
  "devId": "a dev id (for example jhacker)",
  "devContact": "an Org name (for example Yoyodyne, Inc. or James Hacker)",
  "devEmail": "jhacker@yoyodyne.com",
  "mavenSigningKeyEnvProperty": "MAVEN_SIGNING_PRV"
}

Maven PGP Signing key

The last configuration is an environment variable which SHOULD point to an environment variable containing an ASCII armoured, password-less PGP private key used to sign release artifacts. The reason for extracting raw key material in this form is that Github Secrets already offers a good protection mechanism for storing they key material without having to go through the management hassles of PGP key rings, and it works well and safely in practice.

The key material should look similar to this:

-----BEGIN PGP PRIVATE KEY BLOCK-----
...a lot more characters...
-----END PGP PRIVATE KEY BLOCK-----

The key material will be passed on to the Gradle Signing plugin.

Given enough community interest, it should be possible to extend this part of the framework to ask for a key's passphrase, also stored as a Github secret.

The next section discusses the rest of the Org config file format.

Development Features Configuration

The last section of the Org config file is defined as:

"devConfig": {
  "jdkDistribution": "https://api.adoptopenjdk.net/v3/binary/latest/11/ga/linux/x64/jdk/hotspot/normal/adoptopenjdk",
  "pmdRulesUrl": "https://vacco-oss.s3.us-east-2.amazonaws.com/vacco-pmd.xml",
  "dependencyExcludedGroups": ["net.sourceforge.pmd"],
  "versions": {
    "gradle": "gradle-6.8.3",
    "j8Spec": "io.github.j8spec:j8spec:3.0.0",
    "pmd": "6.27.0"
  }
}

jdkDistribution requires a URL to download a specific version of the JDK that will only be used by Github actions to execute a build. This will not affect your local development environment, so you're free to choose what JDK you use to build locally. In the example above, the JDK gets downloaded from api.adoptopenjdk.net, and it MUST point to a plain tar .tar or compressed .tar.gz file which, upon de-compression, will extract the JDK's contents into a single output directory.

Note: in practice, most JDK distribution bundles follow this format, but you SHOULD verify with any custom JDKs you decide to use. For example, you could copy the JDK distribution file from the web site above into your own private S3 bucket (or some other cloud or internal organization storage location) so that your Github Actions builds do not depend on the availability of other websites.

The remaining configuration options are configured here, but get actually activated by the build.gradle.kts file in your source tree, depending on which features you decide to use, and are further explained in later sections.

pmdRulesUrl is optional. It is only required if you decide to make use of the Gradle PMD plugin. It points to a location where a PMD xml rule-set can be downloaded and applied to the current build.

dependencyExcludedGroups defines a set of Maven groups which will be excluded when the Gradle Versions plugin gets applied to the build.

In other words, all builds will apply the Gradle versions plugin by default, and will report on which outdated dependencies your project has. In the example above, I am deciding to ignore any outdated versions under the net.sourceforge.pmd Maven group since I am certain that they are not relevant to any of my Github projects.

Lastly, the versions block allows you to configure:

• gradle - The Gradle distribution version to execute this build. It is specified in X.Y.Z format and is downloaded from services.gradle.org to execute a build under Github actions. It does not affect which Gradle distribution you use for local development.
• j8spec - The version of jspec to use. It is defined in Maven dependency notation format.
• pmd - The version of the PMD tool that the Gradle plugin will use (when configured) in a build.

Configuration Evolution

It is possible (and likely) that you may maintain different source projects which require slight variations amongst each other. For example:

• Create a java8.json configuration if you maintain old projects which mandate version 8 of the JDK (or different PMD rules for a good portion of them).
• Create a java11.json configuration for your stable projects.
• Create a java15.json configuration for research and prototyping projects.

https://my-bucket.s3.us-east-2.amazonaws.com/my-org-config-java8.json
https://my-bucket.s3.us-east-2.amazonaws.com/my-org-config-java11.json
https://my-bucket.s3.us-east-2.amazonaws.com/my-org-config-java15.json

This can help you maintain control on how quickly do you wish to move your projects to new versions of the JDK, Gradle itself, etc.

Gradle Features

In this section we'll discuss how to configure a Gradle project to make use of this framework. The key idea is that this framework SHOULD be minimally intrusive to your Gradle project, and SHOULD allow you to detach from it at any moment.

Start by defining publication information in your gradle.properties file:

libGitUrl=https://github.com/my-org/my-project.git
libDesc=This is a brief description of what my library does
libLicense=Apache License, Version 2.0 (choose a license)
libLicenseUrl=https://opensource.org/licenses/Apache-2.0 (a link to your license text)

Now include and configure the Gradle plugin in your source tree in build.gradle.kts:

plugins { id("io.vacco.oss.gitflow") version "0.9.8" }

group = "com.myorg.mylibrary" // your project's target maven coordinates.
version = "0.1.0" // or whichever version you have

configure<io.vacco.oss.gitflow.GsPluginProfileExtension> {
  // add other configuration features here
  sharedLibrary(true, true)
  addJ8Spec()
  addPmd()
  addClasspathHell()
}

The currently supported set of optional features are:

sharedLibrary(boolean publish, boolean internal) configures the Gradle project (or sub-project) to produce a shared Java library. When publish is false, the java libraries produced will NOT be published to target Maven repositories. For example if you have test support libraries (like my-library-test-assets) that are not intended to be used as part of a library's main binaries.

The internal parameter will only have effect when publish is true, and will determine if the compiled artifacts will get published to your organization's internal Maven repository, or any repositories you configured for public SNAPSHOT and RELEASE access (like Sonatype's OSS servers).

addJ8Spec() will add j8spec to your testImplementation class path.

addPmd() will apply PMD code quality checks based on the rule set defined by the Org config used by the Gradle project.

addClasspathHell() will apply the Class path Hell Gradle plugin.

addGoogleJavaFormat() will apply the Google Java format Gradle plugin to the build, automatically formatting sources during a build.

Note: when applying these optional features, all of them can still be customized by their respective declarative configuration blocks. Thus your source project is still free to customize each optional feature as it deems fit.

Org Config / Project bootstrap cheat-sheet

The previous sections outlined the concepts and structure behind this framework.

The following is a quick walk-through on how to create an Org config from scratch, and configure a single Gradle project to adhere to it. In this example, PMD is disabled and only external OSS publication support is enabled.

Start by creating an Org config template, filling in required parameters:

{
  "orgId": "my-org-id",
  "snapshotsRepo": {
    "id": "SonatypeOSSSnapshots",
    "url": "https://oss.sonatype.org/content/repositories/snapshots/",
    "usernameEnvProperty": "SONATYPE_USER",
    "passwordEnvProperty": "SONATYPE_PASSWORD"
  },
  "releasesRepo": {
    "id": "SonatypeOSSStaging",
    "url": "https://oss.sonatype.org/service/local/staging/deploy/maven2/",
    "usernameEnvProperty": "SONATYPE_USER",
    "passwordEnvProperty": "SONATYPE_PASSWORD"
  },
  "publishing": {
    "id": "an Org id (for example yoyodyne)",
    "devId": "a dev id (for example jhacker)",
    "devContact": "an Org name (for example Yoyodyne, Inc. or James Hacker)",
    "devEmail": "jhacker@yoyodyne.com",
    "mavenSigningKeyEnvProperty": "MAVEN_SIGNING_PRV"
  },
  "devConfig": {
    "gradleVersion": "7.0",
    "gradleDistribution": "https://services.gradle.org/distributions/gradle-7.0-bin.zip",
    "jdkDistribution": "https://api.adoptopenjdk.net/v3/binary/latest/11/ga/linux/x64/jdk/hotspot/normal/adoptopenjdk",
    "versions": {
      "j8Spec": "io.github.j8spec:j8spec:3.0.0"
    }
  }
}

Next, go to the Github Secrets section of your repository, and configure the following values:

• SONATYPE_USER - your Sonatype OSS repository username.
• SONATYPE_PASSWORD - your Sonatype OSS account password.
• MAVEN_SIGNING_PRV - An ASCII armoured version of you Maven PGP signin key.

You can export the PGP private key intended to sign releases with the following command:

gpg --output private.pgp --armor --export-secret-key jhacker@yoyodyne.com

Note: In general, it should be safe to export subkeys derived from a PGP master private key. By consequence, it is never recommended that you store a master private key as a Github Secret to sign Maven artifacts.

Set the values required, and upload this to an S3 bucket, Dropbox or a Github Gist. Keep the URL where you uploaded this configuration.

Local development configuration

On your local development machine, create a file called .gsOrgConfig.json inside your home folder. Include the following contents:

{
  "orgId": "my-org-id",
  "orgConfigUrl": "https://<Location where you uploaded the Org Config file>"
}

Gradle Project setup

Next, switch over to the project you'd like to manage with this framework, and set:

.github/workflows/main.yml

name: Gradle Build
on: {push: {tags: null}}
jobs:
  build:
    runs-on: ubuntu-18.04
    steps:
      - uses: actions/checkout@v2
      - uses: vaccovecrana/gitflow-oss-java-slim@0.9.8
        with:
          orgConfig: https://<Location where you uploaded the Org Config file>
        env:
          SONATYPE_USER: ${{secrets.MY_ORGS_SONATYPE_USER}}
          SONATYPE_PASSWORD: ${{secrets.MY_ORGS_SONATYPE_PASSWORD}}
          MAVEN_SIGNING_PRV: ${{secrets.MY_ORGS_MAVEN_SIGNING_KEY}}

gradle.properties:

libGitUrl=https://github.com/my-org/my-project.git
libDesc=This is a brief description of what my library does
libLicense=Apache License, Version 2.0 (choose a license)
libLicenseUrl=https://opensource.org/licenses/Apache-2.0 (a link to your license text)

Finally, include and configure the Gradle plugin in your source tree in build.gradle.kts:

plugins { id("io.vacco.oss.gitflow") version "0.9.8" }

group = "com.myorg.mylibrary" // your project's target maven coordinates.
version = "0.1.0" // or whichever version you have

configure<io.vacco.oss.gitflow.GsPluginProfileExtension> {
  sharedLibrary(false, true) // external library with publication support
  addJ8Spec()
  addPmd()
  addClasspathHell()
}

Once this is done, the following managed builds will take place:

• Committing a feature/XYZ branch will produce and upload SNAPSHOT artifact into the Sonatype OSS snapshots repository. One such branch could be named feature/performance-improvements.
• Committing into the develop branch will build and upload a MILESTONE artifact into the Sonatype OSS snapshots repository.
• Creating a tag out from master or main will build and upload a RELEASE artifact into the Sonatype OSS releases repository. Once you close and release the temporary repository, the artifact will be available in Maven Central.