Using version catalog from buildSrc buildlogic.xyz-common-conventions scripts

Hi

I have a toy example of a hello world project to learn more about version catalog
I was using gradle 8.7 (tested on 8.8-rc-1 also, same result)
Created a standard projects that is created calling gradle init

So - I’ve stumbled into something and I kinda hope its not a bug - don’t know - here it is:

here is a dependency block from app/build.gradle.kts

dependencies {
//    implementation("org.apache.commons:commons-text")
    implementation(libs.commons.text)
    implementation(project(":utilities"))
    implementation(project(":sutilities"))
}

Implementation for commons-text from version catalog works fine

here is a dependency block from buildSrc/.../kotlin/buildlogic.kotlin-common-conventions.gradle.kts

dependencies {
    implementation("org.apache.commons:commons-text")
//        implementation(libs.commons.text)
}

Implementation for commons-text from version catalog doesn’t works, need to use old way

My preference is to have dependencies in the buildSrc common conventions whenever I can
However, if I can’t do this from the version catalog that kind of a bummer

What am I doing wrong?

To use libs.versions.toml in buildSrc, you need to do this, add a settings.gradle.kts to buildSrc that reads as follows

dependencyResolutionManagement {
	versionCatalogs {
		create("libs") {
			from(files("../gradle/libs.versions.toml"))
		}
	}
}

for example

Unfortunately what @livk says is correct, but not helpful for your question.
It only allows you to use the version catalog in buildSrc/build.gradle.kts, but not inside your convention plugin.
Inside your convention plugin you either have to use a string-y API to get the values from the version catalog of the main build, or my hack-around documented at Make generated type-safe version catalogs accessors accessible from precompiled script plugins · Issue #15383 · gradle/gradle · GitHub.

Thanks ivik! But I humbly admin that the example you provide is way over my head. It looks like a nice plugin implementation - which wasn’t exactly what I seek

First I’ll add that I have the toml file nicely configured, with all settings.gradle as advised
Within my subprojects I can easily access the lib.* dependencies

Question is: how do I access lib.* from the common buildSrc/src/main/kotlin/*common*.gradle.kts files?

Dear Vampire,

I’ve tried the hack our of curiosity and failed miserably to compile it :slight_smile:
I also must admit that in my production env if one of my colleges would try to pass this the PR will never be accapted :joy:

string-y API meaning the old school string based?

Thanks ivik! But I humbly admin that the example you provide is way over my head. It looks like a nice plugin implementation - which wasn’t exactly what I seek

It is not a plugin implementation.
It just tells the buildSrc build to use the same version catalog that your main build is using, but only allows to use the version catalog in buildSrc/build.gradle.kts, but not in the precompiled script plugins where you intended to use it.
But actually it is a partly solution, as it is also a necessary part of my hack-around.

I’ve tried the hack our of curiosity and failed miserably to compile it

Then you didn’t properly follow it.
All shown parts are necessary that it works and many projects out there successfully use it for several years already.

I also must admit that in my production env if one of my colleges would try to pass this the PR will never be accapted

That’s quite possible, as I said, it is a hack-around. :slight_smile:

string-y API meaning the old school string based?

If you mean using implementation("org.apache.commons:commons-text"), then no, that is not using a string-y version catalog API, but just the string representation of the dependency directly.
The string-y API would for example be implementation(versionCatalogs.named("libs").findLibrary("commons-text").orElseThrow(::AssertionError)).

It is documented at Sharing dependency versions between projects and since Gradle 8.5 you can use versionCatalogs instead of extensions.getByType<VersionCatalogsExtension>()or the<VersionCatalogsExtension>().

Thank you thank you thank you :raised_hands:
string-y API example worked-out and solved my issue

Here is a small boilerplate removal function


fun DependencyHandler.implementationFromCatalog(libName: String) {
    versionCatalogs.named("libs").findLibrary(libName).ifPresentOrElse(
        { implementation(it) },
        { println("Library '$libName' not found in version catalog.") }
    )
}

dependencies {
    implementationFromCatalog("commons-text")
}

Follow-up question: We have several build-logic scripts, how can I share this function among them?

edit:
I couldn’t help myself from adding some more version catalog goodies in our build scripts
so I found that we need a different function for each dependency type. As someone who dislike boilerplate I did some refactoring:

fun DependencyHandler.testImplementationFromCatalog(libName: String) = dependencyFromCatalog(libName, ::testImplementation)
fun DependencyHandler.implementationFromCatalog(libName: String) = dependencyFromCatalog(libName, ::implementation)
/** add you own as needed */

fun dependencyFromCatalog(libName: String, dependency:  (Any) -> Dependency?) {
    versionCatalogs.named("libs").findLibrary(libName).ifPresentOrElse(
        { dependency(it) },
        { logger.warn("Library '$libName' not found in version catalog.") }
    )
}

Put it in a regular .kt file and then use it.

You can imagine things in a .kts file as the content of a class, which is roughly what happens, so the other scripts cannot access the things defined there.

import org.gradle.api.artifacts.VersionCatalogsExtension
import org.gradle.api.Project
import org.gradle.api.artifacts.Dependency
import org.gradle.api.logging.Logger

class VersionCatalog(project: Project) {

    private val versionCatalog = project.extensions.getByType(VersionCatalogsExtension::class.java).named("libs")
    private val logger: Logger = project.logger

    fun dependencyFromCatalog(libName: String, dependency: (Any) -> Dependency?) {
        versionCatalog.findLibrary(libName).ifPresentOrElse(
            { dependency(it) },
            { logger.warn("Library '$libName' not found in version catalog.") }
        )
    }
}

fun versionCatalog(project: Project, libName: String, dependency: (Any) -> Dependency?) =
    VersionCatalog(project).dependencyFromCatalog(libName, dependency)

and in the common build-logic

dependencies {
    versionCatalog(project, "commons-text", ::implementation)
}

OK, got this working

  1. Is there a way to infer project?
  2. is there a way to infer ::implmentation?
    So we can have implementationVersionCatalog("commons-text")

You probably want something like

fun Project.implementationVersionCatalog(libName: String) {
    the<VersionCatalogsExtension>()
        .named("libs")
        .findLibrary(libName)
        .ifPresentOrElse(
            { configurations["implementation"].dependencies.add(dependencies.create(it)) },
            { logger.warn("Library '$libName' not found in version catalog.") }
        )
}

instead of your VersionCatalog class.

But this is now more a Kotlin implementation question, not so much a Gradle question anymore. :smiley:

So, again many thank :slight_smile: :grin:

Your snippet gave some strange errors, however it did open the door to the two missing pieces, here is what eventually worked for me:

import org.gradle.api.Project
import org.gradle.api.artifacts.VersionCatalogsExtension
import org.gradle.api.logging.Logger

class VersionCatalogHelper(private val project: Project) {

    private val versionCatalog = project.extensions.getByType(VersionCatalogsExtension::class.java).named("libs")
    private val logger: Logger = project.logger

    fun dependency(libName: String, configurationName: String) {
        versionCatalog.findLibrary(libName).ifPresentOrElse(
            { library ->
                project.dependencies.add(configurationName, library)
            },
            { logger.warn("Library '$libName' not found in version catalog.") }
        )
    }
}

// Extension functions for different configurations
fun Project.implementationFromCatalog(libName: String) {
    VersionCatalogHelper(this).dependency(libName, "implementation")
}

fun Project.testImplementationFromCatalog(libName: String) {
    VersionCatalogHelper(this).dependency(libName, "testImplementation")
}

fun Project.compileOnlyFromCatalog(libName: String) {
    VersionCatalogHelper(this).dependency(libName, "compileOnly")
}

fun Project.testRuntimeOnlyFromCatalog(libName: String) {
    VersionCatalogHelper(this).dependency(libName, "testRuntimeOnly")
}

edit: Fixed according to comment below :slight_smile:

1 Like

I don’t think library.get().toString() is a good idea.
Probably better project.dependencies.addProvider(configurationName, library).

1 Like

Glad to report that tips / guidance in thread thanks to @Vampire really helped us tidy up all of our projects! Kudos!

We added to version catalog toml the following plugins

[plugins]
badass = { id = "org.beryx.runtime", version = "a.b.c" }
jib = { id = "com.google.cloud.tools.jib", version = "x.y.z" }

It took some time and several clean/builds until this next lines didn’t produce errors

plugins {
    alias(libs.plugins.jib)
}

not sure if this was my problem or a gradle problem?

using alias from subprojects works fine
using alias from buildSrc/src/main plugins don’t work

The problem I’m trying to solve is define a common jib/application blocks that use minimal (shared) parameter. Remove code duplicity so each application can share (we have many many applications and we always get bugs when a conventions changes, basically only the app name and jvm args need to change)

What I’m trying to do is this:
in app-common-conventions.gradle.kts

plugins {
    application                                     // Works fine!
    aliasFromCatalog("plugins.jib")                 // Doesn't work
    id("com.google.cloud.tools.jib")                // Doesn't work
    alias(libs.plugins.jib)                         // Doesn't work
}
val appClassName = ...
val appJvmParams = ...
// define jib {} 
// define application {} 

Perhaps the direction of my solution is a little stupid :slight_smile:
Any advice how can I share code? Is the idiomatic solution to write a kt file also like before?

You just cannot, not even with my hack-around.

The plugins { ... } block is extracted and applied to a dummy project where neither the effects of my hack-around would be present, nor anything you put in a .kt file.

You have there to use the id("...") variant that you say does not work.

But of course you need to add the plugin as a dependency to your plugin build.
If you just want to apply it by id, runtimeOnly would be sufficient, if you also need the types for example to configure an extension of it and so on, you need implementation.

I want to share solution we ended up using on the account it might help someone

First, the problem we tried to solving were:

  1. Use toml version catalog libraries within buildSrc plugins
  2. Use toml version catalog plugins within buildSrc plugins

For #1 all the the posts up to this point explain how this was solved

For #2, we ended up with changing the problem to: “write a plugin act as a facade to other plugins”.
The Rational behind this is simple - we actually want a single point of through. The reason we want a version catalog is to have a single place to update the specific version we need. If this single point exist in a different location (gradle file) it is also a valid solution.

Specifically our applications uses JIB and application plugins. Initially we wanted to make sure that all apps use the same JIB version, but thinking it over we noticed that actually both plugins share the same configuration and we figured this could also be addressed

So, we defined a custom extension!

Here is AppConventionExtension.kt class

open class AppConventionExtension(private val project: Project) {
    var mainClass: String = ""
    var artifactName: String = ""
    var jvmFlags: List<String> = ...
    val baseImage: String = ...
    val imageTag: String by lazy { ... }
}

It is a class that will be used to carry information from our subproject apps to the inner plugins

And, a matching custom plugin called app-common-conventions.gradle.kts

plugins {
    application
    id("com.google.cloud.tools.jib")
}

// Register the custom extension
val app = extensions.create("app", AppConventionExtension::class.java, project)

/**
 * make sure the app {} block is evaluated before running JIB/application
 */
gradle.projectsEvaluated {
    application {
        if (app.mainClass.isEmpty()) {
            logger.warn("While building application: app.mainClass is empty, please define where you apply the 'app-common-conventions.gradle.kts' plugin")
        }
        if (app.artifactName.isEmpty()) {
            logger.warn("While building application: app.artifactName is empty, please define where you apply the 'app-common-conventions.gradle.kts' plugin")
        }

        mainClass.set(app.mainClass)
        applicationDefaultJvmArgs = app.jvmFlags
    }

    jib {
        if (app.mainClass.isEmpty()) {
            logger.warn("While building JIB: app.mainClass is empty, please define where you apply the 'app-common-conventions.gradle.kts' plugin")
        }
        if (app.artifactName.isEmpty()) {
            logger.warn("While building JIB: app.artifactName is empty, please define where you apply the 'app-common-conventions.gradle.kts' plugin")
        }

        from {
            image = app.baseImage
        }
        container {
            appRoot = "/app-${app.artifactName}"
            mainClass = app.mainClass
            jvmFlags = app.jvmFlags
        }
        to {
            image = "ghcr.io/ssi-dnn/${app.artifactName}"
            tags = setOf(app.imageTag, "latest")
        }
    }
}

One last thing in the configuration side - in buildSrc/build.gradle.kts we also added a dependency to the specific version of the JIB plugin

dependencies {
    implementation(libs.kotlin.gradle.plugin)
    implementation("com.google.cloud.tools:jib-gradle-plugin:3.3.1")
}

This is how we use it in our subprojects

plugins {
    // Add the app plugin 
    id("app-common-conventions")
}

// Add the app block
app {
    mainClass = ...
    artifactName = ...
}

And that’s it!
Next step will be to publish this as a plugins so all of our different repos can share this

Some more notes. :slight_smile:

  • You can define the plugin version in your version catalog and declare in the buildSrc/settings.gradle.kts to use that version catalog. This makes the version catalog accessible in buildSrc/build.gradle.kts and you can use it to depend on the plugin like you do with the string now. You just have to either translate the plugin from the version catalog to a dependency, or right away declare it as library, unless Accept plugin declarations from version catalog also as libraries · Issue #17963 · gradle/gradle · GitHub gets implemented. The marker artifact is <plugin id>:<plugin id>.gradle.plugin:<plugin version>.
  • When defining an extension you should always use Propertys, so that you can do lazy binding and also deriving the value of one from the value of another. See here for more information: Configuring Tasks Lazily. That should also then avoid using things like gradle.projectsEvaluated { ... }

I tried to get rid of afterEvaluate but failed

this is what I have fixed - now Using properties

abstract class PublishExtension(private val project: Project) {
    @get:Input abstract val artifactId: Property<String>
    @get:Input abstract val groupId: Property<String>
    @get:Input abstract val version: Property<String>
    @get:Input abstract val sandbox: Property<Boolean>
}

as for the plugin itself - here are the main parts

import java.net.HttpURLConnection
import java.net.URL
import java.util.Base64

plugins {
    // id("java") // not needed here - when added gives trouble
    id("maven-publish")
}

// Register the `publish` custom extension
val publish = extensions.create("publish", PublishExtension::class.java, project)

val artifactory_username: String by project
val artifactory_password: String by project
val artifactory_publish_url: String by project
val artifactory_sandbox_url: String by project

fun hasSandbox() = hasProperty("sandbox") || hasProperty("snapshot") || publish.sandbox.getOrElse(false)
fun hasSandboxOrElse(sandboxCase: String, orElseCase: String) = (if (hasSandbox()) sandboxCase else orElseCase)

//afterEvaluate {
    publishing {
        publications {
            create<MavenPublication>("maven") {
                from(components["java"])
                groupId = publish.groupId.getOrElse(project.group.toString())
                artifactId = publish.artifactId.get()
                version = publish.version.getOrElse(versionFromFile())
            }
        }

        repositories {
            maven {
                url = uri(hasSandboxOrElse(artifactory_sandbox_url, artifactory_publish_url))
                credentials {
                    username = artifactory_username
                    password = artifactory_password
                }
            }
        }
    }
//}

/**
 * check if the artifact we are about to publish already exists in artifactory
 * do not check for sandbox since it is OK to publish twice
 */
val checkArtifactExists: TaskProvider<Task> = tasks.register("checkArtifactExists") {
    doLast { ... }
}

// Ensure the publish task depends on the checkArtifactExists task
tasks.named("publish") {
    dependsOn(checkArtifactExists)
}

This works only when I comment in afterEvaluate - is that so harmful?
What are my alternatives?

Independent of the question, why do you inject the project into the extension?
And why it is a class at all?
This should do the same:

interface PublishExtension {
    @get:Input val artifactId: Property<String>
    @get:Input val groupId: Property<String>
    @get:Input val version: Property<String>
    @get:Input val sandbox: Property<Boolean>
}

is that so harmful?

The main effect of afterEvaluate { ... } is to add timing problems, odering problems, and race conditions.

What are my alternatives?

The actual problem is, that you try to read the extension too soon.
If you create it and then immediately read the values, the consumer of course had no chance to configure it.
If you use afterEvaluate you give it a chance to configure the extension before you read it.
But if he then himself uses afterEvaluate for some reason, you again read the values too early and get no or a different value, …

The sense of the properties / providers is to wire things together and read them as late as possible, optimally only at execution time where not configuration change should happen anymore.

The problem is, that for example the group, artifact, and version fields of the publication are not yet properties, so cannot be wired properly. Maybe this changes with Gradle 9, where a big step in the provider migration should hopefully be done.

Actually, for these fields of the publication my strong advice is, to not manipulate them at all, never. This for example disturbs easy usage in a composite build but then needs manual substitution rules while it would otherwise just work. So better fix the group, name, and version of the project and keep the defaults for the publication settings.

Another option if you really want to go that route is, to not have properties in your extension, but a function that then does the configuration. For example a function that you give group, artifactid, and version and that then configures the publication accordingly.

For the url, it might maybe be easier to just declare two repositories, then the caller can just decide by task which to publish to instead of project property. Alternatively the same as above, have a function in the extension that sets the url of the repository.

Thank you for the detailed reply

I’m interested in trying to implement again using your ideas
group/version are simple. We can avoid passing Project to the extension.

I have two unknowns

  1. artifactId parameter
  2. publish/publishToSandbox tasks

Is this the plan for the extension andartifactId?

interface PublishExtension {
    @get:Input val artifactId: Property<String>
}

Or do you imply that I can somehow write a function so the use can tell me what the artifactId is?

Regarding publishToSandbox
We have two objects here: 1) mutate group or cause the publishing to use a different group. 2) mutate the url or cause the publishing to use a different url

When this task is executed the battle has lost since the configuration phase is long gone. what is the proposed approach here?