Configuration and transitive dependency management by extension

mkobit · August 22, 2017, 2:23am

Sorry for a long post, most of it is the code sample and output!

I’m developing a plugin for writing libraries for Jenkins Pipelines and running into an issue with structuring configurations for various dependencies.

My goals right now:

Get source code completion to work in IDE (IntelliJ targeted) automatically
Delay resolution of configurations until execution phase (if possible)
Split out runtime needed dependencies that have different extension
Include transitive dependencies that have a specific extension into a runtimeOnly-type configuration, while also those with that extension

More on that last point - dependencies with an hpi extensions should be added to the runtimeOnly-type configuration, and a @jar type dependency of the same group/name/version should be added to the implementation-type configuration.

I’m having trouble figuring out the right way to tackle the dependency resolution while still getting automatic source code completion in IDE tooling without having to generate files and just using the built-in IntelliJ support. A way I can think of doing it is:

eagerly resolving the runtimeOnly configuration
going through the resolved artifacts
Find ones who have the hpi extension
Add a @jar extension dependency to the implementation dependency for each found hpi

This seems bad because I’ll be resolving a configuration in the configuration phase to determine the right dependencies to add the to implementation dependency. I can’t think of another way yet. I tried to look at https://docs.gradle.org/4.1/dsl/org.gradle.api.artifacts.ComponentSelectionRules.html and https://docs.gradle.org/4.1/javadoc/org/gradle/api/artifacts/ModuleDependency.html#exclude(java.util.Map) but those don’t have any way to deal with extensions.

Here is an example of some combinations of what comes after resolving the artifact:

wrapper.gradleVersion = '4.1'

repositories {
  maven {
    url = uri("https://repo.jenkins-ci.org/public/")
  }
}

configurations {
  mkobitTestNormal
  mkobitTestJar
  mkobitTestJarTransitive
  mkobitTestHpi
  mkobitTestHpiTransitive
}

dependencies {
  final dependencyName = "org.jenkins-ci.plugins.workflow:workflow-basic-steps:2.6"
  "mkobitTestNormal"(dependencyName)
  "mkobitTestJar"("${dependencyName}@jar")
  "mkobitTestJarTransitive"("${dependencyName}@jar") {
    transitive = true
  }
  "mkobitTestHpi"("${dependencyName}@hpi")
  "mkobitTestHpiTransitive"("${dependencyName}@hpi") {
    transitive = true
  }
}

tasks.create("mkobitTestConfigurations") {
  doFirst {
    final mkobitTestNormal = configurations.mkobitTestNormal
    final mkobitTestJar = configurations.mkobitTestJar
    final mkobitTestJarTransitive = configurations.mkobitTestJarTransitive
    final mkobitTestHpi = configurations.mkobitTestHpi
    final mkobitTestHpiTransitive = configurations.mkobitTestHpiTransitive
//    mkobitTest.exclue(mapOf("ext" to "hpi"))
    [
      mkobitTestNormal,
      mkobitTestJar,
      mkobitTestJarTransitive,
      mkobitTestHpi,
      mkobitTestHpiTransitive
    ].each {
      println("Configuration: '${it.name}")
      it.resolvedConfiguration.firstLevelModuleDependencies.each {
        println("  First level: $it")
      }
      println()
      it.resolvedConfiguration.resolvedArtifacts.each {
        println("  Resolved: $it")
      }
      println()
      println('-' * 100)
      println()
    }
  }
}

Output:

:mkobitTestConfigurations
Configuration: 'mkobitTestNormal
  First level: org.jenkins-ci.plugins.workflow:workflow-basic-steps:2.6;default

  Resolved: workflow-basic-steps.hpi (org.jenkins-ci.plugins.workflow:workflow-basic-steps:2.6)
  Resolved: workflow-api.hpi (org.jenkins-ci.plugins.workflow:workflow-api:2.16)
  Resolved: workflow-step-api.hpi (org.jenkins-ci.plugins.workflow:workflow-step-api:2.12)
  Resolved: mailer.hpi (org.jenkins-ci.plugins:mailer:1.18)
  Resolved: structs.hpi (org.jenkins-ci.plugins:structs:1.6)
  Resolved: display-url-api.hpi (org.jenkins-ci.plugins:display-url-api:0.2)
  Resolved: junit.hpi (org.jenkins-ci.plugins:junit:1.3)
  Resolved: symbol-annotation.jar (org.jenkins-ci:symbol-annotation:1.6)
  Resolved: annotation-indexer.jar (org.jenkins-ci:annotation-indexer:1.9)

----------------------------------------------------------------------------------------------------

Configuration: 'mkobitTestJar
  First level: org.jenkins-ci.plugins.workflow:workflow-basic-steps:2.6;default

  Resolved: workflow-basic-steps.jar (org.jenkins-ci.plugins.workflow:workflow-basic-steps:2.6)

----------------------------------------------------------------------------------------------------

Configuration: 'mkobitTestJarTransitive
  First level: org.jenkins-ci.plugins.workflow:workflow-basic-steps:2.6;default

  Resolved: workflow-basic-steps.jar (org.jenkins-ci.plugins.workflow:workflow-basic-steps:2.6)
  Resolved: workflow-api.hpi (org.jenkins-ci.plugins.workflow:workflow-api:2.16)
  Resolved: workflow-step-api.hpi (org.jenkins-ci.plugins.workflow:workflow-step-api:2.12)
  Resolved: mailer.hpi (org.jenkins-ci.plugins:mailer:1.18)
  Resolved: structs.hpi (org.jenkins-ci.plugins:structs:1.6)
  Resolved: display-url-api.hpi (org.jenkins-ci.plugins:display-url-api:0.2)
  Resolved: junit.hpi (org.jenkins-ci.plugins:junit:1.3)
  Resolved: symbol-annotation.jar (org.jenkins-ci:symbol-annotation:1.6)
  Resolved: annotation-indexer.jar (org.jenkins-ci:annotation-indexer:1.9)

----------------------------------------------------------------------------------------------------

Configuration: 'mkobitTestHpi
  First level: org.jenkins-ci.plugins.workflow:workflow-basic-steps:2.6;default

  Resolved: workflow-basic-steps.hpi (org.jenkins-ci.plugins.workflow:workflow-basic-steps:2.6)

----------------------------------------------------------------------------------------------------

Configuration: 'mkobitTestHpiTransitive
  First level: org.jenkins-ci.plugins.workflow:workflow-basic-steps:2.6;default

  Resolved: workflow-basic-steps.hpi (org.jenkins-ci.plugins.workflow:workflow-basic-steps:2.6)
  Resolved: workflow-api.hpi (org.jenkins-ci.plugins.workflow:workflow-api:2.16)
  Resolved: workflow-step-api.hpi (org.jenkins-ci.plugins.workflow:workflow-step-api:2.12)
  Resolved: mailer.hpi (org.jenkins-ci.plugins:mailer:1.18)
  Resolved: structs.hpi (org.jenkins-ci.plugins:structs:1.6)
  Resolved: display-url-api.hpi (org.jenkins-ci.plugins:display-url-api:0.2)
  Resolved: junit.hpi (org.jenkins-ci.plugins:junit:1.3)
  Resolved: symbol-annotation.jar (org.jenkins-ci:symbol-annotation:1.6)
  Resolved: annotation-indexer.jar (org.jenkins-ci:annotation-indexer:1.9)

----------------------------------------------------------------------------------------------------


BUILD SUCCESSFUL in 0s
1 actionable task: 1 executed

If you look at the first section, you see the various hpi dependencies. I want to have these added to my runtimeOnly dependency. I’d like the jars of these and their transitive jar dependencies to end up in my implementation configuration.

Configuration: 'mkobitTestNormal
  First level: org.jenkins-ci.plugins.workflow:workflow-basic-steps:2.6;default

  Resolved: workflow-basic-steps.hpi (org.jenkins-ci.plugins.workflow:workflow-basic-steps:2.6)
  Resolved: workflow-api.hpi (org.jenkins-ci.plugins.workflow:workflow-api:2.16)
  Resolved: workflow-step-api.hpi (org.jenkins-ci.plugins.workflow:workflow-step-api:2.12)
  Resolved: mailer.hpi (org.jenkins-ci.plugins:mailer:1.18)
  Resolved: structs.hpi (org.jenkins-ci.plugins:structs:1.6)
  Resolved: display-url-api.hpi (org.jenkins-ci.plugins:display-url-api:0.2)
  Resolved: junit.hpi (org.jenkins-ci.plugins:junit:1.3)
  Resolved: symbol-annotation.jar (org.jenkins-ci:symbol-annotation:1.6)
  Resolved: annotation-indexer.jar (org.jenkins-ci:annotation-indexer:1.9)

Any ideas, or does there need to be features added to the dependency management APIs for this to work?

mkobit · August 22, 2017, 12:15pm

I was thinking maybe that the dependency could take a Callable or the new Provider type but neither of those seem to be consumable.

mkobit · September 4, 2017, 10:58pm

The workaround I currently have is to create a detachedConfiguration, add dependencies to it and resolve it, and thne provide those as a Callable<FileCollection> since those seem to accept the Callable type.

It currently looks like this:

val callablePluginLibraries: Callable<FileCollection> = Callable {
  val pluginHpiJpiConfiguration = configurations.getByName(PLUGIN_HPI_JPI_CONFIGURATION)
  val pluginHpiJpiDependencies = pluginHpiJpiConfiguration.dependencies
  logger.debug { "Creating a detached configuration from configuration $PLUGIN_HPI_JPI_CONFIGURATION dependencies $pluginHpiJpiDependencies" }
  val hpiJpiDetached = configurations.detachedConfiguration(*pluginHpiJpiDependencies.toTypedArray())
      .resolvedConfiguration

  val hpiDependencies  = hpiJpiDetached.resolvedArtifacts.filter {
    it.extension in setOf("hpi", "jpi")
  }
  val jarDependencies = hpiJpiDetached.resolvedArtifacts.filter {
    it.extension == "jar"
  }.map {
    it.file
  }

  val hpiJars = configurations.detachedConfiguration(*hpiDependencies.map { dependencies.create("${it.moduleVersion}@jar") }.toTypedArray())

  hpiJars + project.files(jarDependencies)
}
dependencies.add(PLUGIN_LIBRARY_CONFIGURATION, project.files(callablePluginLibraries))

Lance · September 5, 2017, 5:56am

Take a look at the Project.files(…) documentation which states you can pass a Closure or a Task.

eg:

dependencies {
   runtime files({
      println 'unzip1' 
      zipTree(configurations.foo.singleFile).matching {
         include '**/*.xyz'
      }
   })
   runtime files(unzipTask)
}
task unzipTask(type: Copy) {
   destinationDir = "$buildDir/unzipTask"
   from zipTree(configurations.foo.singleFile).matching {
      include '**/*.xyz'
   }
   doLast {
      println 'unzip2' 
   } 
}

mkobit · October 13, 2017, 4:47pm

It took me a long time to get there, but I think I finally found a more satisfactory way to deal with this.

I tried using extendsFrom and beforeResolve on the “parent” configuration, but I did not realize the “superconfigurations” are not also resolved.

pluginDeclarations.incoming.afterResolve {
  pluginDeclarations.resolvedConfiguration.resolvedArtifacts
    .filter { it.extension in setOf("hpi", "jpi") }
    .map { "${it.moduleVersion}@${it.extension}" }
    .forEach { dependencies.add(pluginHpiAndJpi.name, it) }
  // Map each included HPI to that plugin's JAR for usage in compilation of tests
  pluginDeclarations.resolvedConfiguration.resolvedArtifacts
    .filter { it.extension in setOf("hpi", "jpi") }
    .map { "${it.moduleVersion}@jar" } // Use the published JAR libraries for each plugin
    .forEach { dependencies.add(pluginLibraries.name, it) }
  // Include all of the additional JAR dependencies from the transitive dependencies of the plugin
  pluginDeclarations.resolvedConfiguration.resolvedArtifacts
    .filter { it.extension == "jar" }
    .map { "${it.moduleVersion}@jar" } // TODO: might not need this
    .forEach { dependencies.add(pluginLibraries.name, it) }
}

configurations.forEach { config ->
  config.incoming.beforeResolve {
    if (config.hierarchy.contains(pluginHpiAndJpi) || config.hierarchy.contains(pluginLibraries)) {
      // Trigger the dependency seeding
      pluginDeclarations.resolve()
    }
  }
}

This doesn’t really allow for users to query those other dependencies to see what dependencies are in each configuration but it does improve how they are resolved and could allow for the other dependency management techniques to be used.