Is there a (sane) way to define a maven repository once and then reuse it multiple times?

Maven repositories are used for both producing and consuming artifacts. A standard Maven repository is configured with:

  • a URL
  • a username
  • a password (possibly lazily-computed and cached, for example for repositories hosted on AWS CodeArtifact)

In my multi-project Gradle build I cannot find a way to avoid configuration duplication. The following is the ideal solution:

-- Inside vendor-html-theme/build.gradle
publishing {
  repositories {
    mavenContoso()
  }
}

-- Inside app1/build.gradle
repositories {
  mavenContoso()
}

How do I achieve it? There are multiple solutions online (see this one) but don’t feel right, are not written in Groovy, and don’t work with both consuming and publishing artifacts.

More importantly, in the case of AWS CodeArtifact, the authentication token is not a static string but an expensive network call, so should only be retrieved if actually needed and cached for the rest of the build, but this is very difficult to achieve because the current API MavenArtifactRepository.credential accepts either:

  • a configuration Action that is immediately executed to store the username/password
  • a Class object that makes it impossible to reuse a fresh token among different subprojects in the same build

The code to reuse the credentials looks like this:

public class AwsCodeArtifactCredentials {

  private String cachedToken;

  @Override
  public String getUsername() {
     return "aws";
  }
  
  @Override
  public synchronized String getPassword() {
    if (cachedToken == null || cachedToken.isExpired()) {
      cachedToken = getFreshAccessTokenWithAwsSdk();
    }
    return cachedToken;
  }
}

Sample project is on GitHub.

Why does the solution by me not work?
As written there, it works for consumption in Groovy and Kotlin.
And it can as well be added to the RepositoryHandler for publication as it can be added to the other one.

That the code that actually adds the extension is written in Kotlin doesn’t matter much, you can do the same in any other JVM language, should be fairly easy to translate.


Besides that, I strongly recommend you switch to Kotlin DSL. By now it is the default DSL, you immediately get typesafe build scripts, actually helpful error messages if you mess up the syntax, and an amazingly better IDE support when using a good IDE like IntelliJ IDEA or Android Studio. :slight_smile:

There are a couple of issues beside the fact that I am simply not able to translate that Kotlin example in my Gradle/Groovy codebase. For example the class Foo in your example is abstract (why?! who is supposed to instantiate it?) and does not have any constructor dependency while MavenArtifactRepository has a lot; the method bar() neither returns anything nor manipulates memory so I cannot figure out what is supposed to do in my use case; finally I see no way to instantiate a single credentials cache object in the whole build and reuse the very same object to provide fresh or cached password to the repository configuration. Can you point me to a working (even Kotlin) example beside that one?

Sample project on GitHub

For example the class Foo in your example is abstract (why?! who is supposed to instantiate it?)

Gradle does.
Gradle saves you quite some boilerplate when you let it instantiate things like domain objects, extensions, plugins, tasks, and so on. It can automatically Property fields and similar, can inject various things in constructors or fields, and automatically makes every class implement ExtensionAware even if it does not technically declare it, which is the reason you can cast virtually anything to ExtensionAware and add extensions even if it doesn’t declare it, which is why I usually also declare it explicitly.

and does not have any constructor dependency while MavenArtifactRepository has a lot; the method bar() neither returns anything nor manipulates memory so I cannot figure out what is supposed to do in my use case;

The method bar() only prints one line to stdout.
In the thread you linked, the topic was only how to make the method callable conveniently by consumers from Groovy and Kotlin DSL in the exact intended scope, i.e. in repositories blocks where it should be available.
The asker in that thread already had a working method that adds the repository.
So that code by me just shows how you bring it to the right scope using the bar method.
Your real implementation of course will have to implement the repository adding, Kotlin is not that magical. :smiley:
A more complete example would for example be

abstract class Foo(val repositories: RepositoryHandler) {
    fun bar() {
        repositories.maven("https://foo.bar")
    }
}
(repositories as ExtensionAware).extensions.create<Foo>("foo", repositories)

finally I see no way to instantiate a single credentials cache object in the whole build and reuse the very same object to provide fresh or cached password to the repository configuration.

Having something build-scoped, you usually do using a shared build service.
So you would for example create a shared build service that creates an instance of your class and then everyone needing that one instance is getting it from that shared build service.

1 Like

And just for you here the Groovy version:

abstract class Foo {
    RepositoryHandler repositories
    Foo(RepositoryHandler repositories) {
        this.repositories = repositories
    }
    def bar() {
        repositories.maven {
            url = 'https://foo.bar'
        }
    }
}
repositories.extensions.create('foo', Foo, repositories)

First of all thank you! Your posts are a treasure trove of information. I was able to use a method mavenContoso() inside repositories {} with just one minor glitch (it’s called like contoso.mavenContoso() instead of just mavenContoso() because one needs to include the name of the extension - however this is just cosmetic) but two main problems remain:

  • Using Shared Build Services proved not to be as easy as advertised (I grasped the purpose but don’t know where to put things). Namely, I can’t find a way to connect the new build service to the configuration APIs exposed by MavenArtifactRepository
  • There’s no easy way to extend the RepositoryHandlers that are added by the maven-publish plugin to publishing {} blocks on a per-project basis

I think the easiest way to show how it’s done is opening a pull request in the linked GitHub project. If you have an AWS account is not difficult to make an end-to-end test because to create a repository one just needs to issue a couple of shell commands:

aws codeartifact create-domain --domain temporary
aws codeartifact create-repository --domain temporary --repository m2

# Then when you're done...
aws codeartifact delete-repository --domain temporary --repository m2
aws codeartifact delete-domain --domain temporary

Namely, I can’t find a way to connect the new build service to the configuration APIs exposed by MavenArtifactRepository

Just register the build service in your plugin where you also add the extension, then give the provider also to the constructor arguments, and when the method is called by your consumer, get the value from the build service.

There’s no easy way to extend the RepositoryHandlers that are added by the maven-publish plugin to publishing {} blocks on a per-project basis

Why not?
Should be exactly the same.

pluginManager.withPlugin("maven-publish") {
    val publishingRepositoryHandler = the<PublishingExtension>().repositories
    (publishingRepositoryHandler as ExtensionAware).extensions.create<Foo>("foo", publishingRepositoryHandler)
}

respectively

pluginManager.withPlugin("maven-publish") {
   publishing.repositories.extensions.create('foo', Foo, publishing.repositories)
}

If you have an AWS account

I do not

1 Like

Btw. allprojects { ... } and other means to do cross-project configuration is highly discouraged bad practice. :wink:

For some reason I do not need to mess with the plugin manager. I have working code but do not know why (link to GitHub commit). I followed your advice of using an abstract class and somehow Gradle injected all the RepositoryHandlers of my build…why? Do you have any documentation for this behavior?

The last bit would be using build services to cache the authorization token. The difficult thing here is somehow connect the repository configuration with the build service or any other pull-based credential supplier. Can you help me with the API of MavenArtifactRepository too?

For some reason I do not need to mess with the plugin manager.

Because you didn’t test properly what you did and didn’t follow properly what I said. :wink:

Gradle injected all the RepositoryHandler s of my build…why?

Because it can, that’s one of the beauties of using abstract classes, if you do it right.
In your case you cannot use it, it just works because you did not do what I said. :slight_smile:

Do you have any documentation for this behavior?

The injection per-se is documented at Understanding Services and Service Injection, though you can inject more than what is written there, like for example RepositoryHandler for extensions on a Project like you did in your try.

The problem with your current approach is, that you did not follow my instructions to create an extension on the RepositoryHandler instances where you want that method to be available, but instead added it to project.

This way you can call that method anywhere in your build script, not only within the repositories { ... } blocks, and at the same time you everywhere use the RepositoryHandler of the project, no matter where you call the method. So calling it in the publishing.repositories { ... } block will still just define the repository for resolving dependencies, not for publishing.

The difficult thing here is somehow connect the repository configuration with the build service

Why? Does it not work like I described it?

You are right, and sure I’m not following your instructions, but not because I’m here to waste your time, so maybe you could adapt your approach at giving help. I find this post on StackOverflow useful and tried to give you a minimal and reproducible example on GitHub:

because after all these posts it’s clear that you know way much more than me on this topic, but the official doc is incomplete/misleading, or simply too difficult for me to understand at my current level.

The problem with the provided snippets is that you don’t tell where the code is meant to be put, so I put it where I think it should go and suddenly I face the real-world problem of not having the objects you mention in scope, start to struggle to get a reference, and then I drift and the end result is totally different from what you meant.

Would you like to adapt your suggestions to the precise files in the linked repository, or even answer with a pull request that fixes everything? For example, regarding your last question, it’s not that it doesn’t work, I was not even able to write the code because type signatures do not match!

Thanks a lot

I’m sorry, but :spoon:s are out.
I gave you explicit instructions what to do.
You did not take what I said and just did it in the wrong place.
I said
repositories.extensions.create('foo', Foo, repositories)
and
publishing.repositories.extensions.create('foo', Foo, publishing.repositories)
you did
project.extensions.create('foo', Foo)

That is not just doing what I said in the wrong place, that is doing something totally different.
repositories and pluginManager are fields on project,
publishing is an extension on project if the maven-publish plugin is applied,
so just prefix add project. in front .

The problem with the provided snippets is that you don’t tell where the code is meant to be put, so I put it where I think it should go and suddenly I face the real-world problem of not having the objects you mention in scope, start to struggle to get a reference

Well, you could as well have just asked. :slight_smile:
Besides that the “where” depends, as you could do it in a build script, in a precompiled script plugin, in a classical binary plugin, …
But there is also JavaDoc available, which could have easily help to find out, that for example getPluginManager() and getRepositories() are on Project if you are not aware of that.

or even answer with a pull request that fixes everything?

You should google the term “help vampire”, because now you are leaning towards that behavior.
Is it not enough that I spend my sparse spare time to help you like I do?
Do you really think it is appropriate to ask me for doing your implementation for you?
I gave you concrete instructions how to solve your issue, if you struggle with that, it’s fine to ask further about the problems you hit.
Asking me to write your code for you is not really nice.
Providing the MCVE is great, and if I feel like helping by providing a PR, that’s also great, but you really shouldn’t ask “please write the code for me”.

1 Like

No, I’m not asking you write my code for me (btw, that on GitHub is a POC I wrote for you, not my real code, which works perfectly fine even with a little duplication). I simply want to agree upon another framework because the interaction so far has been unfruitful. For both of us.

Let me assure you (I help others just for the sake of it and my profile on StackOverflow can tell it) I know quite well how you feel and why it won’t work as long as we keep interacting this way. Do you think a pull request is ineffective? You don’t feel like? No problem. I still appreciated your help and learnt lot of things.

No, I’m not asking you write my code for me

Yes you do, even if it is only for a PoC.
I gave you exact instructions that you just would need to follow.

because the interaction so far has been unfruitful. For both of us.

Well, if you would have followed my instructions, it would have been more fruiful. :wink:

I know quite well how you feel and why it won’t work as long as we keep interacting this way.

Well, if you would just have changed then to doing what I said, it would have worked. :stuck_out_tongue:

Do you think a pull request is ineffective?

Not at all.
But giving you the information you need to do it yourself instead has multiple advantages. For example

  • you learn more if you manage to do it yourself instead of being :spoon:fed
  • it costs me some minutes to give you the instructions that you need to do it yourself
  • it costs me significantly more effort and time to write the code for you and I have plenty other stuff to do, including my day-job
  • others can benefit from the information here too, while the PoC will most probably be deleted and others cannot see the changes

Having said all that, lucky you I had some free minutes.
I’ll attach it as patch here instead of a PR because of the last point above.
You probably will need to polish it some more like adding a ReentrantReadWriteLock for accessing the token and adding the expiry check.
And it of course is greatly untested, as I neither have an AWS account, not aws utility.

diff --git a/buildSrc/src/main/groovy/contoso/ContosoAwsCredentialsService.groovy b/buildSrc/src/main/groovy/contoso/ContosoAwsCredentialsService.groovy
new file mode 100644
index 0000000..234e7cd
--- /dev/null
+++ b/buildSrc/src/main/groovy/contoso/ContosoAwsCredentialsService.groovy
@@ -0,0 +1,38 @@
+package contoso
+
+import org.gradle.api.provider.Property
+import org.gradle.api.services.BuildService
+import org.gradle.api.services.BuildServiceParameters
+import org.gradle.process.ExecOperations
+
+import javax.inject.Inject
+
+abstract class ContosoAwsCredentialsService implements BuildService<Parameters> {
+    @Inject
+    abstract ExecOperations getExec()
+    private String cachedToken
+
+    String getAccessToken() {
+        if ((cachedToken == null) || cachedAccessTokenExpired) {
+            cachedToken = freshAccessTokenWithAwsSdk
+        }
+        return cachedToken
+    }
+
+    private boolean isCachedAccessTokenExpired() {
+        return false
+    }
+
+    private String getFreshAccessTokenWithAwsSdk() {
+        ByteArrayOutputStream stdout = new ByteArrayOutputStream()
+        exec.exec {
+            it.commandLine('aws', 'codeartifact', 'get-authorization-token', '--domain', 'temporary', '--query', 'authorizationToken', '--output', 'text', '--profile', parameters.awsProfile)
+            it.standardOutput = stdout
+        }
+        return stdout.toString()
+    }
+
+    interface Parameters extends BuildServiceParameters {
+        Property<String> getAwsProfile()
+    }
+}
diff --git a/buildSrc/src/main/groovy/contoso/ContosoGradlePlugin.groovy b/buildSrc/src/main/groovy/contoso/ContosoGradlePlugin.groovy
index bcef72d..8c3cf1b 100644
--- a/buildSrc/src/main/groovy/contoso/ContosoGradlePlugin.groovy
+++ b/buildSrc/src/main/groovy/contoso/ContosoGradlePlugin.groovy
@@ -1,11 +1,25 @@
 package contoso;

 import org.gradle.api.Plugin;
-import org.gradle.api.Project;
+import org.gradle.api.Project
+import org.gradle.api.services.BuildServiceRegistry
+
+import javax.inject.Inject;
+
+abstract class ContosoGradlePlugin implements Plugin<Project> {
+    @Inject
+    abstract BuildServiceRegistry getSharedServices()

-class ContosoGradlePlugin implements Plugin<Project> {
     @Override
     void apply(Project project) {
-        project.getExtensions().create("contoso", ContosoMavenExtension)
+        def contosoAwsCredentials =  sharedServices.registerIfAbsent("contosoAwsCredentials", ContosoAwsCredentialsService) {
+            parameters {
+                awsProfile = project.property('contoso.aws.profile')
+            }
+        }
+        project.repositories.extensions.create("contoso", ContosoMavenExtension, project.repositories, project, contosoAwsCredentials)
+        project.pluginManager.withPlugin("maven-publish") {
+            project.publishing.repositories.extensions.create("contoso", ContosoMavenExtension, project.publishing.repositories, project, contosoAwsCredentials)
+        }
     }
-}
\ No newline at end of file
+}
diff --git a/buildSrc/src/main/groovy/contoso/ContosoMavenExtension.groovy b/buildSrc/src/main/groovy/contoso/ContosoMavenExtension.groovy
index 56e4a2d..0a5b170 100644
--- a/buildSrc/src/main/groovy/contoso/ContosoMavenExtension.groovy
+++ b/buildSrc/src/main/groovy/contoso/ContosoMavenExtension.groovy
@@ -2,25 +2,28 @@ package contoso

 import org.gradle.api.Project
 import org.gradle.api.artifacts.dsl.RepositoryHandler
+import org.gradle.api.provider.Provider

 abstract class ContosoMavenExtension {

     private final RepositoryHandler repositories
     private final Project project
+    private final Provider<ContosoAwsCredentialsService> contosoAwsCredentials

-    ContosoMavenExtension(RepositoryHandler repositories, Project project) {
+    ContosoMavenExtension(RepositoryHandler repositories, Project project, Provider<ContosoAwsCredentialsService> contosoAwsCredentials) {
         this.repositories = repositories
         this.project = project
+        this.contosoAwsCredentials = contosoAwsCredentials
     }

     def codeArtifact() {
-        def endpoint = project.property("contoso.m2.endpoint")
-        def awsProfile = project.property('contoso.aws.profile')
+        def contosoAwsCredentials = contosoAwsCredentials
+        def endpoint = project.property('contoso.m2.endpoint')
         repositories.maven {
             url endpoint
             credentials {
                 username = 'aws'
-                password = "aws codeartifact get-authorization-token --domain temporary --query authorizationToken --output text --profile ${awsProfile}".execute().getText()
+                password = contosoAwsCredentials.get().accessToken
             }
         }
     }

Thanks! It worked like a charm :hearts:

One minor point remains, but I think it’s a limitation of the current Gradle API and there’s really nothing we can do. The CodeArtifact authorization token is not static text, but a string returned by a network call. With the current implementation we request a new token every time Gradle is run, even when Gradle doesn’t need to contact the remote Maven repository, for example because dependencies are already resolved and cached or we are in offline mode (well, technically the extension could check if we are in offline mode, but it’s more a workaround than a solution).

As far as you know, is there a way to make the MavenArtifactRepository (link to Javadoc) pull the password when it needs to authenticate instead of requiring one at configuration time? I think no API is suitable for this, because we can only pass a String or a Class object in a push style.

Afair no.
You either need to give the credentials, or configure one of the built-in hard-coded types.
You cannot even use a custom subclass or similar.

The question is, whether you cannot use the built-in AWS credentials class.

The builint AwsCredentials only accepts:

  • access key + secret access key, which should not even be used on development machines because they are hardly rotated; anyway one tipically has many keys on a given machine, and selects one by a profile and this selection is not supported (sure, one can mess with the password file or duplicate the keys and store it in ~/.gradle/gradle.properties but it’s insecure and cumbersome)
  • session token, which is short lived and must be rotated manually every 20 minutes or so

Also the source code seems to confirm that it’s a known limitation:

@Nullable
@ToBeReplacedByLazyProperty
String getSessionToken();

The very same @ToBeReplacedByLazyProperty annotates the fields in PasswordCredentials too.

I don’t think so.
This probably means the Property migration happening for Gradle 9 instead of using just String.