Custom Gradle Build System for Handling Android Submodules as Resources

I need assistance with setting up a specialized Gradle build system. The goal is to build Android submodules and include their processed artifacts as resources in another submodule. Specifically, the submodules use the Android plugin and produce APK files, which I need to be processed and included as DEX files in a JAR artifact produced by the main module.

Example Project Structure

myProject
|-- build.gradle
|-- jarModule
|   |-- build.gradle
|   |-- src
|       |-- main
|           |-- java
|           |-- resources
|               |-- someResourceDirectory
|                   |-- androidModule
|                       |-- build.gradle
|                       |-- src
|                           |-- main
|                               |-- java
|                               |-- res

Desired Output

When I build the project, the final JAR should look like this:

artifact.jar
|-- other
|-- someResourceDirectory
|   |-- artifact.dex

Requirements

  1. Automatic Registration and Build of Submodules:

    • Gradle should automatically register and build submodules under any directory within the resources directory when the main project is built.
  2. APK Processing to DEX:

    • After building the submodules, their APK artifacts should be processed to extract the DEX file.
    • The DEX file should then replace the original submodule in the same resource directory of the JAR.

Main Challenges

  1. Configuring Gradle to automatically detect and build Android submodules under the resources directory.
  2. Processing the APK artifacts to extract the DEX file and include it as a resource in the final JAR artifact.

Real-world usage

To further explain why I require this specialized build system, here’s my current scenario:

I am developing a patch plugin system designed to modify APK files. My patcher library allows applications to supply an APK file along with patch classes, which the patcher uses to modify the APK. Developers set up a Gradle project, add the patcher library as a dependency, write their patch classes, and then build the project to produce a patches.jar. This patches.jar is then used by the patcher to apply modifications to the supplied APK files.

Extended Functionality

I want to enhance this system by allowing not only patching of APKs but also extending them with new classes. The idea is to precompile these new classes using an Android Gradle project, producing an APK with a DEX file. The DEX file is then supplied to the patcher, which merges the new classes into the target APK, enabling the addition of complex business logic. The patches will modify the APK to call methods from these newly added classes.

Modular System Design

Given the modular nature of this system, each patch should have the capability to include its own DEX file if needed. The process should be as follows:

  1. Development: Create a new patch in a Gradle project and add an Android submodule.
  2. Build: When building the patches.jar, the build system should compile the new patch, extract the DEX file from the Android submodule’s APK, and package it into the patches.jar.

Automation Requirement

This system needs to automatically detect and process Android submodules, compiling them into DEX files and adding them to the resulting patches.jar. This automation is crucial because manual adjustments to the build system for each new patch would be inefficient and error-prone, especially as the number of patches grows.

Summary of Requirements

  1. Automatic Registration and Build of Submodules:

    • Detect and build Android submodules located within the resources directory when the main project is built.
  2. APK Processing to DEX:

    • Extract DEX files from the built APKs of these submodules.
    • Include these DEX files as resources in the final patches.jar.

Here is a real example:

MyPatch.kt:

val myPatch = patch(name="myPatch", dexPath="dex/my-patch.dex") {
 execute {
   // ...
 }
}

Under the resource directory dex/ I would initialize my Android project:

myPatches
|-- build.gradle
|-- patchesModule
|   |-- build.gradle
|   |-- src
|       |-- main
|           |-- kotlin
|				|-- MyPatch.kt
|           |-- resources
|               |-- dex
|                   |-- myPatchAndroidModule
|                       |-- build.gradle
|                       |-- src
|                           |-- main
|                               |-- java
|									|--MyPatch.java
|                               |-- res

The resulting ´patches.jar` looks like this:

patches.jar
|-- MyPatch.kt
|-- dex
|   |-- my-patch.dex

Project links

I am open to different designs. Putting projects inside the resources directory for example may be odd.

Thank you if you made it this far!

I am open to different designs. Putting projects inside the resources directory for example may be odd.

Indeed. What is in src/main/resources is put to the jar usually, maybe processed a bit by processResources. So you would need to then exclude those and everyone seeing it at first might be confused.

I’d put it somewhere else. src/main/patches/dex/myPatchAndroidModule, or src/patches/dex/myPatchAndroidModule or patches/dex/myPatchAndroidModule or any other place.

Automatically including those projects is easy. Just scan in the settings script for those projects and include them as they are found.

Regarding the second point, I think an artifact transform would be most suitable. Used right you then can for example just depend on all projects in the build and the artifact transform will deliver the dex file for those projects. The dependencies will be for a custom dependency scope configuration, and you will have another resolvable configuration that extends it, declares the attributes so that the artifact transform will fire and then will have this configuration (or maybe its elements property) as srcDir for sourceSets.main.resources.

1 Like

Thanks for your reply. Wouldn’t this location be treated as source since it’s under src and fail compilation?

The build task seems to generate a dex file under build/intermediates/dex. I’ll probably retrieve the dex that way instead of extracting it from the APK artifact.

Right now I am still struggling with a proper project structure. Creating an Android project every time is quite the boilerplate. Perhaps a plugin could reduce it.

Thanks for your reply. Wouldn’t this location be treated as source since it’s under src and fail compilation?

Why should it?
As long as you do not configure src/main/patches to be a source directory, or apply some plugin that does it, it is just a regular directory, nothing special.

The build task seems to generate a dex file under build/intermediates/dex

For sure not the build task, as it is a lifecycle task without actions unless someone raped it.
But some task it triggers could well be.
So if that is the dex you need, even better, then you do not need to build the full apk.
Find out which task builds the dex and then do not directly depend on it or get the file by path, but have a look at Sharing outputs between projects which explains how you safely hand-over artifacts from one project to another.

1 Like

Thanks for the reply.

I can see the task minifyReleaseWithR8 when building the Android project. Checking its output showed some files, including a folder “minifyReleaseWithR8” which I assume to be the folder that contains the dex file build\intermediates\dex\release\minifyReleaseWithR8\classes.dex.

Unfortunately, the page you sent is going way deeper into Gradle than I am experienced with.
It says:

In order to be safe to share between projects and allow maximum performance (parallelism), such artifacts must be exposed via outgoing configurations .

However I do not have any control over the Android Gradle plugin task to set that up.

Currently I am desperately trying my ways with his:

 register<Copy>("copyExtensions") {
        file("../extensions").list()?.map { extension ->
            project(":extensions:$extension")
        }?.forEach { extensionProject ->
            val extensionResourceName: String by extensionProject.ext
            val extensionResourcePath = Path.of(extensionResourceName)

            val extensionDexPath = extensionProject.layout.buildDirectory.get()
                .dir("intermediates/dex/release/minifyReleaseWithR8")

            from(extensionDexPath) {
                include("classes.dex")
            }
            into(extensionResourcePath.parent.pathString)
            rename { extensionResourcePath.name }
        }
    }

However I do not have any control over the Android Gradle plugin task to set that up.

Of course you have.
Just do it.
You can configure whatever you want.
And actually you do not even need to change the configuration of that task in this case, but just add it as artifact to an outgoing configuration as described on that page I linked you to.

Currently I am desperately trying my ways with his:

Which is exactly what you should not do as documented on that page I linked you to.
It is flaky at best and unsafe.
You really should not do it that way, but do it properly as documented on that page.

1 Like

I added this to the producer:

And this to the consumer:

As you can see the project doesn’t build (even with Gradle build).
But even if it did, how would I get the artifact to process as a resource? The documentation does not explain how to actually get the artifact after that step I did.

Did you actually create the configurations?
Consumable on the producer side, and dependency scope on the consumer side.

And then I already told you in the first response how to continue.
A resolvable configuration that extends from that dependency scope configuration where you declare the dependency on.
And then, use it as srcDir for the resources of the main source set.

val extensionDex by configurations.dependencyScope("extensionDex")
val extensionDexFiles = configurations.resolvable("extensionDexFiles") {
    extendsFrom(extensionDex)
}
sourceSets.main {
    resources.srcDir(extensionDexFiles)
}
1 Like

Did you actually create the configurations?
Consumable on the producer side, and dependency scope on the consumer side.

I have followed what the documentation said. The documentation said to add this to the producer:

And it said to add this to the consumer:

The documentation you linked seems to add the block to the dependencies which is not what I need. I do not need to depend on any artifact, I need to process the artifact as a resource.

Your suggestion looks to be marked as unstable:

is there any way to not rely on any unstable APIs?

Unfortunately I am barely understanding what a resolvable configuration is. From what I just read on the Gradle documentation, it is primarily used for dependencies. Like mentioned earlier, I do not need any dependencies. Instead I want to process the artifacts as a resource.

In addition to that the documentation from earlier mentions to add this property to the producer:

But why am I declaring a property, if the variable is unused?

I have removed the block from the consumer dependency block as that didn’t seem right to me, despite the docs you linked telling me to add them. Instead I added your block to the consumer. When building the project no artifacts are added to the resources of the final built JAR.

Because my breakpoints are actually breaking my IDE I figured I’ll debug by throwing an exception:

As you can see, the exception message is null, therefore no artifacts were added to the resources.

Also, why am I not using the processResources block and instead this?

sourceSets.main {
    resources.srcDir(extensionDexFiles)
}

I have not defined extensionDexFiles anywhere and the artifact I am trying to add is a single file, not a folder but the inline docs of resources.srcDir mention:

image

But instead I need to add a file not a directory.

It is likely we talk past each other with my lack of expertise in Gradle.

And it said to add this to the consumer:

It said to do it within the dependencies { ... } block. If you do it outside, you are not following the documentation properly.

Unfortunately, that page is expecting some transfer knowledge abilities and does not show that you of course also have to create the configuration on the consumer side, but as you can only declare dependencies on configurations and not on thin air, you have to do it as I asked about above. Feel free to open an improvement issue for the Gradle guys on GitHub to add a note or example.

The documentation you linked seems to add the block to the dependencies which is not what I need.

Yes, it is.

I do not need to depend on any artifact

Yes you do. You might just have a different understanding of the meaning of “depending” than Gradle.

I need to process the artifact as a resource.

And thus depend on the artifact in the consumer project.
Depending on something does not imply it is a jar or that you need in some classpath.
Depending on something with the implementation configuration or any other configuration that ends up on a classpath does.
But that’s why you need to create your own configuration to declare the dependency on, so that you can do with it whatever you need, like adding it to your resources.

Your suggestion looks to be marked as unstable:
is there any way to not rely on any unstable APIs?

Yeah, they are new handy convenience methods that you can use and I don’t really expect them to change any further.
But if you prefer, just create the configuration traditionally and set the properties accordingly.
For dependencyScope that would be

val extensionDex by configurations.creating {
    isCanBeResolved = false
    isCanBeConsumed = false
    isVisible = false
}

for resolvable that would be

val extensionDexFiles by configurations.creating {
    isCanBeResolved = true
    isCanBeConsumed = false
    isVisible = false
    extendsFrom(extensionDex)
}

Unfortunately I am barely understanding what a resolvable configuration is.

Just what the name says, a configuration that you can resolve, that is get files out of it.
There are consumable configurations that other projects or builds can request directly or via feature variants.
There are dependencyScope configurations on which you just declare dependencies.
There are resolvable configurations from which you request the artifacts after dependency resolution happened.
And there are legacy configurations that have multiple such roles which should not be used and will be removed in the future. These are what you get if you don’t configure the isCanBe... properties due to backwards compatibility.

But why am I declaring a property, if the variable is unused?

It is used or IntelliJ would color it grey, not purple.
The name is used (through the “by” and thus property delegation of Kotlin) to create a configuration with that name.
That is just convenient and more type-safe syntax sugar provided through Kotlin.
You can also instead do

configurations.create("extensionDex") { ... }

but I would instead use extensionDex.name instead of "extensionDex" in the artifacts.add call, then it also is more obviously used and safer.

I have removed the block from the consumer dependency block as that didn’t seem right to me, despite the docs you linked telling me to add them. Instead I added your block to the consumer. When building the project no artifacts are added to the resources of the final built JAR.

Of course, if you remove the dependency declaration, there is nothing to add.
My snippet was just the missing pieces to show how you should create the configurations and how to add it to the resources.

Because my breakpoints are actually breaking my IDE I figured I’ll debug by throwing an exception:

Why should a breakpoint break the IDE? Debugging build scripts works for me like a charm since years.
If something breaks by a breakpoint there might be something else f***ed up.

As you can see, the exception message is null, therefore no artifacts were added to the resources.

Of course, you removed the part that adds the dependency.

Also, why am I not using the processResources block and instead this?

Because with what I gave you, you declare those files as resources, so any task that needs resources gets them and also has the necessary task dependencies automatically, including the processResources task. Configuring it for the processResources task directly just adds it for that one task which is usually not desirable.

I have not defined extensionDexFiles anywhere

You said you added the snippet I gave, so there you defined it.

and the artifact I am trying to add is a single file, not a folder but the inline docs of resources.srcDir mention:

The artifact might be a single file, but the configuration is a collection of files. And if you have multiple such subprojects you depend on, all those single files are part of that collection. The collection “is” the source directory containing those files. So yes, using srcDir is exactly what you need there and thus also get the necessary task dependencies for everything consuming the resources.

1 Like

Yes, that’s what I did. Sorry it did not look like in my image I took.

I would like to do that, but with my lack of experience, I could not contribute anything other than “I barely understand the page”. Therefore I first need to learn more about this. If possible, I’ll try to contribute.

You set the properties to false. Why do I need to explicitly set those properties? Are they not false by default but instead null? Is a custom setter involved that needs to be called? What issue would it oppose if it were consumable, resolvable or visible? What is each properties purpose in the sense of a configuration? Why is “resolving” a concept or what purpose does it have? What does “resolving” actually do? Resolving is a very broad and generic term, it does not further explain what really happens.
Whats the use of a configuration that can not be consumed or which is invisible? What is the reason there is a property for visibility and consumability? If it is not visible, it implies I can’t consume it so why do I need to explicitly set isCanBeConsumable to false, and why does the property exist if isVisible exists?

You mention For dependencyScope that would be but what is meant by dependencyScope? You show a code snipped for a configuration. I thought its a configuration for the current Gradle project, not a dependency. What is a scope of a dependency or why are dependencies scoped? Same questions for resolvable.

You create a extensionDexFiles property but I never have any more than one dex file per Android project. Why is it called “extensionDexFiles”? What does it mean to extend from a configuration? What purpose does it have to make the extensionDex configuration not resolvable, but the extensionDexFiles configuration resolvable?

On a side note, why is the function called creating instead of create or new? Why was no regular constructor used? Is this a convention in Gradle? In addition to that this API leverages delegated properties to create a name for the configuration, but the following API doesn’t, which is unusually inconsistent:

Just what the name says, a configuration that you can resolve, that is get files out of it.

But why is this something exposed to the user? Why does it not resolve for me and instead I have to resolve it myself? What is the purpose of exposing this functionality to the user?

There are consumable configurations that other projects or builds can request directly or via feature variants.

What does it mean that a “build requests a configuration”? For me, a “build” is a process that can be executed in a project. Why does a “process” request a configuration and not the project that executes the process?

There are dependencyScope configurations on which you just declare dependencies.

What does it mean to declare a dependency on a dependency scope? Is a dependency scope a collection of dependencies? Why can you depend on a collection of dependencies instead of just the dependencies directly?

There are resolvable configurations from which you request the artifacts after dependency resolution happened.

Why do I request the artifacts after a dependency resolution? What is the reason for this ordering or why does requesting an artifact require first resolving dependencies?

It is used or IntelliJ would color it grey, not purple.

creating is a function that is being called. Function calls in themselves are not highlighted as unused even if their return value is unused. For that reason, IntelliJ should not color it grey (unused).

In addition to that the declared property is not used as visible in my previous screenshot as seen by the tooltip IntelliJ showed. Here is another example of a property I just declared that is not grey and (trust me) not used anywhere:

image

The name is used (through the “by” and thus property delegation of Kotlin) to create a configuration with that name.

Unfortunately this is missuse of property delegates. I strongly believe that Gradle added this for the purpose of later on using the property. If you do not intend to further use the delegated property, there is no reason to declare one. Instead, like you suggested the create function should be used, which I think exists exactly for the situation when you create a configuration and do not need it further anymore in your project.
Because the property is unused, I do not think it is correct practice to declare the property for that reason and instead the create function should have been used.
A question in extension to the one I already asked above in this regard is, why the function is called create instead of creating? Is it because it does not return a delegatable?

but I would instead use extensionDex.name instead of "extensionDex" in the artifacts.add call, then it also is more obviously used and safer.

Given that now the property extensionDex is used this way, this also justifies the declaration of the property.

Of course, if you remove the dependency declaration, there is nothing to add.

Add what to what? The configuration to the resources?

My snippet was just the missing pieces to show how you should create the configurations and how to add it to the resources.

Why should a configuration be added to the resources of the JAR? I want the dex to be added to the JAR not a “configuration”.

Why should a breakpoint break the IDE? Debugging build scripts works for me like a charm since years.
If something breaks by a breakpoint there might be something else f***ed up.

I would send a recording but the issue is sporadic and I am not reproducing it now. If I do, I’ll send a recording right away. The issue is comparable to when you try to debug a dependency of which your sources are outdated. The IDE thinks you set a breakpoint somewhere else than you actually did. In my case it jumped to completely different Gradle files.

Because with what I gave you, you declare those files as resources

I not only want to just add them, I want to rename them as well. So in a zip file system they should appear in a subdirectory under different names.

You said you added the snippet I gave, so there you defined it.

Your snipped mentioned extensionDexFiles but like I said earlier, I only have one dex per Android project, so this multiplicity doesn’t make sense semantically. When I inspect the type of that property, it returns me a list though even though I just need a single artifact.

but the configuration is a collection of files

You suggested this block:

val extensionDex by configurations.creating {
    isCanBeResolved = false
    isCanBeConsumed = false
    isVisible = false
}

If a configuration is a collection of files, why is the property singular. Further on you suggest the following block:

val extensionDexFiles by configurations.creating {
    isCanBeResolved = true
    isCanBeConsumed = false
    isVisible = false
    extendsFrom(extensionDex)
}

But this property is now plural. The property is a configuration and not a list of files so the name extensionDexFiles or extensionDex is deceiving in addition to that.

I inherited this deceiving naming from the official docs:

And also, the docs use plural sometimes only. Here they use singular once again:

So which is it now?

I have now added back everything you mentioned to the consumer:

The producer has this:

When I run the build task I get this exception:

I am not sure if I am missing something because I was not getting this exception previously.

Thanks for making it through til here! I am learning with every of your responses, bit by bit.

Yes, that’s what I did. Sorry it did not look like in my image I took.

Ah, ok then.
Then you just did forget to create the configuration.
Also the text seemed to suggest you added it outside, not inside. :slight_smile:

I would like to do that, but with my lack of experience, I could not contribute anything other than “I barely understand the page”. Therefore I first need to learn more about this. If possible, I’ll try to contribute.

I said “issue”, for that it is fine to just say “this docs are unclear”. But a PR that improves the docs would of course even be nicer, but not what I meant. :slight_smile:

Are they not false by default but instead null?

They are boolean, they cannot be null.
They are all true by default and thus create a legacy configuration that shouldn’t be used.
As I already said earlier, this is due to backwards compatibility.

What issue would it oppose if it were consumable, resolvable or visible? What is each properties purpose in the sense of a configuration?

I just explained it in detail in my last answer.
If you have further questions, maybe read it again.
And if you then have further questions, feel free to ask them, but please bit more targetedly.
With the question you just asked, I could just copy & paste what I wrote above.

Why is “resolving” a concept or what purpose does it have?

I don’t understand the question.
Why should it not be a concept?
And the purpose is, to resolve the dependencies and their transitive dependencies, and resolve version conflicts, and so on.

Resolving is a very broad and generic term, it does not further explain what really happens.

The term on its own maybe not, but in the context it is pretty focussed and clearly defined.
There is even a whole page in the user manual for the topic: Understanding dependency resolution.

Whats the use of a configuration that can not be consumed or which is invisible? What is the reason there is a property for visibility and consumability? If it is not visible, it implies I can’t consume it so why do I need to explicitly set isCanBeConsumable to false, and why does the property exist if isVisible exists?

Again, I already explained the different roles configurations can have and what they are for in my last post.
And “invisible” you always want.
It is a badly named flag of a legacy mechanism that shouldn’t be used anymore anyway and also is just true by default for backwards compatibility.
Thus setting the configuration to “invisible” does not mean you do not see it or cannot consume it and it does not have to be “visible” for being able to consume it.

You mention For dependencyScope that would be but what is meant by dependencyScope? You show a code snipped for a configuration. I thought its a configuration for the current Gradle project, not a dependency. What is a scope of a dependency or why are dependencies scoped? Same questions for resolvable .

Again, I explained the different roles a configuration can have above and what they are for, so what exactly is unclear? Why do you think a dependency has a scope I did not say anything implying that. What you talk about is a configuration with role “dependencyScope”, which is the name of the new incubating convenience methods you don’t want to use. It means it is neither resolvable, nor consumable, but declarable, meaning you declare dependencies for it. Configurations like implementation or runtimeOnly or compileOnly or api are “dependency scope” as those are the configurations where you declare dependencies on, not the ones other projects consume nor the ones you use to resolve for example a full classpath from those definitions.

You create a extensionDexFiles property but I never have any more than one dex file per Android project. Why is it called “extensionDexFiles ”?

You can also name it “honeypot”, or “theFlash”, the name doesn’t matter, name it however you like it. But even if you only have one dex file per project, as soon as you depend on two projects, this configuration contains the dex files of both projects, hence the ...Files.

On a side note, why is the function called creating instead of create or new ?

new is a keyword, you cannot name a function that easily.
There is another one called create that you give a name by string as I also showed you above.
But for the property delegation where the name is taken from the variable name by creating reads much better than by create. That’s at least what I guess why they named it that. If you want a definite answer, ask the one who invented it. I’m just a user like you.

Why was no regular constructor used? Is this a convention in Gradle?

Because you now simply create an instance, you add an instance to the configurations container and yes, it’s always like that with domain object containers in Gradle. In built-in ones, as well as in well designed 3rd party plugins.

Also, most domain objects you could not or at least should not construct yourself ever, but always let Gradle do it, because you can save quite some boilerplate by Gradle magic that implements various methods and injects various things for you and also makes the classes automatically implement ExtensionAware, which is why you can practically everywhere add extensions or set extra properties (even if you shouldn’t do the latter in most cases). But these are topics you should not care about too much unless you develop custom plugins.

In addition to that this API leverages delegated properties to create a name for the configuration, but the following API doesn’t, which is unusually inconsistent:

I fully agree and are pretty unlucky about their reaction: Kotlin DSL property delegates for fixed-type configuration factory methods · Issue #27204 · gradle/gradle · GitHub, but :man_shrugging:.

But why is this something exposed to the user? Why does it not resolve for me and instead I have to resolve it myself? What is the purpose of exposing this functionality to the user?

I don’t understand the question, you do not resolve anything yourself.
You create a configuration whose role is to be resolved and when you tell it “get me the files” it does its work.
Without “exposing this functionality to the user” you could not get any files from the configuration and they were useless at all, so I really don’t get the question.

What does it mean that a “build requests a configuration”?

That was maybe expressed a bit misleading. What I meant was “projects in the same build” as well as “projects in other builds”, the latter for example if you published the configuration as feature variant or use composite builds to combine multiple builds into one.

For me, a “build” is a process that can be executed in a project.

No, that’s wrong, that is “the execution of a build”.
A “build” is the entirety of the projects that you define / include in your settings script.

What does it mean to declare a dependency on a dependency scope? Is a dependency scope a collection of dependencies? Why can you depend on a collection of dependencies instead of just the dependencies directly?

Again, you do not depend on a “dependency scope”, you have a configuration with role “dependency scope” on which you declare dependencies like implementation("commons-io:commons-io:+") where implementation is the confiugration with role “dependency scope”. So yes, you could say a dependency scope is a collection of dependencies.

Why do I request the artifacts after a dependency resolution? What is the reason for this ordering or why does requesting an artifact require first resolving dependencies?

I start to think that you intentinally try to misinterpret every word I say. :-/
When you request the files of a resolvable configuration, resolution happens and after it finished you get the resulting files.
The reason for the ordering or why you cannot get an artifact without resolving first is obvious isn’t it?
You also cannot eat the meal before you have cooked.

creating is a function that is being called. Function calls in themselves are not highlighted as unused even if their return value is unused. For that reason, IntelliJ should not color it grey (unused).

We are not talking about creating, we are talking about extensionDex. Nowhere I said creating would be colored grey.

In addition to that the declared property is not used as visible in my previous screenshot as seen by the tooltip IntelliJ showed.

I start to think you didn’t properly read my answers.
Also that I already explained.
It is used and IntelliJ knows that.
by creating is using it.
This is just not a usage you get listed by “find usages”.
Complain to JetBrains if you think it should be listed.
image

Here is another example of a property I just declared that is not grey and (trust me) not used anywhere:

Well, seems there is a bug in the Kotlin IntelliJ Plugin then, report it.
On non-top-level it works as expected:
image

Unfortunately this is missuse of property delegates.

No, it is not.
You might consider it a misuse, but the Kotlin guys seem to not agree, as it works fine and also is not marked as unsued.
If you don’t want to use it like that, noone forces you to do so, you can always use the string-y methods.
I prefer using the delegates, then when I need them later, I already have them available and do not need to change from create to creating or have the name duplicated otherwise.

which I think exists exactly for the situation when you create a configuration and do not need it further anymore in your project.

You think wrongly. :wink:
It exists, because there is more than Kotlin out there.
create exists longer than Kotlin even exists.
And there is also Groovy DSL, and also any other JVM language you could use to create configurations.,
creating is just a syntactic sugar that you can use in Kotlin code or Kotlin DSL if you want, but noone forces you to use the imho nicer-looking syntax.

Because the property is unused, I do not think it is correct practice to declare the property for that reason and instead the create function should have been used.

Well, you need to accept that we disagree in this point.
Use whatever makes you happy.
Besides that, all the configurations are used further.
extensionDex on the consumer side is used in the dependencies block for declaring dependencies and in the extensionDexFiles configuration to extend from it. extensionDexFiles is used as srcDir for the resources source directory set, extensionDex on the producer side should be used for the artifacts.add call imho.

Is it because it does not return a delegatable?

It has a different behavior, why should it be called the same?
But well, ask the one that invented it.
But as I said, it most probably has nothing to do with whether it returns a delegatable or not, because when it was added, not even Kotlin existed.
And for the delegated property it most probably simply reads better.

Of course, if you remove the dependency declaration, there is nothing to add.

Add what to what? The configuration to the resources?

Why should a configuration be added to the resources of the JAR? I want the dex to be added to the JAR not a “configuration”.

Again, do you do this intentionally?
You declare the configuration as srcDir for the resources source directory set of the main source set, and by doing so you add the contents of the resolution result for the configuration which are your dex files to the resources which are than packed to your jar.

In my case it jumped to completely different Gradle files.

A right, that’s an unfortunate effect of Gradle or the Kotlin compiler when compiling the build scripts only putting the name of the build script into it, not the full path. And if you then have multiple projects in your build and all their build scripts are called build.gradle.kts this happens.

I almost forgot about this, because due to various reasons (this issue being one of them) I at most call the root project build script build.gradle.kts if at all. All other build scripts I usually name after the project name, so for example foo/foo.gradle.kts. You can configure that in the settings script. It also makes it much easier to open a build script of a specific project if you have many projects. If you e.g. have a multi-project build with 100 projects and tell the IDE to open build.gradle.kts, that’s not a nice experience.

I not only want to just add them, I want to rename them as well. So in a zip file system they should appear in a subdirectory under different names.

You could control this for example on the producer side.
Or you could use an artifact transform to move the files into a subfolder on-the-fly, then it is the same for all resource consumers.
If you only do it on the processResources task, again it is only for that task and things that use its output, but not for things that use the resources or “allSource” on the source set.

If a configuration is a collection of files, why is the property singular.

That configuration has role “dependency scope”.
While technically it is a collection of files, semantically it is not.
It is a scope on which you declare dependencies.
An in that context the naming makes sense.
But as I said already, you can also name it supercalifragilisticexpialidocious
It is just a name, and naming is one of the hardest parts in software developement.
Just name it whatever you like, if you don’t like the conventions Gradle folks and me are following.

But this property is now plural. The property is a configuration and not a list of files

This is a configuration with role “resolvable” and thus it technically and now also semantically is a collection of files, the result of resolving the dependencies declared on it directly (which should not be done) or implicitly through the extensions it extends. If you only add one dependency to one Android subproject, it will still be a collection with only one file in it. If you do not declare any dependency it will be a collection with no entries. If you declare multiple dependencies, there will be multiple files in the result.

I inherited this deceiving naming from the official docs:

It is not at all deceiving, stop saying that.
It is absolutely appropriate and fitting perfectly semantically.
But again, name the configuration harryJamesPotter if you prefer, or maybe anastasiaRoseSteele, doesn’t matter, whatever makes you happy.

And also, the docs use plural sometimes only. Here they use singular once again:

Not really if you think about it. A “classpath” is a collection of files (and directories) and thus the plural of containing multiple files is merged into the name being a collection itself.

If you feel better with that style, use extensionDexDirectory (even if technically not fully correct it would then be semanitically correct, matching how you use it if you declare it as srcDir, and a singluar word), or extensionDexCollection.

I am not sure if I am missing something because I was not getting this exception previously.

That’s a Kotlin script quirk.
It allows forward reference.
You use extensionDex in line 22.
It is defined and initialized in line 32.
So when you use it in line 22 it works as Kotlin script allows such forward reference which would work if the code for example is in a lambda that is evaluated after line 32 got evaluated.
But in your case it is evaluated immediately and thus has a null value which is not ok as it has a non-nullable type.

This is like having a non-nullable lateinit var and using it before assigning a value.
Move line 32 up above the dependencies { ... } block and the error should be gone.

Thanks for making it through til here! I am learning with every of your responses, bit by bit.

:ok_hand:

1 Like

I start to think that you intentinally try to misinterpret every word I say

Before I continue, please note that the volume of information here is large, and it takes quite a bit to keep track of all the terminology and concepts. For that reason, I will not understand everything you say directly. I have to assume, interpret, and make conclusions based on incomplete information or past information I have either not fully or incorrectly understood. I do not try to intentionally misinterpret every word you say. Sometimes when you say something, I have multiple ideas of what it could mean which is why I will ask you about what you said word by word to make sure I did understand you correctly, especially when I assume that it’s information I would have to rely on later on. Bear with me and thanks for understanding and helping me!

Back to the topic:

I have managed to share the artifact. Here is what I did:

My producer has this:

My consumer has this:

Can you recommend me better names than I currently have for the configurations or are those correct already?

Also, the resulting artifact is now added as a resource now:

Now, since I am working with multiple producers that produce the same exact artifact name, I need to move them to subdirectories. You mentioned:

You could control this for example on the producer side.
Or you could use an artifact transform to move the files into a subfolder on-the-fly, then it is the same for all resource consumers.

The artifact is generated by a task from the Android Gradle plugin. I can and do not want to rename them on the producer side. The producer project should not change just to satisfy the needs of the consumer. On the other hand, the consumer should handle this. Previously I renamed the artifact through processResources, but since this is not recommended, I am not sure how else to proceed here.

The previous strategy was the following:
My producer would define this property:

image

Based on that, the consumer would transform the artifact name in processResources.

Do you have any better idea in regards to that?

I’ve read upon Transforming dependency artifacts on resolution but got lost quickly again.

Bear with me and thanks for understanding and helping me!

I’m sorry, it was 2 AM here and I maybe was a bit too harsh. :slight_smile:

My producer has this:

A “dependency scope” configuration is just for declaring dependencies.
It would have isCanBeConsumed and isCanBeResolved both set to false.
But anyway in the producer this does not make any sense at all anyway, just remove it.

builtBy(tasks["build"])

This is semantically wrong and a huge waste of precious time.
It makes no sense to cause the whole build task and its dependencies to run if you just need the dex file.
It will work as long as build directly or transitively depend on the task that creates the dex file, but you should really specify the exact task that generates the file you are adding as artifact here.

Also you add the whole directory as artifact while you said you only want the dex file.
If the dex file is the only thing in that directory, it will probably work, but it would probably be cleaner to only add the file directly as artifact.

It would also be advisable to not hard-code the path to the file or directory, but get the information from the task that produces it, so that if the task gets reconfigured this is also automatically matching properly and publishing the correct file.

Can you recommend me better names than I currently have for the configurations or are those correct already?

“correct” in relation to naming in software development can never be true or false, naming is one of the hardest parts in software development. :smiley:
Actually if the producer has only one dex file, I would probably name the configuration there dexFile, not dexFiles.
On the consumer the plural is fine, because you can add dependencies to multiple producer projects so on the consumer side there could be indeed multiple.
But how you concretely name them, is mainly up to your liking. :slight_smile:

Also, the resulting artifact is now added as a resource now:

Which was the intended outcome, wasn’t it?

I can and do not want to rename them on the producer side.

Well, yes you could without disturbing anything.
You could simply add a Sync task that copies the dex file to an accordingly named subdirectory and rename the file and then configure that directory (or actually the parent directory) as artifact and it will end up in that directory in the consumer too.

My producer would define this property:

ext / extra properties are never a good idea but practically always a dirty work-around, especially if you then get the propery from a different project which is highly discouraged.

Also having that property in the producer would imho already fall into the category of “The producer project should not change just to satisfy the needs of the consumer.”, but if this is appropriate and the producer should actually control the path and name, then I’d indeed suggest doing it the Sync way I just described in the last paragraph.

but got lost quickly again

Yeah, artifact transforms are not exactly a light topic when new to them. Probably best to go with the Sync strategy just described.

1 Like

I have assumed this as well. Unfortunately, I have tried to find the task responsible for generating said artifact, but when I tried to use it, the artifact could not be found. Is there a way I can identify the task that is responsible for creating the artifact I need to share?

Yes, only a dex file is present in that directory, luckily. I am not aware of any method to share a file. The producer receives a source directory of files as we established previously, so I would need to resolve the dex file in that directory, but resources.srcDir expects a directory:

How can I share just the dex file?

I believe the task output can be used for that. I saw that the minifyReleaseWithR8 had an output in a folder. The folder had another folder, and the dex file would be inside that.

I disagree. If I name a variable files and it is a single File, then the naming is incorrect. Proper naming is key to preventing full-on chaos.

But by definition, a configuration is a collection of artifacts:

Naming it “dexFile” implies something completely different. It implies the variable represents a file. Instead, it represents a configuration that is an alias for “a group of artifacts”, more specifically given the context “a group of dex files” Therefore, the name was set to “dexFiles” and not “dexFile”. A group with one element is still a valid group, so the naming “dexFiles” works for a single dex file but not the other way around, like you suggest.

As you have seen, I would not have been able to choose “dexFiles” over “dexFile” like you suggested if I did not know the definition of a Gradle Configuration. Because I am inexperienced with Gradle, it is likely I have made similar other mistakes with naming. Just choosing an arbitrary name is insufficient because it can completely obfuscate the meaning of the code. Renaming the dexFiles variable to dexFile would start to imply it is a single file. Without type annotations, assumptions could be made, such as if it is of type File. When suddenly used as a Configuration, it behaves like a group of files. Therefore, I am asking you if I made a mistake in naming something here. Unfortunately, I did not agree with your previous naming suggestion for a reason, so if you still have a better name, please let me know.

You suggested not to use ext / extra. The idea is that there are multiple producers, and more can be added over time. I do not want to always go back to the consumer and make changes to its build system. Instead, it should get all the producers’ artifacts and add them as a resource wherever the producer has declared it to be added.

So if you suggest not using ext, how can I achieve that? Additionally, you suggested to use a Sync task. But how do I rename each artifact to its destined name? I only get a list of artifacts without any mapping to their original producer; therefore, I do not go to any ext where I could look up the artifacts’ destined names.

Is there a way I can identify the task that is responsible for creating the artifact I need to share?

Sure.
Some or multiple of:

  • documentation of the plugin that does it
  • source code of the plugin that does it
  • asking aunt Google
  • checking programmatically the outputs.files like tasks.configureEach { doFirst { outputs.files.find { it.name == "classes.dex" }?.let { println("${this.name} creates ${it.absolutePath}") } } }
  • …

Many roads lead to rome.

I am not aware of any method to share a file.

I don’t understand that sentence.
It is exactly the same way you share the directory right now and you already did it in the past, for example in the post you wrote 7 days ago.

The producer receives a source directory of files as we established previously

That’s not something we established and not something that makes sense.
The producer does not receive anything, it is the producer, it delivers something.
It can deliver a file, it can deliver a directory.

so I would need to resolve the dex file in that directory

Exactly

but resources.srcDir expects a directory

Which has nothing to do with it at all.
The “directory” that you give to srcDir is a “virtual directory”, the configuration you give to it is the directory, even if it only contains one file. This “virtual directory” merges the things provided by the producers into this “virtual directory”. If it contains a directory like you set it up now, it would be a directory with a directory in it. But the srcDir configuration is a consumer-side thing, while the artifact configuration is a producer-side thing, which you just mixed up here.

How can I share just the dex file?

Look at your own code of 7 days ago.
The consumer side does not change.

I believe the task output can be used for that.

Once you found out the correct task, yes.
You said before you know the task already.
That it was not the right task was new information in your most recent post.

I disagree. If I name a variable files and it is a single File , then the naming is incorrect.

Agreed on that. But just because singular/plural is correct, does not mean the whole name is correct.
But yeah, if singular/plural is not correct, it could probably be considered always incorrect, that’s why I said you should rename the producer configuration, where the plural is not really correct as long as you do not do or plan to add artifacts to it within one project, which I understood you will not do.

But by definition, a configuration is a collection of artifacts:

That’s a matter of technical fact vs. semantic.
Yes, a configuration by definition technically is always a collection of files.
And a collection of 0 or 1 files is of course still is a collection of files.

But if you know for sure that you always will ever only add one file to the collection,
the singular might be more appropriate semantically.
But as I said, naming is one of the hardest parts in software engineering, so name it however you like it.

But one other example just for you to think about it:

boolean[] flag = { false };
Stream.of().forEach(__ -> flag[0] = true);
System.out.println("Stream was empty: " + flag[0]);

Would you really call it flags, only because its type is boolean[] and thus could hold a boolean array of any size?
I wouldn’t. :slight_smile:

But again, naming is extremely hard and subjective, so choose any names that make you happy.
Maybe just give them animal names, then at least everyone else is pissed and confused, including your future-self. :smiley:

more specifically given the context “a group of dex files”

No, not on the producer side.
On the consumer side when you add dependency to multiple projects yes, I did not suggest to change that.
But on the producer side I understood you will always just add one dex file, so it is not “a group of dex files”, but “a group of one dex file”, at which point we again are at dexFile on the producer side :slight_smile:

But yet again, name it however you like it most. Your future-self has to live with it (or change it).

Renaming the dexFiles variable to dexFile would start to imply it is a single file.

It is a single file on the producer side.
Again, I did not suggest to rename it on the consumer side, only on the producer side.

Therefore, I am asking you if I made a mistake in naming something here.

I’d like to revise my previous statement.
Any name you or me will choose for anything will always be wrong in some context.
It will always confuse someone, maybe including future-you, it will always make someone unhappy, …

To cut it short, I will not discuss your naming.
You have to be happy with it and it has to express what you want to express.
I told you my opinion about it multiple times now.

You suggested not to use ext / extra. The idea is that there are multiple producers, and more can be added over time. I do not want to always go back to the consumer and make changes to its build system. Instead, it should get all the producers’ artifacts and add them as a resource wherever the producer has declared it to be added.

So if you suggest not using ext, how can I achieve that? Additionally, you suggested to use a Sync task. But how do I rename each artifact to its destined name? I only get a list of artifacts without any mapping to their original producer; therefore, I do not go to any ext where I could look up the artifacts’ destined names.

Seems you again totally got me wrong or only read half of what I wrote. :frowning:
The Sync task I suggested will be in the producer.
You have a Sync task in the producer that moves and renames the dex file (not into the consumer but into producer/build/whatever/extensions/shared.rve and then you add producer/build/whatever/ as artifact in the producer. The consumer then does not need to do anything about it but just gets the result controlled by the producer, so just what you depicted above with the ext and now also asked for explicitly.

1 Like

I did that, but with “classes.dex” nothing showed up, whereas with “minifyReleaseWithR8” the task “minifyReleaseWithR8” showed up. The classes.dex file is inside the “minifyReleaseWithR8” folder, which seems to be the output of the “minifyReleaseWithR8” task.

I tried adding the task to `buildBy´ but I got this error:

But the task exists:

I did that, but with “classes.dex” nothing showed up, whereas with “minifyReleaseWithR8” the task “minifyReleaseWithR8” showed up. The classes.dex file is inside the “minifyReleaseWithR8” folder, which seems to be the output of the “minifyReleaseWithR8” task.

Ah right, it of course shows the directory when the directory is configured.
But that is already enough to know it indeed is the minifyReleaseWithR8 you are after.

But you can modify your check so that it runs after the task action and resolves to the individual files like tasks.configureEach { doLast { outputs.files.asFileTree.find { it.name == "classes.dex" }?.let { println("${this.name} creates ${it.absolutePath}") } } }

I tried adding the task to `buildBy´ but I got this error:
But the task exists:

It might exist at execution phase.
But at the time you do the builtBy call it seems to not yet being registered.
Just do builtBy("minifyReleaseWithR8") and it should work, this String is then evaluated later when the task should be registered already.

1 Like

My producer now has this:

When building my consumer, the output folder “revanced” is not created. Only when I run the build a second time, it is and the consumer adds the extensions directory to the resources as expected.

I am still not satisfied with the quality of the code. Hardcoding paths like I did here for the sources is not ideal and still feels just as hacky as with my initial attempt. Note, this all is step one. I plan to turn this into a plugin for Gradle as well, otherwise when I have multiple producers, I would need to duplicate all this.
Once all this is done, the idea is to have a single block such as extension { }. I would like to configure Android in it and all the business logic for me such as:

extension {
 name = "extensions/shared.rve"

 android {
  // configure Android plugin
 }
}

But this is for later. For now, the sync task only works on the second time I build. Likely because it can not find the source files. Can I make it depend on the minifyReleaseWithR8 task? Also do I need to specify the source and target paths like I did or is there a simpler way to reference them?

My producer now has this:

You did not create a Sync type task as I advised, you do a sync { ... } directly at configuration time.
Instead you should create a task of type Sync that has the minifyReleaseWithR8 as input, syncs and renames the file and then use that task for the builtBy.

And don’t use preserve, that makes the sense of Sync void and you could use Copy instead as you then have the same drawbacks Copy would have.

I would have quickly made the changes and show it, but you always send screenshots of code which is extremely unlucky and bad practice. Code in images can not be viewed as nicely, can not be copied, can not be searched, …
Unless you want to show something like an IDE highlight of an error or whatever, never use Screenshots of text but always share the text directly, but don’t forget to use the code block markup.

Only when I run the build a second time, it is and the consumer adds the extensions directory to the resources as expected.

Not exactly “as expected”, you use the stale results of the last build run, so if something changed, you do not get that.

Hardcoding paths like I did here for the sources is not ideal and still feels just as hacky as with my initial attempt.

I fully agree and said it before.
Better use paths of task properties where possible.

1 Like