Understanding the underlying gradle magic - back to basics

Help/Discuss
1

I’ve been using gradle for a few years now and can do some very advanced things. But there is still a significant does of ‘magic’ that escapes me and frequently catches me off guard. I need help with this - a primer somewhere that will clear the haze.

Some questions:

  1. How do *.gradle files exactly become groovy and at what point? What *.gradle constructs map to what in Groovy?

  2. Exact details on evaluationDependsOn ordering. I have had to judiciously set those to avoid headaches but sometimes I get circular reference errors when there should be none … and I don’t understand why. I superficially looked at Gradle source code and this seems to be the trigger point … (note that I am “making adjustments” to the dependent project from its dependency)

    private Project evaluationDependsOn(DefaultProject projectToEvaluate) {
    if (projectToEvaluate.getState().getExecuting()) {
    throw new CircularReferenceException(String.format(“Circular referencing during evaluation for %s.”,
    projectToEvaluate));
    }
    return projectToEvaluate.evaluate();
    }

  3. Some documentation seems incomplete. For example, java plugin should create configurations with names prefixed with the source set name, but “main” source set doesn’t seem to yield any prefix - i.e. it is “compile”, not “mainCompile” as it is in “testCompile”.

  4. What default magic causes multi-project compilation tasks to execute in proper order when I only declare configuration dependencies and not task dependencies? When I change that configuration a bit to try to explicitly handle stuff, build falls apart. For example, relying on “default” configuration as a dependency isn’t good enough for us. Dependent’s “compile” needs to depend on the dependency “jar” alone, dependent’s “runtime” on dependency’s runtime (plus the jar), etc. However, when I change the simple dependency on project’s default configuration, not only I am forced to also declare/add the compilation task dependencies but also build fails. I declared a “mainProduct” configuration that has the 'jar in it. I log this to make sure it is happening. It is. Yet, when I run the “dependencies” task, it shows as empty. Dependent projects that depend on this mainProduct configuration (jar) fail to build… And I don’t know where to start debugging. Logging indicates all is done but nothing takes hold…

What set of basics am I missing?

Update: Note - some posts in the thread went a bit off onto a tangent, solving one specific issue. What I am really looking for is a more detailed help/reference/guide that will help me learn what I am missing so that I won’t have these questions any more. There are some more of these added in posts below.

2

The .gradle files are compiled to byte code first time the build runs (and any time a .gradle file changes). The .gradle file is Groovy with a couple of ASTs on top of it (e.g., you can use task name(...) instead of task("name", ...). Every method or property you find in a .gradle file is going to be a method or property on the particular object (a Task, an extension, the Project, etc).

I’m not sure why you’re doing this. Why is a dependency changing the dependent project?

Yes, the built-in source set main doesn’t follow that convention. I don’t know the history, but it either predated that convention or it was considered worth the inconsistency for the 99% case.

Configurations, FileCollections and other Buildable things have information about which task produces them. When you have a project dependency, Gradle uses the default configuration which has artifacts registered to it. For a Java project, the jar file is registered as an artifact produced by the jar task. The jar task knows that it archives sourceSets.main.output, which is produced by the processResources and compileJava tasks. That information is used when building the task execution DAG.

The correct execution order is a product of the dependency information and the DAG.

My suspicion is that your mainProduct configuration only knows about the particular jar file and not the task that produced it. You should have something like…

artifacts {
   mainProduct jarTaskThatProducesTheExtraThing
}

There are a lot of samples that come in the Gradle distribution that may help and there are some free resources. Udacity course, while made for Android users, covers some Gradle basics and is free.

2 Likes
3

Yes, that much I knew. It is the exact “ASTs on top of it” that escape me.

Working around Eclipse classpath model deficiencies. You can see some details in my other question post here: https://discuss.gradle.org/t/main-vs-test-compile-vs-runtime-classpaths-in-eclipse-once-and-for-all-how/17403
In short, we maintain having in-project “main” and “test” source sets outside Eclipse. However, when we detect that Eclipse-related tasks (such as eclipseClasspath) are executed we declare additional projects for each exiting Java project. If a project is named we dynamically create a project .test. These are local only (not in source repo) and are otherwise ignored by gradle builds. Their Eclipse configuration is set from the “parent” (main) project - source location, test classpath, etc.

Thank you!

Thank you! I suspected there is something about this but never had officially found it.

Thank you! That must be it. I explicitly set the file because I believed that i what was needed, hence cutting the branch I was sitting on.

Thanks! Will have a look.

4

Re mainProduct.

Here’s the configuration definition:

javaProject.configurations {
  mainProduct {
    description = "..."
    transitive = false
  }
}

I changed the dependency declaration from:

javaProject.dependencies {
  println("${javaProject.getName()}.mainProduct: ${javaProject.jar.archivePath}")
  mainProduct javaProject.files(javaProject.jar.archivePath)
  ...
}

to:

javaProject.dependencies {
  println("${javaProject.getName()}.mainProduct: ${javaProject.getName()}.jar")
  mainProduct javaProject.files(javaProject.jar)
  ...
}

The output of (actual project names changed):

gradle <project-name>:dependencies

… is:

...
<project-name>.mainProduct: <project-name>.jar
...
mainProduct - ...
No dependencies
...

I tried building a downstream project that depends on this configuration and it fails, so it does not look like “dependencies” task is simply not showing this one.

5

Woops… I didn’t change all. I am tired. New changes:

javaProject.dependencies {
  println("${javaProject.getName()}.mainProduct: ${javaProject.getName()}.jar")
  mainProduct javaProject.jar
}

But now I get:

...
<project>.mainProduct: <project>.jar
...
FAILURE: Build failed with an exception.

* Where:
Build file '...\build.gradle' line: 1

* What went wrong:
A problem occurred evaluating project ':<project>'.
> Failed to apply plugin [id 'enhanced-dependencies']
   > Cannot convert the provided notation to an object of type Dependency: task ':<project>:jar'.
     The following types/formats are supported:
       - Instances of Dependency.
       - String or CharSequence values, for example 'org.gradle:gradle-core:1.0'.
       - Maps, for example [group: 'org.gradle', name: 'gradle-core', version: '1.0'].
       - FileCollections, for example files('some.jar', 'someOther.jar').
       - Projects, for example project(':some:project:path').
       - ClassPathNotation, for example gradleApi().

     Comprehensive documentation on dependency notations is available in DSL reference for DependencyHandler type.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
6

Wait… artifacts { … }? Let me read up on that.

7

OK, found some examples here:

http://mrhaki.blogspot.ca/2016/02/gradle-goodness-inter-project-artifact.html

I need to have both configuration and an artifact definition and the configuration must be declared as transitive (I had it intentionally at transitive=false). I originally thought that artifacts are only used for publishing. Now the build miraculously passes but the mainProduct is still shown as empty. Why are these things omitted?

8

Unfortunately, this just continues the “magic”. I can see people recommending it and I can see it working, but I don’t know why and how. What is the magic behind same-named artifacts and configurations? They obviously can exist on their own, but when they are together something new happens - 1+1=3. Why does it need to be transitive? I don’t want dependencies of the artifact, just the artifact itself. Where can I learn about this?

9

More basics that escape me… Gradle has three documented phases- initialization, configuration and execution. Yet it has “evaluationDependsOn”? What is the “evaulation” exactly? I always thought of it as a part of configuration but have no details. I also know that if I don’t specify it I risk the configuration of dependent project failing by referencing still-unconfigured dependency project. Yet I don’t understand the magic of how is it possible that, somehow, “evaluationDependsOn” is executed what seems to be configuration … but other configuration things are delayed. What is the default evaluation order?..

I am missing lots of these things. Trying to find books that explain this as well. Is there anything, short of actually looking at Gradle source code, that will help me understand the details?

10

Evaluation (in particular project evaluation) means the configuration phase.

More pedantically speaking:

  • The evaluation of the build.gradle files an their included snippets constitutes the configuration phase.
  • The evaluation of settings.gradle (if any) constitutes the initialization phase.
  • I am not sure what is the nomenclature for the evaluation of the init.gradle scripts - are they lumped into the initialization or in some other pre-init phase.

For the record, I am using Gradle from the early 1.x days, in relatively complicated scenarios (legacy and such) and I haven’t had the need to use explicit hints as evaluationDependsOn. Gradle is quite good at figuring the eval order based on the actual dependencies you specify, as long as you give it enough info (i.e. if you depend on the compiled classes, declare that your source fileset is compile, not file('build/classes'), etc.)

11

What is unclear to me is that declaring dependencies is configuration. At the very least it is intermingled with it. So, I can see how the execution can be affected by this but I do not understand how it can affect the order of the configuration itself.

13

This one really helped me get my head around a few things:

1 Like
14

Yes, that is indeed helpful… but I was thrown and the deep end (wrote plugins) and figured that bit already in the process. It may help other readers of this thread, though. It still does not answer my deeper questions.