How do i hook the LinkExecutable task in Native Builds

ImGanesh · November 18, 2016, 2:30pm

Here is a set-up of my multi-project build

project
|_build.gradle
|_subprojA
|__build.gradle
|__build
|___exe
|____subprojA
|_____subprojA
|_subprojB
|__build.gradle
|__build
|___exe
|____subprojB
|_____subprojB

For linking objects created in different sub-directories, I would like to create a new custom task on the root project by using the LinkExecutable task type provided by Native Gradle. Attached is my incomplete build.gradle containing this custom task.

apply plugin: 'c'
model {

task myLinker(type: LinkExecutable) {
	  
       dependsOn << ":subprojA:subprojA_componentExecutable"
       dependsOn << ":subprojB:subprojB_componentExecutable"

		source {
                        subprojA/build/exe/subprojA/subprojA
                        subprojB/build/exe/subprojB/subprojB
                 }
       outputFile = file("build/exe/project")
  }

}

However, I get an error. I would like to link these object files and get a partially linked Executable. Any tips on the usage of this task type would be nice.

Daniel_L · December 2, 2016, 9:47pm

My bad for not responding to this question sooner. Although the LinkExecutable task is part of the public API, it isn’t expected for a user to create his own instance of that task. There are several factors to take into consideration. The public API of this task is the type for filtering (tasks.withType(LinkExecutable)).

We had other discussions regarding partial linking. If this is still a valid question, we can try to come up with a way to use LinkExecutable as a custom type. As a quick help, this is how the LinkExecutable task is created in Gradle.

That been said, it is still a very valid use case to create your own LinkExecutable. You could open a feature request to document this in the user guide as well as making sure you can achieve your specific use case without using internal API.

I how this helps,

Daniel

ImGanesh · December 3, 2016, 7:34am

Hi Daniel,

Thank you so much for the reply. I did manage to find a way of using the public API of the LinkExecutable task for my build system. It was a bit tricky to do it because I thought the Gradle documentation was not very comprehensive for this and will open a request to enhance it. I also went through your other answer on partial linking. And yes that I what I intend to do - extend the Native Model for building Partially Linked objects. However, I do have one nagging concern about the design.

Currently, for my root project’s build.gradle, I have the components part structured as follows:

components {
	CSWtooldb (NativeExecutableSpec) {
		targetPlatform "mySparc"
	}	
	CSWswdb (NativeExecutableSpec) {
		targetPlatform "mySparc"
	}
	tasks.withType (LinkExecutable) {
		if (name == "linkCSWtooldbExecutable") {

// the two tasks below copy their executables to ../objs_exe/tooldb

			dependsOn << ":root_proj/sub_projA:libsub_projA_tooldbExecutable"
			dependsOn << ":root_proj/sub_projB:libsub_projB_tooldbExecutable"
			source {
				fileTree (dir: 'root_proj/objs_exe/tooldb')
			}
		}		
		else if (name == "linkCSWswdbExecutable") {

// the two tasks below copy their executables to ../objs_exe/swdb

			dependsOn << ":root_proj/sub_projA:libsub_projA_swdbExecutable"
			dependsOn << ":root_proj/sub_projB:libsub_projB_swdbExecutable"	
			source {
				fileTree (dir: 'root_proj/objs_exe/swdb')
			}			
		}
	}
}

The idea is to build the root project using the partially linked sub-project objects. There are two components for the root project as well as for the sub-projects - they have the same source set but have different api linkages. (one is linked using header files from toolDB and other from swDB). Next, there is a second phase of link on the root project side. Clearly, the source for the linker in this case are the partially linked sub-projects’s objects. I want to also check if the sources are up to date before I do this second part of linking. If the sources have changed, then I want Gradle to first build and link the sub-project before doing the second phase of linking. So, I added the dependsOn part to the task.

I would like to know if this is the right way to do it given Gradle’s existing features. Or are there other ways in which Gradle helps with this type of builds? (for example - two components having the same source set but different linkages)
I have an if and else if closure in my task usage because I have no idea how to customize tasks component wise. For example, when building the tooldb variant of the root project, I want only the sub-project’s tooldb objects as source to the linker.
Is it possible to encapsulate these activities when I extend the Native model?
I think this is a bit far fetched question but anyways, I would also like to know if similar use cases have come up with other teams which use Gradle and if there is some documentation on how it was done.

Thanks a lot again!
Ganesh

Daniel_L · December 9, 2016, 10:06pm

Hi Ganesh,

Always nice to answer your question as they are full of learning opportunities.

I would like to know if this is the right way to do it given Gradle’s existing features. Or are there other ways in which Gradle helps with this type of builds? (for example - two components having the same source set but different linkages)

Given my comprehension to what you are trying to do, your solution seems legit in solving your current use case. I have opened an issue regarding better dependency management for native. You could have a look at it and add your comment/use case if you feel we are missing some information. A better dependency management would really help those kinds of scenario.

I have an if and else if closure in my task usage because I have no idea how to customize tasks component wise. For example, when building the tooldb variant of the root project, I want only the sub-project’s tooldb objects as source to the linker.

Thankfully, each BinarySpec has a getTasks() method that returns all the task affecting the current binary. With this container (it’s a NamedDomainObjectSet) you can use withType as well as matching. However, it is assumed that only one LinkExecutable exists (may not always be true).

Is it possible to encapsulate these activities when I extend the Native model?

It is very easy to encapsulate those activities. I suggest creating rules in a plugin using RuleSource as it’s a bit more powerful than Groovy rules (using $.) and you can write them in Java to take advantage of static typing and Intellij code completion. For example, you could have a rule applied to every NativeComponentSpec like this:

class Rules extends RuleSource {
    @Mutate
    void applyToNativeComponent(@Each NativeComponentSpec component) {
        // do something
        component.binaries.withType(NativeExecutableBinarySpec) {
            tasks.withType(LinkExecutable) {
                // do something with this binary's link executable task
            }
        }
    }
}

I think this is a bit far fetched question but anyways, I would also like to know if similar use cases have come up with other teams which use Gradle and if there is some documentation on how it was done.

No question is far fetched. Native is a complex sphere as most build tools requires the user to specify how they want the job to be done. Gradle paradigm is aligned with what you want to do. For your particular case of partial linking, I don’t think this was explored with Gradle. Don’t hesitate to open feature request for the various scenario that isn’t trivial to solve with Gradle.

Bonus

The following code:
dependsOn << ":root_proj/sub_projA:libsub_projA_swdbExecutable"
is preferred to be written as:
dependsOn ":root_proj/sub_projA:libsub_projA_swdbExecutable"
The reason is you are invoking the getter for dependsOn property and appending to the list. This assumes the returned list from getDependsOn is the internal reference for the list. This may not always be true. In some case (elseware in the codebase), it’s a copy of the list that is returned and changing the returned list won’t have any effect as it’s going to get garbage collected right after.

I hope this helps,

Daniel

ImGanesh · January 19, 2017, 9:59am

Thank you. This was so helpful for me. I have used it for my activities. However, I am unable to use some substitutions like $buildDir and ${baseName} as part of my rules. Like for example below

class PartialLinkRules extends RuleSource {
@Mutate
void applyToNativeComponent (@Each NativeComponentSpec component) {
component.binaries.withType(NativeExecutableBinarySpec) {
tasks.withType(LinkExecutable) {
doFirst {
outputFile = file(“$buildDir/exe/${baseName}/${baseName}.o”)
}

I get the error such as

Could not get unknown property ‘baseName’ for task ‘:my_path:linkLibmysourceExecutable’ of type org.gradle.nativeplatform.tasks.LinkExecutable. How do I bring these variables within scope of my rules?

Daniel_L · January 19, 2017, 1:15pm

Thanks for coming back on this. For buildDir, you have to take a File input property and specify the model path with @Path annotation. As for baseName, it should pick it up according to the delegation rule of Groovy. However, in case it doesn’t pick it up, you can name the implicit argument for the closure passed to withType(NativeExecutableBinarySpec) and access the property via that named variable. For example,

class PartialLinkRules extends RuleSource {
  @Mutate
  void applyToNativeComponent (@Each NativeComponentSpec component, @Path("buildDir") File buildDir) {
    component.binaries.withType(NativeExecutableBinarySpec) { binary ->
      tasks.withType(LinkExecutable) {
        doFirst {
          outputFile = file("$buildDir/exe/${binary.baseName}/${binary.baseName}.o")
        }
      }
    }
  }
}

ImGanesh · January 20, 2017, 8:18am

Thank you. Unfortunately the delegation rules or the implicit arguments passed to the function is not recognized by Gradle.

apply plugin: 'c'

class PartialLinkRules extends RuleSource {
	@Mutate
	void applyToNativeComponent (@Each NativeComponentSpec component, @Path("buildDir") File buildDir) {
		component.binaries.withType(NativeExecutableBinarySpec) { binary ->
			tasks.withType(LinkExecutable) {
				doFirst {
					outputFile = file("$buildDir/exe/${binary.baseName}/${binary.baseName}.o")
				}
			}
		}
	}
}

apply plugin: PartialLinkRules

No such property: baseName for class: org.gradle.nativeplatform.NativeExecutableBinarySpec

Does it make a difference that this project is actually a sub-project in a multi-project configuration and this Rule definition is within the sub-project? The idea is to factor this out and make it apply to all sub-projects in the Multiconfig Project structure.

Daniel_L · January 20, 2017, 12:05pm

My bad, the baseName is on the component. So the code should look like this:

class PartialLinkRules extends RuleSource {
  @Mutate
  void applyToNativeComponent (@Each NativeComponentSpec component, @Path("buildDir") File buildDir) {
    component.binaries.withType(NativeExecutableBinarySpec) {
      tasks.withType(LinkExecutable) {
        doFirst {
          outputFile = file("$buildDir/exe/${component.baseName}/${component.baseName}.o")
        }
      }
    }
  }
}

ImGanesh · January 20, 2017, 12:52pm

Thanks. But unfortunately I get another error.

Could not find method file() for arguments [/to/my/dir/build/exe/my_component/my_component.o] on task ‘:path_to_component:linkLibComponentExecutable’ of type org.gradle.nativeplatform.tasks.LinkExecutable.

Daniel_L · January 20, 2017, 2:46pm

The method file(...) is part of the Project class. Since you default delegation of a build script is a Project instance, calling file(...) in the scope of the build script will work. The suggested workaround would be to use normal Java API like this:

outputFile = new File(buildDir, "exe/${component.baseName}/${component.baseName}.

ImGanesh · February 10, 2017, 9:22am

Thanks. I have one more question regarding this. Can I access the outputFile property of this task at configuration time? When I removed the doFirst closure, it did not work.