Limitation of generatedBy in native software model


(Luc Pelletier) #1

I need to use a task to generate the sources for a source set. For that, I’m using generatedBy. It works well but I’ve hit what I think is a limitation. The following basic scenario works great:

apply plugin: 'cpp'

task generateSource {
    ext.sourceDir = new File(buildDir, 'generatedSrc/cpp')
    ext.headerDir = new File(buildDir, 'generatedSrc/headers')
    doFirst {
        sourceDir.mkdirs()
        headerDir.mkdirs()
        println "Generating sources..."
    }
}

model {
    components {
        main(NativeLibrarySpec) {
            sources {
                more(CppSourceSet) {
                    generatedBy tasks.generateSource
                }
            }
        }
    }
}

As expected, when building main, the more source set is generated before compiling. I get the following output:

:buildSrc:compileJava UP-TO-DATE
:buildSrc:compileGroovy UP-TO-DATE
:buildSrc:processResources UP-TO-DATE
:buildSrc:classes UP-TO-DATE
:buildSrc:jar UP-TO-DATE
:buildSrc:assemble UP-TO-DATE
:buildSrc:pluginUnderTestMetadata UP-TO-DATE
:buildSrc:compileTestJava UP-TO-DATE
:buildSrc:compileTestGroovy UP-TO-DATE
:buildSrc:processTestResources UP-TO-DATE
:buildSrc:testClasses UP-TO-DATE
:buildSrc:test UP-TO-DATE
:buildSrc:check UP-TO-DATE
:buildSrc:build UP-TO-DATE
:a:compileMainX64DebugStaticLibraryMainCpp
:a:generateSource
Generating sources...
:a:compileMainX64DebugStaticLibraryMainMore UP-TO-DATE
:a:createMainX64DebugStaticLibrary
:a:mainX64DebugStaticLibrary

BUILD SUCCESSFUL

Total time: 0.949 secs

So far, so good. However, my use case is quite a bit more complex. I would like to define a custom source set type (say, MyCustomSourceSet), register it using @ComponentType and then based on these source sets, generate CppSourceSet sets. Given the following build.gradle:

apply plugin: 'cpp'

@Managed
public interface MyCustomSourceSet extends LanguageSourceSet {
}

model {
    components {
        main(NativeLibrarySpec) {
            sources {
                more(MyCustomSourceSet) {
                    source {
                        srcDir 'src/more/customIdl'
                        include '**/*.idl'
                    }
                }
            }
        }
    }
}

I want to write custom model rules that will create a task (‘generateMoreCppSources’) and a new source set ‘moreCpp’ which has its generatedBy attribute set to tasks.generateMoreCppSources. However, this is where I’ve hit a wall due to circular dependencies. I need to create a task dynamically based on the source sets of each component and then I need to modify the source sets so that they’re generated by this task. I’m able to create a rule that creates the task that generates the sources but I then can’t create a rule that creates a new CppSourceSet and sets its generatedBy attribute to point to my task. This is what I’ve got so far:

apply plugin: 'cpp'

@Managed
public interface MyCustomSourceSet extends LanguageSourceSet {
}

class GenerateSource extends DefaultTask {
    @InputFiles
    List<File> inputSourceFiles

    @OutputDirectory
    File outputLocation

    File sourceDir
    File headerDir

    // probably more stuff here

    @TaskAction
    void generateSources() {
        outputLocation.mkdirs()
        println "Generating sources..."
    }
}

class Rules extends RuleSource {
    private String getGenerateSourceTaskName(String customSourceSetName, String componentName) {
        return "generateCppSourceFor${componentName.capitalize()}${customSourceSetName.capitalize()}"
    }

    @ComponentType
    void registerMyCustomSourceSet(TypeBuilder<MyCustomSourceSet> builder) {}

    @Mutate
    void createGenerateSourceTasks(ModelMap<Task> tasks, ComponentSpecContainer components) {
        components.withType(NativeLibrarySpec).each { NativeLibrarySpec component ->
            component.sources.withType(MyCustomSourceSet).each { MyCustomSourceSet customSourceSet ->
                tasks.create(getGenerateSourceTaskName(customSourceSet.name, component.name), GenerateSource) { GenerateSource generateSourceTask ->
                    generateSourceTask.inputSourceFiles = customSourceSet.source.files.asList()
                    generateSourceTask.outputLocation = project.file("build/generatedSrc/${component.name}/${customSourceSet.name}")
                }
            }
        }
    }

    // ******* TODO *******
    // I need to add another rule here that creates a CppSourceSet for each MyCustomSourceSet
    // and set the generatedBy attribute of the new CppSourceSet to point to the task that I
    // created in createGenerateSourceTasks
}

apply plugin: Rules

model {
    components {
        main(NativeLibrarySpec) {
            sources {
                more(MyCustomSourceSet) {
                }
            }
        }
    }
}

Seems like the way things are structured, generatedBy will never be usable with a task created by a model rule. This is very limiting since the generatedBy task cannot be created based on what’s been created in the model.

Any ideas?

Thanks


(Sterling Greene) #2

Yes, I ran into this same situation about a month ago and raised it internally. I think this is a limitation with the way you can specify inputs/subjects with model rules that we still need to address. There are people actively working on making that better.

For my particular case, it wasn’t a huge problem to keep generatedBy and the task definition in the build script, so I didn’t try to solve the problem some other way. I didn’t try it, but I thought there might be a way to work backwards. Generate all the ‘generator’ tasks from the defined source sets but don’t try to wire them together. Then after all the tasks are generated (compile and generator), figure out which generator tasks the compile tasks should depend on.

I raised GRADLE-3445 since I’m not alone for wanting to do this.


(Luc Pelletier) #3

Thanks Sterling.

I tried your alternate solution and it looked promising but it got quite complicated when depending on generated source from other projects in a multi-project setup.

In the end, I decided to backtrack and create my source generation tasks in my plugin’s apply method. I search for all idl files in the project and create tasks based on the files that are found. My idl files are grouped per component so I create one task per component. I don’t have access to the components but I can infer their name from the location of the idl files (eg. for all files under src/main/idl/, I know main is the component name). I always follow the same pattern for naming the tasks: a set prefix followed by the component name (eg. compileIdlMain). I then have to explicitly define a CppSourceSet and set its generatedBy attribute for each component in each build.gradle file. This is not ideal but it at least works in all scenarios, including cross-project dependencies.

Please keep me posted if you have any other ideas.

Thanks again.