Native cpp: automatically handle transitive include dependencies


(Ralf Schallenberg) #1

Hi, is there a way to let Gradle handle transitive include dependencies automatically?
Say there’s library component B, which depends on library component C. Then there is an executable component A, which depends on B and includes headers from B which include headers from C. I can’t get this to compile unless I add the dependency on C to components B and A. Is there a smarter way to do this?


(Gary Hale) #2

There’s currently not a way to declare native dependencies transitively. This is a feature that we intend to add, and there is significant work under way to handle dependencies between components in general (i.e. a model that encompasses both native and non-native components). In the meantime, the only way to deal with this currently is to declare both dependencies.


(Ralf Schallenberg) #3

I see, thanks for the information!


(Marco) #4

Hi Gary,
Do you have a deadline to provide this feature? I work in a big software factory in Italy and i would like to use Gradle for native C binaries. I like Gradle, but this is a big limitation compared at other build tools like Cmake.
To resolve this problem i think to put all compiled libraries in a single directory and add a linker.flag “-L/dir_with_all_libraries” to avoid linker errors in the executables

Do you think is this a good solution?
I hope the transitive dependencies will be soon ready!
Thanks for your work!


(Gary Hale) #5

Unfortunately, I can’t give you a definitive timeline for when transitive dependencies for native libraries will be available other than to say it’s something that many users want and something that we want to provide, so the best I can say is “soon” but probably not in the next few months.

Regarding your question about using a linker flag, I’d say that’s a reasonable approach if you’re not willing to model the dependencies directly.


(Marc Pawlowsky) #6

Is there a way to track the progress of this feature?
It is a show stopper for us, and I would like to re-evaluate gradle when this feature is implemented.

When implementing this feature please have the option to transitivly have the include files used, or not used.

For example a library being used you may need to link in other libraries and reveal their headers. Yet for another library the transitive libraries header files should not be revealed as they are not part of the public api.


(Sterling Greene) #7

To give a bit of the progression we’re going through

  • We have dependency management for the JVM (Maven/Ivy) working for Java/Groovy/Scala projects using configurations and a dependencies {} block (what you’ll see most Gradle build files using)
  • All of the native plugins use the software component model
  • Dependency management only exists in the native plugins through the Prebuilt libraries mechanism and project dependencies (nothing transitive or external through Maven/Ivy).
  • We’re implementing JVM plugins on top of the software component model (Android will use this too)
  • For the JVM plugins, we need to have dependency management on par with what we already had. So we’re bringing that into the software component model first for the JVM plugins. You can already see some of that in the ‘javaLibraryPlugin’ samples.
  • For the native plugins, the plan is to take advantage of the dependency management brought in by the JVM work. We think a lot of that is going to be similar to the work we’re doing for JDK9 and modular JDK/Jigsaw. JVM modules have a similar idea of exporting dependencies or not.

So, we know we need to do something here, but it has to happen after the JVM work. This is all good information for when it comes time to plan what we’re going to do. e.g., I’d love to be able to do something like (this is just something I have in mind, not necessarily what we’ll do):

model {
    components {
        lib(NativeLibrarySpec) {
            api {
                exports 'LIB_MACRO_FOO', 'LIB_MACRO_BAR'
                exports 'lib/include'
                exports 'log4cxx'
            }
            macros {
                define 'LIB_MACRO_FOO'
                define 'LIB_MACRO_BAR', 0
            }
            dependencies {
                shared library 'log4cxx' // log4cxx is defined elsewhere in a PrebuiltLibraries, from what's on the system, pkg-config, etc.
            }
        }
        exe(NativeExecutableSpec) {
            dependencies {
                shared library 'lib'
                // exe automatically gets compiler args for defining LIB_MACRO_FOO and LIB_MACRO_BAR
                // links against 'lib' and 'log4cxx'
                // and gets 'lib/include' added as an include path
            }
        }
    }
}

C/cpp plugin - GC overhead limit exceed
(Phil McArdle) #8

Sorry for reviving this topic a year later, but I haven’t seen anything more recent, or as authoritative, and we just hit this issue ourselves.

Is there any information on how far through this rough checklist we are?

Or anything close to an estimate of when native might have this?
Just in terms of informing how much work we should do ourselves to build a solution in the meantime.

Thanks,

Phil


(Daniel Lacasse) #9

Thanks Phil for raising this limitation again. An issue was raised on Github regarding just that. I suggest you subscribe to that issue in order to stay in the loop of any work done on this.

Due to the complexity of this feature, it would also be very helpful if you could add a +1 reaction to any commented scenario that apply to you. Better yet, if you could come up with some of your day-to-day scenario regarding transitive dependencies and comment them back to the issue so we can get a feel of where to invest our energy.

Unfortunately, we cannot give any estimate at the present moment. Custom solution is the only way at the moment. If you do have a open source solution that you are willing to share, I would encourage you to post it here for the other community member to see and use in the meantime.

Thanks for the patience,

Daniel


(Phil McArdle) #10

Thanks Daniel.

I’ve been trying to copy lib dependencies from one source set* to another programmatically within Gradle but I either create a dependency cycle, or get no information at all, and would hugely appreciate any pointers.

* I’m targeting this just now as what should be a simple feat, hah, and it would solve both of our scenarios (I believe?)

Using Donal’s Scenario 1 from GH 801.

In short, I want to append Lib1's list of libs to Exe1's list of libs, but I cannot even get close to this right now. e.g., given:

Lib1(NativeLibrarySpec) {
    sources {
        cpp{
            lib library: 'Lib2', linkage: 'static'
        }
    }
}

Exe1(NativeExecutableSpec) {
    sources {
        cpp{
            lib library: 'Lib1', linkage: 'static'
        }
    }
}

I want to edit this dynamically to be:

Lib1(NativeLibrarySpec) {
    sources {
        cpp{
            lib library: 'Lib2', linkage: 'static'
        }
    }
}

Exe1(NativeExecutableSpec) {
    sources {
        cpp{
            lib library: 'Lib2', linkage: 'static'            
            lib library: 'Lib1', linkage: 'static'
        }
    }
}

And so on, so that Gradle will actually link with Lib2 (and/or for Scenario 2, in the case of shared libraries, include them in the install tasks).

I’ve been trying a few different approaches.

The approach that seemed most natural to me was to try to read the libs map of the source set on Exe1, which works provisionally fine (I get [library:Lib1, linkage:static] from within Exe1)

Exe1(NativeExecutableSpec) {
    sources.withType(CppSourceSet).all { sourceSet ->
        sourceSet.libs.each { lib ->

        }
    }
}

It would have been my intention to move that into common code via components { all { or similar.

However, within that block, I cannot reach Lib1 using code like this project.components.getByName('Lib1') (cycle), nor can I search components with matching { it.name == 'Lib1' }.all (Cannot invoke method all() on null object), and a few other methods that I’ve since removed that weren’t direct references also get no results.

I’ve tried afterEvaluate {}, but to not avail. In several if not all cases, my afterEvaluate code seems to execute earlier than the code in model {}, so I’m not sure what I’m doing wrong there. I’ve tried components.whenObjectAdded, but this is never triggered.

I’ve then briefly tried to parse the complete list of components or binaries to generate a separate list outside of the model that I can iterate later to update the dependencies, but while I can successfully report this data to the console, and access it within that loop, all code outside of that sees the variable as empty.

I’ve thought about task dependencies, since at the moment the task order is:

:compileLib1StaticLibraryLib1Cpp
:createLib1StaticLibrary
:Lib1StaticLibrary
:compileExe1ExecutableExe1Cpp
:linkExe1Executable
:Exe1Executable
:compileLib2StaticLibraryLib2Cpp
:createLib2StaticLibrary
:Lib2StaticLibrary
:compileLib1SharedLibraryLib1Cpp
:linkLib1SharedLibrary
:Lib1SharedLibrary
:compileLib2SharedLibraryLib2Cpp
:linkLib2SharedLibrary
:Lib2SharedLibrary
:assemble

And we know from looking at the build script that linkExe1Executable requires the output of linkLib2SharedLibrary, but I’m struggling to work out how to dynamically infer that even on a task basis, given the example code linked.

Any ideas?

Thanks,

Phil


(Daniel Lacasse) #11

Thanks Phil for sharing your progress here. Dealing with the current and software model is quite tricky. The execution order can be simplified as follow:

  1. Evaluate build script. Most of the code present in the build script will be executed. However, this exclude anything differs to a later stage. This include doLast, doFirst, etc. At this stage, the model rules are discovered but not executed.
  2. Execute afterEvaluate action. Since the build script has finished his evaluation, the afterEvaluate action will now be triggered.
  3. Execute required model rules. Depending on what need to be done, a subset of the rules will be executed. This prevents from executing configuration code that is not needed for the execution tasks. The rules are ordered with different scope @Defaults, @Mutate, @Finalize, @Validate. Those scopes are documented here in the user guide.

As you can see, the execution is not that easy to follow. To make things more complex, the software model tries to enforce immutability once a model node is closed. This is crucial for the well-being of the model rules. This immutability is enforced on @Managed type but not on @Unmanaged type. It is highly recommended to manually enforce that immutability on @Unamanged type. Pretty much all native type (components, binaries, etc.) are unmanaged.

What you are trying to do require will require the use of internal API. If possible always avoid using internal API as they may change a lot between Gradle version. For any internal API that you may think it should be public, please open a feature request on Github with an use case to help us better create public API for plugin author.

All warning aside, your solution path is probably only achievable through Java version of the software model rules. Why using the Java version of the rules? Because you get to order the rules with @Defaults, @Mutate and @Finalize. All Groovy version of the software model are entered as @Mutate and will compete between each other. By taking the care that all components/binaries are create under the @Defaults and @Mutate scope, during the @Finalize scope, any components.getByName should resolve to the component you are trying to reference. You will also need a way to access the model from another project. The only internal supported way to achieve this is demonstrated by the following code from the visual-studio plugin. Basically, you need to get as input the ServiceRegistry node in your model rule and then get an instance of ProjectModelResolver. With this resolver, you can get any ModelRegistry from :any:path:to:a:project. Then you can realize any node from that model. For example, realize("components.myExecutable", NativeExecutableSpec.class) will return the myExecutable NativeExecutableSpec fully configured (all rules affecting this model node will be executed).

An extreme work of caution is needed at this point. This is not meant to be public so they may be some limitation and unexpected behavior. We welcome any bug found as well as PR to fix those bugs. However, no official support will be guaranteed.

I hope this help with your quest for transitive dependencies,

Daniel


(Phil McArdle) #12

That’s a lot of information, which gives me a lot to think about for some of our other problems too, thank you.

I think at this point I might continue with our lightweight workarounds and hope the Gradle team is able to solve at least some of the scenarios, given the complexities involved :slight_smile:

Regards,

Phil