How to supply source jars from flatDir for eclipse debugging

[rstasko (Roger Stasko)]

After upgrading to eclipse 2019-12 and buildship 3.1.3 I found that eclipse wouldn’t let me manually attach source code to its “Project and External Dependencies” node, hence no dependency source code for debugging.

After some buildship searching I found that I could replace the “compile files” dependencies with a “compile” dependency combined with a flatDir repository. Initially this worked as I was able to view the associated source code when debugging in eclipse.

However, after upgrading and rebuilding the dependencies and copying the jars and -source.jars to the flatDir (using the same name and version number), the source code is no longer available in eclipse. I even tried adding an eclipse.classpath.file.whenMerged override but that hasn’t helped.

I haven’t tried recently, but I suspect if I change the filename/versions of the dependent files the source file will be included but I’d prefer not to do that as I’d like all projects to share the latest patch for a specific major.minor version automatically when being rebuilt.

Also, I’d like to steer clear of maven so I presume I’m stuck with flatDir for these locally-managed dependencies.

Here’s my build.gradle file:

apply plugin: 'java'
apply plugin: 'application'
apply plugin: 'eclipse'

repositories {
    jcenter()
    flatDir {
    	dirs "F:/java/lib/gradlerepo"
    }
}

dependencies {
//    api 'org.apache.commons:commons-math3:3.6.1'
//    implementation 'com.google.guava:guava:23.0'
    testImplementation 'junit:junit:4.12'
    compile 'ToolboxCore:ToolboxCore-1.0'
    compile 'ToolboxSwing:ToolboxSwing-1.0'
    compile 'org.slf4j:slf4j-api:1.7.25'
    compile 'org.jopendocument:jOpenDocument:1.3'
    testCompile 'ch.qos.logback:logback-core:1.2.3'
    testCompile 'ch.qos.logback:logback-classic:1.2.3'
}

java {
	sourceCompatibility = JavaVersion.VERSION_13
	targetCompatibility = JavaVersion.VERSION_13
}

mainClassName = 'Financial'

task srcJar(type: Jar) {
	from 'src'
	archiveName mainClassName + '-sources.jar'
}

compileJava.dependsOn srcJar

test {
//	testLogging.showStandardStreams = true
	exclude 'financial/lookup/LookupTestSuite.class'
}

run {
	systemProperty 'application.home', 'f::\\financial'
}

I also tried adding this to the build.gradle file but it didn’t help…

import org.gradle.plugins.ide.eclipse.model.Library

eclipse {
    classpath {
        file {
            whenMerged { 
                def lib = entries.find { it.path.contains 'ToolboxCore-1.0.jar' }
                lib.javadocPath = fileReference(file('ToolboxCore-1.0-javadoc.jar'))
                lib.sourcePath = fileReference(file('ToolboxCore-1.0-source.jar'))
            }
        }
    }
}

[Lance (uklance)]

Don’t use flatDir!! Instead, use the Maven repository directory layout for your jars, sources jars, javadoc jars and poms. You can then configure a local directory as a maven repository in Gradle and your IDE will automatically find the javadocs and sources.

See here

How to use sources / javadoc of local compile dependency in Eclipse / Buildship

I’d say the easiest option would be to store your files in a maven-like folder hierarchy $PATH/xxx/yyy/lib/1.0/lib-1.0.jar $PATH/xxx/yyy/lib/1.0/lib-1.0-javadoc.jar $PATH/xxx/yyy/lib/1.0/lib-1.0-sources.jar $PATH/xxx/yyy/lib/1.0/lib-1.0.pom (not required but maven can generate this for you) Then include it via repositories { maven { url uri("$PATH") } } dependencies { compile 'xxx.yyy:lib:1.0' } If you want to script this process you might want to invoke the maven command line …

[rstasko (Roger Stasko)]

I replaced my flatDir with a directory modeled after maven and I replaced the compile dependency with the appropriate format. However, the compile task fails because I have no corresponding .pom files. Is there a way to generate the .pom file without maven or can this problem be resolved by adding some data to the manifests?

Here’s the error I received…

Execution failed for task ‘:compileJava’.

Could not resolve all files for configuration ‘:compileClasspath’.
Could not find ToolboxCore:ToolboxCore:1.0.
Searched in the following locations:
- https://jcenter.bintray.com/ToolboxCore/ToolboxCore/1.0/ToolboxCore-1.0.pom
- file:/F:/java/lib/mavenrepo/ToolboxCore/ToolboxCore/1.0/ToolboxCore-1.0.pom
Required by:
project :

My updated repositories and dependencies look like this:

repositories {
jcenter()
// flatDir {
// dirs “F:/java/lib/gradlerepo”
// }
maven {
url uri(“F:/java/lib/mavenrepo”)
}
}

dependencies {
// api ‘org.apache.commons:commons-math3:3.6.1’
// implementation ‘com.google.guava:guava:23.0’
testImplementation ‘junit:junit:4.12’
// compile ‘ToolboxCore:ToolboxCore-1.0’
compile ‘ToolboxCore:ToolboxCore:1.0’
compile ‘org.slf4j:slf4j-api:1.7.25’
testCompile ‘org.slf4j:slf4j-simple:1.7.25’
}

[Lance (uklance)]

If a jar was built by maven, the pom should be inside the jar under the META-INF directory so for many jars you could extract this.

For anything else you could create a minimal pom using

<project>
    <modelVersion>4.0.0</modelVersion>
    <groupId>xxx</groupId>
    <artifactId>yyy</artifactId>
    <version>zzz</version>
</project>

See https://maven.apache.org/guides/introduction/introduction-to-the-pom.html#Minimal_POM

[Lance (uklance)]

Another option is to do

dependencies {
   compile 'ToolboxCore:ToolboxCore:1.0@jar'
   ... 
} 

I’m hoping you don’t need a pom when explicitly requesting the jar artifact

[rstasko (Roger Stasko)]

The jar wasn’t built by maven, but creating the minimal .pom manually worked! My build is picking up the latest binary jar and sources jar from my maven “repository” and I can now step through and set breakpoints on the libs in eclipse.

I assume that it should be possible to generate the pom in the future but that would require changes to automatically update the version number, which I’m not ready to tackle yet. I’d like to move from a major.minor version number to a full major.minor.patch version number and I don’t want to have to change the build.gradle every time I run a build.

Thanks for your help!

[rstasko (Roger Stasko)]

I agree – seems odd that a pom is required since the group/artifact/version can all be derived from the compile statement. But since the poms were easy enough to create I tried that first. I’ll give this a try in the next few days.

Thanks!

[Lance (uklance)]

It makes sense to me that gradle needs a pom for “ToolboxCore:ToolboxCore:1.0” since gradle wants to find the transitive dependencies. Did you try “ToolboxCore:ToolboxCore:1.0@jar”?

[rstasko (Roger Stasko)]

Not yet. I ended-up changing my build to use the maven-publish plugin to generate the .pom and to post the updated jars to my maven repo. That led me to replace the ‘java’ and ‘application’ plugins with ‘java-library’ and ‘java-library-distribution’. Oh, and I fixed some gradle deprecation warnings too. Definately a more heavyweight approach as compared to the @jar trick but it saves me the trouble of manually syncing the maven repo.

[Lance (uklance)]

My build is picking up the latest binary jar and sources jar from my maven “repository” and I can now step through and set breakpoints on the libs in eclipse.

Great! Glad to hear

I ended-up changing my build to use the maven-publish plugin to generate the .pom

Even better! Now the pom should contain your artifact’s transitive dependencies. Much better than a minimal pom

it saves me the trouble of manually syncing the maven repo

I’m assuming this is just a hack for you to develop two dependent artifacts by yourself (a team of one) on your machine at home.

The normal “enterprise” solution to this problem is to host a private repository (eg artifactory or nexus) on your network and publish to the repository as part of your release (and possibly CI) process

[Lance (uklance)]

Have you considered a Composite Build?

[rstasko (Roger Stasko)]

You’re correct in that my work products are all managed by a “team of one”. However, some of the products are used by a group so the “team of one” is expected to become a team of a few, or a couple of teams of few. But the direction is not enterprise, it is open-source, or at least cooperatively-owned and managed.

[rstasko (Roger Stasko)]

Not yet – this is the first I’ve heard of composite builds. But at first blush, it doesn’t look like the right approach for anything I’m doing.

[Lance (uklance)]

this is the first I’ve heard of composite builds. But at first blush, it doesn’t look like the right approach

You are building ProjectA, publishing the artifact to a directory only to consume the dependency in ProjectB. This is exactly what a composite build is for. In a composite build the two builds are joined together as one and gradle is smart enough to use the local ProjectA artifact in ProjectB since group/artifact/version is the same (ie no publish step required)

the direction is not enterprise, it is open-source

Many (most?) open source projects publish to a repository. My usage of the word “enterprise” was probably incorrect. As you grow to more than one in your team, you should at least consider publishing to a repository that the team can share

[filchristou (Filippos Christou)]

If someone wants to stick with the flatDir simplicity, for me it worked just renaming the jar sources like:
$GRADLE_PROJECT/libs/library.jar
$GRADLE_PROJECT/libs/library-sources.jar

build.gradle:

repositories {
	flatDir {
		dirs 'libs'
	}
}

dependencies {
	implementation name: 'library'
}

[thofmann (Thomas Hofmann)]

Hi there,

I still would be interested to know why adding the source jar to the library does not work in the first place.
I have a similar need to add a custom source jar to one of the entries of the gradleclasspathcontainer.

My code looks like this:

eclipse {
    classpath {
        if(!tools.isJeeUtilityProject(project)) {
            containers 'org.eclipse.pde.core.requiredPlugins'
            minusConfigurations += [configurations.runtimeClasspath]
        }

        /* If not added here it will be appended at the end of .classpath but we want to customize it below */
        containers 'org.eclipse.buildship.core.gradleclasspathcontainer'

        file.whenMerged {
            /*
             * The following entries are needed in case this plugin wants to export types from these JARs in MANIFEST.MF
             */
            project.configurations.bundled.sort { it.name }.each { File jarFile ->
                def lib = new Library(fileReference(file("lib/${jarFile.getName()}".toString())))
                lib.exported = false
                lib.sourcePath = fileReference(file("lib/${jarFile.getName().take(jarFile.name.lastIndexOf('.'))}-sources.jar".toString()))
                println "lib.sourcePath $lib.sourcePath"
                println "lib.library $lib.library"
                entries += lib
            }
            project.configurations.bundledApi.sort { it.name }.each { File jarFile ->
                def lib = new Library(fileReference(file("lib/${jarFile.getName()}".toString())))
                lib.exported = true
                entries += lib
            }
            if(project.configurations.bundledApi.size() > 0) {
                def gradleclasspathcontainer = entries.find { it.path == 'org.eclipse.buildship.core.gradleclasspathcontainer' }
                gradleclasspathcontainer.exported = true
            }

            // more code here
    }
}

The result is this:

[thofmann (Thomas Hofmann)]

@Lance Do you have a suggestion for me?

[Lance (uklance)]

Yes, as I suggested earlier you should use the maven directory layout which has a convention for storing source jars with the binaries which IDE’s will use

See here

[thofmann (Thomas Hofmann)]

Thanks and according to this post How to use sources / javadoc of local compile dependency in Eclipse / Buildship - #7 by st_oehme what I am trying to do should be possible since Buildship 1.0.18.

@st_oehme would you mind to comment?

Thanks a lot!