Gradle having problems with large folders as task inputs/outputs

I’m trying to integrate an Angular project into my Gradle build. Of course, Angular means node_modules, so the “large size” is both in volume (~250 MB, which would be huge for a Java project folder, but that’s how it is), and a lot of files (~40k). This is no more than the project I got from “ng new”. I am using pnpm to counter some of bloat of having multiple Angular projects, but that only helps with disk space.

The project in question is available here: https://github.com/Clashsoft/angular-gradle-demo

Now to the problem:

Gradle seems to have quite some problems with a folder of that size. I configured a task that uses it as inputs and outputs, so I guess Gradle attempts to scan all the files for changes and keeps track of hashes and such.

When I run “gradle build”, the build passes, but it takes a long time, and at least the installDependencies task is never up-to-date. I tried --scan, but that does not always work (sometimes I get a 404). Here is a scan that worked: https://scans.gradle.com/s/6prvg3qjv4z3w. Also, I had OutOfMemory with the default max memory and had to increase it significantly.

What am I doing wrong that causes this horrible performance? Has anyone worked with Angular or Node.js in general in a Gradle project before and faced similar issues?

For quick reference, here is build.gradle:

plugins {
	id 'java'
	id 'application'
}

repositories {
	jcenter()
}

dependencies {
	implementation 'com.google.guava:guava:28.1-jre'

	testImplementation 'junit:junit:4.12'

	// https://mvnrepository.com/artifact/com.sparkjava/spark-core
	compile group: 'com.sparkjava', name: 'spark-core', version: '2.9.1'
}

application {
	mainClassName = 'de.clashsoft.demo.angulargradle.App'
}

// --------------- Angular ---------------

// adapted from https://blog.softwareforen.de/2018/10/integration-von-spring-und-angular-mit-gradle/

// User Configuration

def appDir = "$projectDir/angular/angular-gradle-demo"
def outputDir = "$appDir/dist/angular-gradle-demo"

def packageManager = guessPackageManager()
def packageManagerArgs = [ 'install', '--shamefully-hoist' ]

// Gradle Glue Code

sourceSets.main.resources.srcDir outputDir

processResources.dependsOn 'buildAngular'

static def isWindows() {
	return System.getProperty('os.name').toUpperCase().contains('WINDOWS')
}

static def guessPackageManager() {
	return readPackageManagerFromNgConfig() + (isWindows() ? '.cmd' : '')
}

static def readPackageManagerFromNgConfig() {
	def process = 'ng config cli.packageManager'.execute()
	def local = process.text
	if (process.exitValue() == 0) {
		return local.trim()
	}
	return 'ng config -g cli.packageManager'.execute().text.trim()
}

tasks.register('buildAngular', Exec) {
	it.group = BasePlugin.BUILD_GROUP
	it.dependsOn 'installAngularDependencies'

	it.workingDir = appDir
	it.inputs.dir appDir
	it.outputs.dir "$appDir/dist"

	if (isWindows()) {
		it.commandLine 'ng.cmd', 'build'
	}
	else {
		it.commandLine 'ng', 'build'
	}
}

tasks.register('installAngularDependencies', Exec) {
	it.group = BasePlugin.BUILD_GROUP

	it.workingDir = appDir

	def modulesDir = "$appDir/node_modules"
	def files = [
			"$appDir/package.json",
			// lock files:
			"$appDir/package-lock.json",
			"$appDir/yarn.lock",
			"$appDir/pnpm-lock.yaml",
	]

	mkdir(modulesDir)

	it.inputs.files(files)
	it.inputs.dir(modulesDir)
	it.outputs.files(files)
	it.outputs.dir(modulesDir)

	it.commandLine(packageManager, *packageManagerArgs)
}

I fixed it by removing this part of build.gradle:

	def modulesDir = "$appDir/node_modules"
	def files = [
			"$appDir/package.json",
			// lock files:
			"$appDir/package-lock.json",
			"$appDir/yarn.lock",
			"$appDir/pnpm-lock.yaml",
	]

	mkdir(modulesDir)

	it.inputs.files(files)
	it.inputs.dir(modulesDir)
	it.outputs.files(files)
	it.outputs.dir(modulesDir)

The package manager is able to figure out when it needs to do no work. It wastes a few seconds but thats better than the minutes wasted by Gradle checking for changes to the node_modules contents.

In addition, I was able to improve the runtime of the buildAngular task by excluding dist and node_modules from the inputs:

	it.inputs.files(fileTree(appDir).exclude('dist', 'node_modules'))
	// instead of: it.inputs.dir appDir