Custom build task does not use pluginManagement

I am trying to create a Gradle script to iterate all sub-directories and run gradle clean in each directory where a build.gradle is found. I have been able to get it mostly running, but I get an error with plugins not found. The reason seems to be because the settings.gradle in each folder is not used.
My task looks like this:

task cleanAll {
	int count = 1
	new File('.').traverse(type:, nameFilter: ~/build.gradle/) { buildFile ->
		if(buildFile.parent != '.') {
			def tempTask = tasks.create(name: "task_$count", type: GradleBuild)
			tempTask.buildFile = "$buildFile"
			tempTask.tasks = ['clean']
			count = count + 1

I have also tried setting the settings/init files on each task using the following:

Is there a way to set to make this work?