Use FileCollection APIs in settings.gradle

Lance · August 31, 2017, 10:02am

I’d like to use the FileCollection APIs in a settings.gradle file. Obviously the Project instances aren’t accessible yet so I can’t call Project.files(...) or Project.fileTree(...).

Is there a way to use FileCollection APIs from settings.gradle or do I have to make do with java.io.*?

eljobe · August 31, 2017, 10:37am

I believe your analysis is correct. There doesn’t appear to be a way to get an instance of a FileCollection while evaluating the settings.gradle file. Out of curiosity, what are you trying to do that makes you want to use the FileCollection API in your settings.gradle file? Maybe there is a missing feature we can identify and deal with in a straight-forward manner?

Lance · August 31, 2017, 11:44am

I want to dynamically determine the list of subprojects, eg find folders with build.gradle under a root folder. Simple to do with a FileTree, more verbose with java.io.*

Perhaps Settings.files(...) and Settings.fileTree(...) would be nice additions to the API?

eljobe · September 21, 2017, 1:49pm

@Lance, sorry for the delay in reply. At this point, I think we’re talking about an actual feature request for Gradle (as opposed to general discussion and help requests.) Would you mind formalizing this request by opening an issue on the https://github.com/gradle/gradle repository?

st_oehme · September 21, 2017, 2:13pm

You can use files() and friends in settings.gradle just like in a project build script. The basedir will be the settings directory.

eljobe · September 21, 2017, 2:58pm

My bad. @st_oehme, is, of course, accurate.

I wonder if we have a documentation bug to file. I think I fell into the same trap as @Lance.

I read these documents:
https://docs.gradle.org/current/userguide/working_with_files.html
https://docs.gradle.org/current/dsl/org.gradle.api.initialization.Settings.html
https://docs.gradle.org/current/javadoc/org/gradle/api/initialization/Settings.html
https://docs.gradle.org/current/javadoc/org/gradle/api/plugins/PluginAware.html

And, missing any documentation about a Settings.files() analogous to the Project.files(), I assumed that it wouldn’t work. In fact, you cannot call Settings.files() but have to just use files() directly. I haven’t yet figured out where this function comes from, but I’ll see if I can figure it out. Then, I’ll open a documentation bug.

eljobe · September 21, 2017, 3:32pm

I found it.
So, all gradle build scripts end up being applied using a ScriptTarget inside DefaultScriptPluginFactory:

github.com

gradle/gradle/blob/32963bec3674809e460ae67aaf50c00a6b7f5ed7/subprojects/core/src/main/java/org/gradle/configuration/DefaultScriptPluginFactory.java#L218-L237


private ScriptTarget wrap(Object target, boolean isInitialPass) {
    if (target instanceof ProjectInternal && topLevelScript) {
        // Only use this for top level project scripts
        return new ProjectScriptTarget((ProjectInternal) target);
    }
    if (target instanceof GradleInternal && topLevelScript) {
        // Only use this for top level init scripts
        return new InitScriptTarget((GradleInternal) target);
    }
    if (target instanceof SettingsInternal && topLevelScript) {
        // Only use this for top level settings scripts
        if (isInitialPass) {
            return new InitialPassSettingScriptTarget((SettingsInternal) target);
        } else {
            return new SettingScriptTarget((SettingsInternal) target);
        }
    } else {
        return new DefaultScriptTarget(target);
    }
}

The ScriptTarget for the settings.gradle file wraps a SettingsInternal typed target, so you can see from the logic above, it’s going to create either a InitialPassSettingsScriptTarget or a SettingsScriptTarget depending on which execution pass Gradle is on. InitialPassSettingsScriptTarget extends from SettingsScriptTarget:

github.com

gradle/gradle/blob/32963bec3674809e460ae67aaf50c00a6b7f5ed7/subprojects/core/src/main/java/org/gradle/configuration/InitialPassSettingScriptTarget.java#L21


* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/


package org.gradle.configuration;


import org.gradle.api.internal.SettingsInternal;


public class InitialPassSettingScriptTarget extends SettingScriptTarget {
   public InitialPassSettingScriptTarget(SettingsInternal target) {
       super(target);
   }


   @Override
   public boolean getSupportsPluginManagementBlock() {
       return true;
   }
}

Which tells the DefaultScriptPluginFactory to construct a SettingsScript class to represent the target by returning that class from its getScriptClass() method:

github.com

gradle/gradle/blob/32963bec3674809e460ae67aaf50c00a6b7f5ed7/subprojects/core/src/main/java/org/gradle/configuration/SettingScriptTarget.java#L33-L36


@Override
public Class<? extends BasicScript> getScriptClass() {
    return SettingsScript.class;
}

when the DefaultScriptPluginFactory asks for it here:

github.com

gradle/gradle/blob/32963bec3674809e460ae67aaf50c00a6b7f5ed7/subprojects/core/src/main/java/org/gradle/configuration/DefaultScriptPluginFactory.java#L171


services.add(TextResourceLoader.class, textResourceLoader);
services.add(StreamHasher.class, streamHasher);
services.add(FileHasher.class, fileHasher);


final ScriptTarget initialPassScriptTarget = initialPassTarget(target);


ScriptCompiler compiler = scriptCompilerFactory.createCompiler(scriptSource);


// Pass 1, extract plugin requests and plugin repositories and execute buildscript {}, ignoring (i.e. not even compiling) anything else


Class<? extends BasicScript> scriptType = initialPassScriptTarget.getScriptClass();
InitialPassStatementTransformer initialPassStatementTransformer = new InitialPassStatementTransformer(scriptSource, initialPassScriptTarget, documentationRegistry);
SubsetScriptTransformer initialTransformer = new SubsetScriptTransformer(initialPassStatementTransformer);
String id = INTERNER.intern("cp_" + initialPassScriptTarget.getId());
CompileOperation<PluginRequests> initialOperation = new FactoryBackedCompileOperation<PluginRequests>(id, initialTransformer, initialPassStatementTransformer, pluginRequestsSerializer);


ScriptRunner<? extends BasicScript, PluginRequests> initialRunner = compiler.compile(scriptType, initialOperation, baseScope.getExportClassLoader(), Actions.doNothing());
initialRunner.run(target, services);


PluginRequests pluginRequests = initialRunner.getData();
PluginManagerInternal pluginManager = initialPassScriptTarget.getPluginManager();

and here:

github.com

gradle/gradle/blob/32963bec3674809e460ae67aaf50c00a6b7f5ed7/subprojects/core/src/main/java/org/gradle/configuration/DefaultScriptPluginFactory.java#L185


CompileOperation<PluginRequests> initialOperation = new FactoryBackedCompileOperation<PluginRequests>(id, initialTransformer, initialPassStatementTransformer, pluginRequestsSerializer);


ScriptRunner<? extends BasicScript, PluginRequests> initialRunner = compiler.compile(scriptType, initialOperation, baseScope.getExportClassLoader(), Actions.doNothing());
initialRunner.run(target, services);


PluginRequests pluginRequests = initialRunner.getData();
PluginManagerInternal pluginManager = initialPassScriptTarget.getPluginManager();
pluginRequestApplicator.applyPlugins(pluginRequests, scriptHandler, pluginManager, targetScope);


// Pass 2, compile everything except buildscript {}, pluginRepositories{}, and plugin requests, then run
final ScriptTarget scriptTarget = secondPassTarget(target);
scriptType = scriptTarget.getScriptClass();


BuildScriptTransformer buildScriptTransformer = new BuildScriptTransformer(scriptSource, scriptTarget);
String operationId = scriptTarget.getId();
CompileOperation<BuildScriptData> operation = new FactoryBackedCompileOperation<BuildScriptData>(operationId, buildScriptTransformer, buildScriptTransformer, buildScriptDataSerializer);


final ScriptRunner<? extends BasicScript, BuildScriptData> runner = compiler.compile(scriptType, operation, targetScope.getLocalClassLoader(), ClosureCreationInterceptingVerifier.INSTANCE);
if (scriptTarget.getSupportsMethodInheritance() && runner.getHasMethods()) {
    scriptTarget.attachScript(runner.getScript());
}

The scriptType is used to create ScriptRunner instances based on the scriptType on lines 177 and 192 above.

The ScriptRunner instances end up instantiating instances of the scriptType to run the scripts. Because the SettingScript class extends from DefaultScript …:

github.com

gradle/gradle/blob/32963bec3674809e460ae67aaf50c00a6b7f5ed7/subprojects/core/src/main/java/org/gradle/initialization/SettingsScript.java#L20


* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.gradle.initialization;


import org.gradle.groovy.scripts.DefaultScript;


public abstract class SettingsScript extends DefaultScript {
   public String toString() {
       return getScriptTarget().toString();
   }
}

… which extends from BasicScript …:

github.com

gradle/gradle/blob/32963bec3674809e460ae67aaf50c00a6b7f5ed7/subprojects/core/src/main/java/org/gradle/groovy/scripts/DefaultScript.java#L63


import org.gradle.process.ExecResult;
import org.gradle.process.ExecSpec;
import org.gradle.process.JavaExecSpec;
import org.gradle.util.ConfigureUtil;


import java.io.File;
import java.net.URI;
import java.util.Map;
import java.util.concurrent.Callable;


public abstract class DefaultScript extends BasicScript {
private static final Logger LOGGER = Logging.getLogger(Script.class);


private FileOperations fileOperations;
private ProcessOperations processOperations;
private ProviderFactory providerFactory;
private LoggingManager loggingManager;


public ServiceRegistry __scriptServices;


@Override

… which implements FileOperations:

github.com

gradle/gradle/blob/master/subprojects/core/src/main/java/org/gradle/groovy/scripts/BasicScript.java#L36


import org.gradle.internal.logging.StandardOutputCapture;
import org.gradle.internal.metaobject.AbstractDynamicObject;
import org.gradle.internal.metaobject.BeanDynamicObject;
import org.gradle.internal.metaobject.DynamicInvokeResult;
import org.gradle.internal.metaobject.DynamicObject;
import org.gradle.internal.service.ServiceRegistry;


import java.io.PrintStream;
import java.util.Map;


public abstract class BasicScript extends org.gradle.groovy.scripts.Script implements org.gradle.api.Script, FileOperations, ProcessOperations, DynamicObjectAware {
private StandardOutputCapture standardOutputCapture;
private Object target;
private ScriptDynamicObject dynamicObject = new ScriptDynamicObject(this);


public void init(Object target, ServiceRegistry services) {
    standardOutputCapture = services.get(StandardOutputCapture.class);
    setScriptTarget(target);
}


public Object getScriptTarget() {

… and, because FileOperations defines the files() and file() methods (among others) …:

github.com

gradle/gradle/blob/32963bec3674809e460ae67aaf50c00a6b7f5ed7/subprojects/core/src/main/java/org/gradle/api/internal/file/FileOperations.java#L33-L43


File file(Object path);


File file(Object path, PathValidation validation);


URI uri(Object path);


FileResolver getFileResolver();


String relativePath(Object path);


ConfigurableFileCollection files(Object... paths);

… you can conclude that you are allowed to use file() and files() directly in your settings.gradle file.

Now, arguably, you are not allowed to use them because the whole FileOperations interface is in an internal package which means Gradle reserves the right to change these implementation details without a deprecation period.

And, it would probably be hard for a user of Gradle to discover how all of this is tied together without walking through the code I just tried to explain above. But, at least we know one way you could find it out.

st_oehme · September 21, 2017, 3:36pm

They come from SettingsScript extends DefaultScript. These are not part of the public API actually, just some magic we weave into Groovy scripts for convenience.

I think it makes sense to add a statically typed API, but we shouldn’t add all those overloaded methods to Settings. In fact we want to get rid of them on Project in the long run. Instead we could provide something like Settings.getFileOperations() (for lack of a better name).

Lance · September 21, 2017, 5:16pm

Ah, I had no idea about the weaving. If it’s not in the javadocs it’s hard to discover. It would be nice to access from java too via Plugin<Settings>

Too many times I’ve been burnt by groovy thinking it’d save time when it ultimately end up wasting my time with quirks. I prefer writing gradle plugins in java these days