Inconsistent behavior from SourceTask exclude?


(Matias Bjarland) #1

Background ----------------- This issue started with a custom plugin class which extends SourceTask and where I could not get the exclude patterns to work the way I expected them to. I have since then boiled the problem down to a sample project as per below. The project can be downloaded as a zip file here and executed from the directory created by unzipping by just issuing:

~> ./gradlew

The Problem ------------------ Now for the problem. The build file in the above zip first creates a sample set of test files under directory ‘files’. The generated sample files have the following directory structure:

$ tree -d files/
files/
├── one
│   ├── one
│   ├── three
│   └── two
├── three
│   ├── one
│   ├── three
│   └── two
└── two
    ├── one
    ├── three
    └── two
  12 directories

Each leaf dir has 21 files which gives us a total of 9 ∗ 21 = 189 files.

The build then executes a number of test tasks which all inherit from a custom task which extends SourceTask. The test tasks are configured with different source expressions and exclude patterns to try to get to grips with how the SourceTask exclude patterns work.

The build file:

defaultTasks 'run'
  ext.FILES = makeFiles() as List<File>
  task test1(type: MySourceTask) {
  source = FILES
  }
  task test2(type: MySourceTask) {
  source = FILES
    exclude '**/one/**'
 //no effect
}
  task test3(type: MySourceTask) {
  source = FILES
    exclude '**/one/'
   //no effect
}
  task test4(type: MySourceTask) {
  source = fileTree('files') {
    exclude '**/one/**' //works as expected
  }
}
  task test5(type: MySourceTask) {
  source = fileTree('files') {
    exclude '**/one/'
 //works as expected
  }
}
  task test6(type: MySourceTask) {
  source = FILES
    exclude '**/MANIFEST_8.MF'
//works as expected, there are 9 of these
}
  task test7(type: MySourceTask) {
  source = FILES
    exclude 'MANIFEST_8.MF'
   //works as expected, there are 9 of these
}
    task run(dependsOn: tasks.withType(MySourceTask))
  class MySourceTask extends SourceTask {
   @TaskAction
  def run() {
    println "
Found: ${source.files.size()} files"
  }
}
  task wrapper(type: Wrapper) {
  gradleVersion = '1.3'
  jarFile = '.gradle/wrapper/gradle-wrapper.jar'
}
  //make some sample files to play with
 List<File> makeFiles() {
  List<File> result = []
    def d = ['one','two','three']
  d.each { a ->
     d.each { b ->
       def parent = new File('files', "$a/$b")
      parent.mkdirs()
      0.upto(20) {
        def f = new File(parent, "MANIFEST_${it}.MF")
        f.text = it
        result << f
      }
    }
    }
      result
 }

the output from an execution of the above file is:

$ ./gradlew
   :test1
  Found: 189 files
:test2
  Found: 189 files
:test3
  Found: 189 files
:test4
  Found: 84 files
:test5
  Found: 84 files
:test6
  Found: 180 files
:test7
  Found: 180 files
:run
  BUILD SUCCESSFUL
  Total time: 0.617 secs

What I expected from the tasks was, in order:

  • test1 - return all 189 of the files created as we have no exclude patterns. Uses List<File> as the source. Works as expected. * test2 - return the 189 files minus all files with a parent dir on some level called ‘one’. Should leave us with 189 ∗ 2/3 ∗ 2/3 = 84 files. Uses the direct exclude method inherited from SourceTask. Uses List<File> as the source. Unexpected result. * test3 - same as test2, just a slight variation of the exclude pattern. Uses List<File> as the source. Unexpected result.

  • test4 - use the same exclude pattern as in test2 but apply it on a file tree which then becomes the source of the task. Works as expected. * test5 - same as test4 but with the pattern from test3. Works as expected. * test6 and test7 - use List<File> as the source. This shows that as long as we apply the exclude pattern on the file name and not intermediate directories, things seem to work as expected.

    So it seems that using the ‘exclude’ method on the source task is a bit broken when using a list of File objects as the source. If we on the other hand use a FileTree as the source things seem to work fine. With file trees (there is no test for this in the above, but I have tested this separately) the direct ‘exclude’ method calls also seem to work fine.

    So what is up with SourceTask and using a List<File> as the source? Also, if this for some arcane reason is by design, how would I go about converting an existing List<File> into a file tree so that a user of my plugin can write things like:

conventionalPluginTask.exclude "**/somedir/**"

(detelinyordanov) #2

I think the reason for this behavior is because each source file is being converted to a SingletonFileTree so at the end the source file tree is a composite tree containing a bunch of singleton file trees. Each pattern is then applied to the relative path of the SingletoneFileTree which happens to be just the file name rather than the full file path. I guess the reason it works fine when using a regular file tree as a source, is because the file tree is created with base directory and then the relative path of each file is not just the filename but the subpath from that base directory.


(Matias Bjarland) #3

That sounds in line with what I saw in my debug session. Thank you for figuring out the reason behind this behavior.

After looking some more at the code for SourceTask, (and as detelin pointed out) it looks like this essentially boils down to Project.files(…) with a List<File> argument returning a file tree which is not entirely functional.

Adding the following two tasks to the above build file:

ext.FILES = makeFiles() as List<File>
  task one << {
   println "FILES: " + files(FILES).asFileTree.matching { exclude '**/one/**' }.files.size()
}
  task two << {
   println "FILES: " + fileTree('files').matching { exclude '**/one/**'}.files.size()
}

gives:

$ ./gradlew one two
:one
FILES: 189
:two
FILES: 84
  BUILD SUCCESSFUL

i.e. this exhibits the same symptoms as my SourceTask extension. The end result where the ‘matching’ method works in totally different ways depending on what internal implementation of FileTree you happen to end up with does not seem entirely ok to me. One would assume that if you get a FileTree instance, you can treat it the same independent of the underlying implementation.

All that being said, how would I go about using a SourceTask extension if I have my sources in a List<File> and want to be able to use exclude patterns? Is there some other kung fu we could apply here?

The files in the list do not have a common base directory and creating a fake file tree with a basedir of ‘/’ leaves me feeling a bit dirty and hackish.


(Peter Niederwieser) #4

‘SourceTask’ is geared towards working with source directories rather than source files. If you feed it with individual source files, you’ll have to use one of the ‘FileCollection.filter’ methods and can’t use Ant-style includes/excludes. If this doesn’t fit your bill, you can write your own replacement for ‘SourceTask’. The main challenge is that ‘include/exclude’ is only supported for ‘FileTree’, but not for ‘FileCollection’.