Can't extract tar.gz defined as a dependency

Douglas_Borg · August 14, 2012, 12:02am

I am trying to do something similiar to what is discussed in this thread: [gradle-user] Re: extracting tar contents via dependencies

However, (as noted in the last message in that thread) I cannot extract files from a tar.gz defined as a dependency. This is even when explicitly defining the resource type.

The code I have fails with the error:

Download https://artifactory.company.com/artifactory/repo/com/flex/flex/4.6.0/flex-4.6.0.tar.gz > :setupSDKs > > FAILURE: Build failed with an exception. > > * What went wrong: > Unable to expand TAR ‘/home/me/.gradle/caches/artifacts-14/filestore/com.adobe/flex/4.6.0/tar.gz/ffc1666ac6feb7e1fdf93bf5d5dd660b3f78b91a/flex-4.6.0.tar.gz’ > The tar might be corrupted or it is compressed in an unexpected way. > By default the tar tree tries to guess the compression based on the file extension. > If you need to specify the compression explicitly please refer to the DSL reference.

dependencies {
  flexSDK (
    [group: 'com.adobe', name: 'flex', version: '4.6.0', ext: 'tar.gz', configuration:"compile"],
  )
}
  task setupSDKs(type:Copy){
  configurations.flexSDK.findAll{it.name.endsWith('.tar.gz')}.each {
    from tarTree(resources.gzip(it))
  }
  into '~/.flexSDKs'
}

I am specifying the compression explicitly so I am not sure what else I can do or why it is failing in this case.

Douglas_Borg · August 14, 2012, 4:12pm

I have now tried this a few different ways with a couple different versions of the artifact and I do not get repeatable results. I have gotten the tarball to extract a couple of times, but after clearing my cache and the target directory and trying it again, I get the same error.

Here is the current state of things:

configurations {
  flexSDK
}
    dependencies {
  flexSDK (
    [group: 'com.adobe', name: 'flex', version: '4.6.0', ext: 'tar.bz2'],
  )
}
    task setupSDKs(type:Copy){
  configurations.flexSDK.findAll{it.name.endsWith('.tar.bz2')}.each {
    from tarTree(resources.bzip2(it))
  }
  into "${project.gradle.gradleUserHomeDir}/flexSDKs"
}
    task setupManual(type:Copy){
  from tarTree(resources.gzip("/home/dborg/flex-4.6.0.tar.gz"))
  into "${project.gradle.gradleUserHomeDir}/flexSDKs"
}

Even the setupManual task I created to test just expanding a hard-coded file with a full path does not work reliably.

luke_daley · August 15, 2012, 9:48am

So the file is the same binary content and intermittently fails to extract?

Could you please run with ‘-s’ and post the stacktrace you get.

Douglas_Borg · August 15, 2012, 5:18pm

Hi Luke,

Yes, I am running into trouble running the task again using the same binary file as the input. It does not seem to matter if I set the ‘from’ source from a dependency definition or by hard-coding a file path.

After running with -s and getting the stack trace I think I have figured out what the problem is. I must have missed something earlier when I ran it with the -d flag. I can fix the issue reliably by deleting the build/tmp/expandedArchives folder AND the target folder.

I think the copy task may be running into trouble because I am expanding the flexSDK archive outside of the root project folder and the up-to-date check is not skipping the task execution. It seems to be running into an issue over-writing or accessing a file in the expandedArchives tmp folder I am not sure if this is during the actual copy or if it is happening when it is checking the up-to-date status.

I would have hoped that the up-to-date check on the copy task would understand that the archive and the output folder have not changed and not run the task at all the second time. Is there a (non-hacky) way I can get the up-to-date check to work even though I am expanding the archive outside the project’s root folder? Is that even the problem?

For now, I am going to try setting the inputs and outputs of the task explicitly or maybe even adding a check to skip the task entirely if the target folder (something like: ${project.gradle.gradleUserHomeDir}/flexSDKs/flex-${sdkVersion}) exists.

Any additional insight you can give me into what is going on would be much appreciated. Thanks!

Here is the stacktrace (running ‘gradle sS -s’): *** org.gradle.api.GradleException: Unable to expand TAR ‘/home/dborg/.gradle/caches/artifacts-13/filestore/com.adobe/flex/4.6.0/tar.bz2/befa578e5e4f70a534530668519e9280b09bdd8b/flex-4.6.0.tar.bz2’

The tar might be corrupted or it is compressed in an unexpected way.

By default the tar tree tries to guess the compression based on the file extension.

If you need to specify the compression explicitly please refer to the DSL reference.

at org.gradle.api.internal.file.archive.TarFileTree.visit(TarFileTree.java:83)

at org.gradle.api.internal.file.collections.FileTreeAdapter.visit(FileTreeAdapter.java:96)

at org.gradle.api.internal.file.AbstractFileTree$FilteredFileTree.visit(AbstractFileTree.java:136)

at org.gradle.api.internal.file.AbstractFileTree.getFiles(AbstractFileTree.java:37)

at org.gradle.api.internal.file.CompositeFileCollection.getFiles(CompositeFileCollection.java:39)

at org.gradle.api.internal.file.AbstractFileCollection.iterator(AbstractFileCollection.java:60)

at org.gradle.api.internal.changedetection.DefaultFileSnapshotter.snapshot(DefaultFileSnapshotter.java:42)

at org.gradle.api.internal.changedetection.InputFilesChangedUpToDateRule.create(InputFilesChangedUpToDateRule.java:35)

at org.gradle.api.internal.changedetection.CompositeUpToDateRule.create(CompositeUpToDateRule.java:35)

at org.gradle.api.internal.changedetection.DefaultTaskArtifactStateRepository$HistoricExecution.calcCurrentState(DefaultTaskArtifactStateRepository.java:80)

at org.gradle.api.internal.changedetection.DefaultTaskArtifactStateRepository$HistoricExecution.isUpToDate(DefaultTaskArtifactStateRepository.java:88)

at org.gradle.api.internal.changedetection.DefaultTaskArtifactStateRepository$TaskArtifactStateImpl.isUpToDate(DefaultTaskArtifactStateRepository.java:128)

at org.gradle.api.internal.changedetection.ShortCircuitTaskArtifactStateRepository$ShortCircuitArtifactState.isUpToDate(ShortCircuitTaskArtifactStateRepository.java:77)

at org.gradle.api.internal.changedetection.FileCacheBroadcastTaskArtifactStateRepository$1.isUpToDate(FileCacheBroadcastTaskArtifactStateRepository.java:37)

at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:44)

at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:57)

at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:41)

at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:51)

at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:52)

at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:42)

at org.gradle.api.internal.AbstractTask.executeWithoutThrowingTaskFailure(AbstractTask.java:247)

at org.gradle.execution.DefaultTaskGraphExecuter.executeTask(DefaultTaskGraphExecuter.java:192)

at org.gradle.execution.DefaultTaskGraphExecuter.doExecute(DefaultTaskGraphExecuter.java:177)

at org.gradle.execution.DefaultTaskGraphExecuter.execute(DefaultTaskGraphExecuter.java:83)

at org.gradle.execution.SelectedTaskExecutionAction.execute(SelectedTaskExecutionAction.java:36)

at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:61)

at org.gradle.execution.DefaultBuildExecuter.access$200(DefaultBuildExecuter.java:23)

at org.gradle.execution.DefaultBuildExecuter$2.proceed(DefaultBuildExecuter.java:67)

at org.gradle.api.internal.changedetection.TaskCacheLockHandlingBuildExecuter$1.run(TaskCacheLockHandlingBuildExecuter.java:31)

at org.gradle.cache.internal.DefaultCacheAccess$1.create(DefaultCacheAccess.java:111)

at org.gradle.cache.internal.DefaultCacheAccess.useCache(DefaultCacheAccess.java:126)

at org.gradle.cache.internal.DefaultCacheAccess.useCache(DefaultCacheAccess.java:109)

at org.gradle.cache.internal.DefaultPersistentDirectoryStore.useCache(DefaultPersistentDirectoryStore.java:103)

at org.gradle.api.internal.changedetection.DefaultTaskArtifactStateCacheAccess.useCache(DefaultTaskArtifactStateCacheAccess.java:79)

at org.gradle.api.internal.changedetection.TaskCacheLockHandlingBuildExecuter.execute(TaskCacheLockHandlingBuildExecuter.java:29)

at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:61)

at org.gradle.execution.DefaultBuildExecuter.access$200(DefaultBuildExecuter.java:23)

at org.gradle.execution.DefaultBuildExecuter$2.proceed(DefaultBuildExecuter.java:67)

at org.gradle.execution.DryRunBuildExecutionAction.execute(DryRunBuildExecutionAction.java:32)

at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:61)

at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:54)

at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:155)

at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:110)

at org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:78)

at org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:38)

at org.gradle.launcher.daemon.protocol.Build.run(Build.java:68)

at org.gradle.launcher.daemon.protocol.Build.run(Build.java:64)

at org.gradle.launcher.daemon.server.exec.ExecuteBuild.doBuild(ExecuteBuild.java:45)

at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:34)

at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)

at org.gradle.launcher.daemon.server.exec.WatchForDisconnection.execute(WatchForDisconnection.java:45)

at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)

at org.gradle.launcher.daemon.server.exec.ResetDeprecationLogger.execute(ResetDeprecationLogger.java:24)

at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)

at org.gradle.launcher.daemon.server.exec.StartStopIfBuildAndStop.execute(StartStopIfBuildAndStop.java:28)

at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)

at org.gradle.launcher.daemon.server.exec.ReturnResult.execute(ReturnResult.java:34)

at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)

at org.gradle.launcher.daemon.server.exec.ForwardClientInput$4.call(ForwardClientInput.java:116)

at org.gradle.launcher.daemon.server.exec.ForwardClientInput$4.call(ForwardClientInput.java:114)

at org.gradle.util.Swapper.swap(Swapper.java:38)

at org.gradle.launcher.daemon.server.exec.ForwardClientInput.execute(ForwardClientInput.java:114)

at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)

at org.gradle.launcher.daemon.server.exec.LogToClient.doBuild(LogToClient.java:60)

at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:34)

at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)

at org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment.doBuild(EstablishBuildEnvironment.java:61)

at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:34)

at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)

at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy.doBuild(StartBuildOrRespondWithBusy.java:49)

at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:34)

at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)

at org.gradle.launcher.daemon.server.exec.HandleStop.execute(HandleStop.java:34)

at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)

at org.gradle.launcher.daemon.server.exec.CatchAndForwardDaemonFailure.execute(CatchAndForwardDaemonFailure.java:32)

at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)

at org.gradle.launcher.daemon.server.exec.HandleClientDisconnectBeforeSendingCommand.execute(HandleClientDisconnectBeforeSendingCommand.java:21)

at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)

at org.gradle.launcher.daemon.server.exec.StopConnectionAfterExecution.execute(StopConnectionAfterExecution.java:27)

at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)

at org.gradle.launcher.daemon.server.exec.DefaultDaemonCommandExecuter.executeCommand(DefaultDaemonCommandExecuter.java:55)

at org.gradle.launcher.daemon.server.Daemon$1$1.run(Daemon.java:123)

at org.gradle.messaging.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:66) Caused by: org.gradle.api.GradleException: Could not copy tar entry /home/dborg/.gradle/caches/artifacts-13/filestore/com.adobe/flex/4.6.0/tar.bz2/befa578e5e4f70a534530668519e9280b09bdd8b/flex-4.6.0.tar.bz2!flex-4.6.0.23201/frameworks/projects/air/Licensing/Android/AndroidLicensingLib/src/extension.xml to ‘/home/dborg/ReadyTalk/av-flexlib/build/tmp/expandedArchives/flex-4.6.0.tar.bz2_5ach6q8l75s55ahl1ulr8ea1si/flex-4.6.0.23201/frameworks/projects/air/Licensing/Android/AndroidLicensingLib/src/extension.xml’.

at org.gradle.api.internal.file.AbstractFileTreeElement.copyTo(AbstractFileTreeElement.java:68)

at org.gradle.api.internal.file.archive.TarFileTree$DetailsImpl.getFile(TarFileTree.java:125)

at org.gradle.api.internal.file.AbstractFileTree$1.visitFile(AbstractFileTree.java:39)

at org.gradle.api.internal.file.AbstractFileTree$FilteredFileTree$1.visitFile(AbstractFileTree.java:145)

at org.gradle.api.internal.file.archive.TarFileTree.visitImpl(TarFileTree.java:95)

at org.gradle.api.internal.file.archive.TarFileTree.visit(TarFileTree.java:74)

… 82 more Caused by: java.io.FileNotFoundException: /home/dborg/ReadyTalk/av-flexlib/build/tmp/expandedArchives/flex-4.6.0.tar.bz2_5ach6q8l75s55ahl1ulr8ea1si/flex-4.6.0.23201/frameworks/projects/air/Licensing/Android/AndroidLicensingLib/src/extension.xml (Permission denied)

at org.gradle.api.internal.file.AbstractFileTreeElement.copyFile(AbstractFileTreeElement.java:80)

at org.gradle.api.internal.file.AbstractFileTreeElement.copyTo(AbstractFileTreeElement.java:63)

… 87 more ***

luke_daley · August 17, 2012, 10:47am

I don’t quite understand what is going on sorry.

Any chance you could provide a script that I can use to reproduce the problem?

Douglas_Borg · August 17, 2012, 4:51pm

HI Luke,

No worries. I tried to reproduce the issue with a smaller tar.gz and could not. It seems to be something particular to the Flex sdk file structure in a tar format. I have abandoned that approach and I am using a zip file as the format with the archive. I’ll let you know how that works out.

Douglas_Borg · August 21, 2012, 4:05pm

The official zip archive from Adobe causes a similar issue: http://download.macromedia.com/pub/flex/sdk/flex_sdk_4.6.zip

Try setting up a copy task to extract this file and see if you can reproduce the issue I am getting:

$ gradle sS -s >:setupSDK

FAILURE: Build failed with an exception.

What went wrong: >Could not expand ZIP ‘/home/dborg/.gradle/caches/artifacts->13/filestore/com.adobe/flex/4.6/zip/7e66e84bef09e8c2c3183acf548c41d8114516ca/flex_sdk-4.6.zip’.

Try: >Run with --info or --debug option to get more log output.

Exception is: >org.gradle.api.GradleException: Could not expand ZIP ‘/home/dborg/.gradle/caches/artifacts-13/filestore/com.adobe/flex/4.6/zip/7e66e84bef09e8c2c3183acf548c41d8114516ca/flex_sdk-4.6.zip’.

at org.gradle.api.internal.file.archive.ZipFileTree.visit(ZipFileTree.java:97)