Download files with progress information: first release of the gradle-download-task plugin available!


(Michel Krämer) #1

Hi folks!

I just wanted to let you know that the first version of the gradle-download-task plugin has been released: https://github.com/michel-kraemer/gradle-download-task

This plugin contributes a download task that shows progress information just like Gradle does when it fetches artifacts from a repository. This is a feature I was missing for a long time, so I decided to write a plugin.

You can use it like this:

task downloadFile(type: Download) {
    src 'http://www.example.com/file.ext'
    dest buildDir
}

Or like this:

download {
    src 'http://www.example.com/file.ext'
    dest buildDir
}

The output will look like this:

Download http://www.example.com/file.ext
> Building > :downloadFile > 320 KB/6,78 MB downloaded

For a complete description have a look at the README file in my GitHub repository.

I hope you like it! Any feedback is appreciated!

Cheers, Michel


(spencer_allain) #2

Just a quick attempt at using it and it seems like a handy little plugin.

I can grab http://www.example.com/index.html and other things fine, but trying to use the example exactly as written results in:

Caused by: java.io.FileNotFoundException: http://www.example.com/file.ext
        at de.undercouch.gradle.tasks.download.DownloadAction.execute(DownloadAction.java:124)
        at de.undercouch.gradle.tasks.download.Download.download(Download.java:44)
        at org.gradle.api.internal.BeanDynamicObject$MetaClassAdapter.invokeMethod(BeanDynamicObject.java:248)
        at org.gradle.api.internal.BeanDynamicObject.invokeMethod(BeanDynamicObject.java:136)
        at org.gradle.api.internal.CompositeDynamicObject.invokeMethod(CompositeDynamicObject.java:147)
        at de.undercouch.gradle.tasks.download.Download_Decorated.invokeMethod(Unknown Source)
        at org.gradle.api.internal.BeanDynamicObject$MetaClassAdapter.invokeMethod(BeanDynamicObject.java:248)
        at org.gradle.api.internal.BeanDynamicObject.invokeMethod(BeanDynamicObject.java:136)
        at org.gradle.api.internal.CompositeDynamicObject.invokeMethod(CompositeDynamicObject.java:147)
        at de.undercouch.gradle.tasks.download.Download_Decorated.invokeMethod(Unknown Source)
        at org.gradle.util.ReflectionUtil.invoke(ReflectionUtil.groovy:23)
        at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$StandardTaskAction.doExecute(AnnotationProcessingTaskFactory.java:220)
        at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$StandardTaskAction.execute(AnnotationProcessingTaskFactory.java:213)
        at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$StandardTaskAction.execute(AnnotationProcessingTaskFactory.java:202)
        at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:530)
        at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:513)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:80)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:61)
        ... 53 more
Caused by: java.io.FileNotFoundException: http://www.example.com/file.ext
        at de.undercouch.gradle.tasks.download.DownloadAction.openConnection(DownloadAction.java:187)
        at de.undercouch.gradle.tasks.download.DownloadAction.execute(DownloadAction.java:112)
        ... 70 more

-Spencer


(Michel Krämer) #3

Hi Spencer!

Thanks for testing the plugin. Actually I didn’t know that http://www.example.com/index.html is indeed a valid URL to an existing file. I always thought that example.com always returns 404 no matter what you request :slight_smile:

I fixed it in the README file. It seems I cannot edit my previous post. Whatever.

Thanks for the hint!

Michel


(spencer_allain) #4

BTW, forgot to mention this, but if you ever need to edit one of your previous posts, you simply hover over the post itself (the post background should turn a yellowish color), and in the top right-hand corner of the post area you should see the words “edit” and “delete” with a pencil and a trash can next to them respectively. You should be able to select either the word or the icon for the appropriate action.

-Spencer


(Michel Krämer) #5

Thanks. I saw this already when I initially created the post. But it seems as soon as there’s an answer you cannot edit the post anymore. Hovering over the post won’t change the background and the icons won’t appear. It might be different for you: I suppose you’re a moderator or administrator or so.

However. Thanks anyways. I think it should be clear how to use the plugin from the README :slight_smile:

Cheers, Michel


(Guy Rooms) #6

Nice addition indeed, any idea if/when it will be integrated in a gradle release ?


(Michel Krämer) #7

Hi!

This is not up to me to decide. I would indeed be very happy to see this integrated in Gradle. Spencer, what do you think?

Cheers, Michel


(spencer_allain) #8

Being a regular user as well, I have no more influence as to what gets pulled into core Gradle than you do, but I’ll pass along my thoughts.

The most effective way to get a plugin into the core distribution is to fork the current trunk, place the code into the appropriate subproject, generate some integration tests to verify that it does what it is supposed to do and doesn’t break other integration tests, and then submit a pull request.

Aside from Base64Encoder from ant, and one reference to ProjectInternal, the rest of your plugin simply uses core Java capabilities, so there shouldn’t be an issue of additional software dependencies to test. I suspect there will be some feedback on the code content itself, but I see nothing that would need massive refactoring.

-Spencer


(Denis Lutz) #9

Thank you for this awesome plugin! Just started using it.


(Michel Krämer) #10

You’re welcome! :slight_smile:

Michel


(thirsch81) #11

Very awesome plugin!


(Michel Krämer) #12

Hi Thomas,

Thanks a lot!

Michel


(Sampath Kambar) #13

Hi,

How to download multiple files and folders recursively. I mean, i will specify only the URL containing folder. Plugin has to download everything inside it(files/folders) recursively.

Any suggestions/pointers would be welcome :slight_smile:

Regards, Sam


(uklance) #14

The underlying protocol (http) does not provide an API for listing subdirectories and files in a folder.

Perhaps you could use gradle-geb-plugin to scrape an existing web-page for all the url’s? http://forums.gradle.org/gradle/topics/gradle_geb_plugin_released


(Sampath Kambar) #15

Hi lance,

My requirement is, I have few binaries located at a remote repository. I can access them through URL. Now I want to download them using a Gradle plug-in.

The above plugin downloads all the files, but not recursively.

How can I use the plugin specified by you?.

Regards, Sam


(uklance) #16

Unlike java.io.File, java.net.URL does not have an API to list subdirectories and files of a http path. This is a limitation of http, not java / groovy. So, the gradle-download-task can’t download a file tree.

Let’s assume that you can access a webpage which lists all the files you want to download. If this is the case, you could scrape the webpage using a utility (like geb, jaunt, jsoup etc) and get a list of URL’s you want to download. Once you have the list, you can use the gradle-download-task to download them all.


#17

You can also use ‘org.apache.ivy.util.url.ApacheURLLister’ to convert a ‘standard’ Apache directory listing page into a list of URLs.


(Michel Krämer) #18

Cool. I didn’t know this class. Thanks for sharing!

I think it would be worth putting an example how to use ApacheURLLister together with the download plugin in the plugin’s readme. Let me see what I can do. I hope I’ll find some time tomorrow to put it online. I’ll keep you updated.

Michel


(Sampath Kambar) #19

Thanks alot Lance and Daz. :slight_smile:


(Sampath Kambar) #20

I have another doubt. Suppose I am downloading few files from URLs. And one file is not present in the source. At that time, this tool fails and exits. It does not try downloading the next file.

How to add that feature?. Any idea?

Regards, Sampath