Unable to use dynamic versions with an S3-based ivy respository

simtel12 · April 21, 2016, 6:12pm

I’m using a custom plugin that is stored in a private S3-backed ivy repository, but I’m having trouble using dynamic versions when referencing it in my gradle build script. Using a specific version works just fine with the S3-backed repository, and dynamic versions work correctly if I put the plugin in a local standard-file-based repository.

The S3 credentials that I’m using have the ability to list the contents of the bucket (tested using a separate S3 command-line tool), including the contents of the directory that gradle reports as having no versions.

This is what I’m trying to do:

buildscript {
    repositories {
        ivy {
            name "s3Ivy"
            url 's3://ivy-bucket/repo'
            credentials(AwsCredentials) {
                accessKey S3_ACCESS_KEY
                secretKey S3_SECRET_KEY
            }
       }
    }
    dependencies {
        classpath 'com.android.tools.build:gradle:2.0.0'
        classpath 'com.razerzone:library-publishing:1.0.+'
    }
}`

And this is what it results in:

   > Could not find any matches for com.razerzone:library-publishing:1.0.+ as no versions of com.razerzone:library-publishing are available.
     Searched in the following locations:
         s3://ivy-bucket/repo/com.razerzone/library-publishing/

This is what is actually in the S3 repo:

s3cmd -c ~/s3cfg.ivy-bucket ls s3://ivy-bucket/repo/com.razerzone/library-publishing/
                   DIR   s3://ivy-bucket/repo/com.razerzone/library-publishing/1.0.0/
                   DIR   s3://ivy-bucket/repo/com.razerzone/library-publishing/1.0.2/
                   DIR   s3://ivy-bucket/repo/com.razerzone/library-publishing/1.0.3/
                   DIR   s3://ivy-bucket/repo/com.razerzone/library-publishing/1.0/

If I reference one of the versions directly via classpath 'com.razerzone:library-publishing:1.0.2' then it works correctly.

Any ideas on how I can work around this, or where in the gradle source I could start hunting to fix this?

Thanks,
-Greg

mark_vieira · April 22, 2016, 5:28am

Can you try running with --debug, this should give some indication of what Gradle thinks is in that module directory.

github.com

gradle/gradle/blob/5436ea4a17968e385bc5c55950ecbf0ad8d6be0e/subprojects/dependency-management/src/main/java/org/gradle/api/internal/artifacts/repositories/resolver/ResourceVersionLister.java#L64-L93


private List<String> listRevisionToken(ExternalResourceName versionListPattern) {
    String pattern = versionListPattern.getPath();
    if (!pattern.contains(REVISION_TOKEN)) {
        LOGGER.debug("revision token not defined in pattern {}.", pattern);
        return Collections.emptyList();
    }
    String prefix = pattern.substring(0, pattern.indexOf(REVISION_TOKEN));
    if (revisionMatchesDirectoryName(pattern)) {
        ExternalResourceName parent = versionListPattern.getRoot().resolve(prefix);
        return listAll(parent);
    } else {
        int parentFolderSlashIndex = prefix.lastIndexOf(fileSeparator);
        String revisionParentFolder = parentFolderSlashIndex == -1 ? "" : prefix.substring(0, parentFolderSlashIndex + 1);
        ExternalResourceName parent = versionListPattern.getRoot().resolve(revisionParentFolder);
        LOGGER.debug("using {} to list all in {} ", repository, revisionParentFolder);
        if (!directories.add(parent)) {
            return Collections.emptyList();
        }
        result.attempted(parent);
        List<String> all = repository.list(parent.getUri());

This file has been truncated. show original

simtel12 · April 22, 2016, 11:00pm

Thanks for replying, Mark.

Running it with --debug looks like it doesn’t find anything, but running the same request with curl seems to show all the correct directories.

./gradlew assembleStandardRelease --debug
...
...
13:39:27.237 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.DynamicVersionResolver] Attempting to resolve version for com.razerzone:library-publishing:1.0.+ using repositories [localIvy, s3Ivy, BintrayJCenter]
13:39:27.241 [DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.ResourceVersionLister] Listing all in file:/Users/gleach/.ivy2/repo/com.razerzone/library-publishing/[revision]/ivy-[revision].xml
13:39:27.242 [DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.ResourceVersionLister] using org.gradle.internal.resource.transport.file.FileResourceConnector@f1be122 to list all in file:/Users/gleach/.ivy2/repo/com.razerzone/library-publishing/
13:39:27.242 [DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.ResourceVersionLister] Listing all in file:/Users/gleach/.ivy2/repo/com.razerzone/library-publishing/[revision]/library-publishing-[revision].jar
13:39:27.245 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Opening cache module-versions.bin (/Users/gleach/.gradle/caches/modules-2/metadata-2.16/module-versions.bin)
13:39:27.247 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.CachingModuleComponentRepository] Version listing in dynamic revision cache is expired: will perform fresh resolve of 'com.razerzone:library-publishing:1.0.+' in 's3Ivy'
13:39:27.248 [DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.ResourceVersionLister] Listing all in s3://ivy-bucket/repo/com.razerzone/library-publishing/[revision]/ivy-[revision].xml
13:39:27.249 [DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.ResourceVersionLister] using s3Ivy to list all in s3://ivy-bucket/repo/com.razerzone/library-publishing/
13:39:27.249 [DEBUG] [org.gradle.internal.resource.transport.aws.s3.S3ResourceConnector] Listing parent resources: s3://ivy-bucket/repo/com.razerzone/library-publishing/
13:39:27.358 [DEBUG] [com.amazonaws.request] Sending Request: GET https://ivy-bucket.s3.amazonaws.com / Parameters: (prefix: repo/com.razerzone/library-publishing/, delimiter: /, max-keys: 1000, ) Headers: (User-Agent: aws-sdk-java/1.9.19 Mac_OS_X/10.11.3 Java_HotSpot(TM)_64-Bit_Server_VM/24.75-b04/1.7.0_75, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 
13:39:27.438 [DEBUG] [com.amazonaws.services.s3.internal.S3Signer] Calculated string to sign:
"GET

application/x-www-form-urlencoded; charset=utf-8
Fri, 22 Apr 2016 20:39:27 GMT
/ivy-bucket/"
13:39:27.595 [DEBUG] [org.apache.http.impl.conn.PoolingClientConnectionManager] Connection request: [route: {s}->https://ivy-bucket.s3.amazonaws.com:443][total kept alive: 0; route allocated: 0 of 50; total allocated: 0 of 50]
13:39:27.610 [DEBUG] [org.apache.http.impl.conn.PoolingClientConnectionManager] Connection leased: [id: 0][route: {s}->https://ivy-bucket.s3.amazonaws.com:443][total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]
13:39:28.143 [DEBUG] [com.amazonaws.http.conn.ssl.SdkTLSSocketFactory] socket.getSupportedProtocols(): [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2], socket.getEnabledProtocols(): [TLSv1]
13:39:28.144 [DEBUG] [com.amazonaws.http.conn.ssl.SdkTLSSocketFactory] TLS protocol enabled for SSL handshake: [TLSv1.2, TLSv1.1, TLSv1]
13:39:28.145 [DEBUG] [org.apache.http.impl.conn.DefaultClientConnectionOperator] Connecting to ivy-bucket.s3.amazonaws.com:443
13:39:28.146 [DEBUG] [com.amazonaws.http.conn.ssl.SdkTLSSocketFactory] connecting to ivy-bucket.s3.amazonaws.com/54.231.168.166:443
13:39:28.416 [DEBUG] [com.amazonaws.internal.SdkSSLSocket] created: ivy-bucket.s3.amazonaws.com/54.231.168.166:443
13:39:28.431 [DEBUG] [org.apache.http.client.protocol.RequestAddCookies] CookieSpec selected: default
13:39:28.432 [DEBUG] [org.apache.http.client.protocol.RequestAuthCache] Auth cache not set in the context
13:39:28.433 [DEBUG] [org.apache.http.client.protocol.RequestProxyAuthentication] Proxy auth state: UNCHALLENGED
13:39:28.433 [DEBUG] [com.amazonaws.http.impl.client.SdkHttpClient] Attempt 1 to execute request
13:39:28.434 [DEBUG] [org.apache.http.impl.conn.DefaultClientConnection] Sending request: GET /?prefix=repo%2Fcom.razerzone%2Flibrary-publishing%2F&delimiter=%2F&max-keys=1000 HTTP/1.1
13:39:28.501 [DEBUG] [org.apache.http.impl.conn.DefaultClientConnection] Receiving response: HTTP/1.1 200 OK
13:39:28.506 [DEBUG] [com.amazonaws.http.impl.client.SdkHttpClient] Connection can be kept alive indefinitely
13:39:28.526 [DEBUG] [com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser] Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
13:39:28.527 [DEBUG] [org.apache.http.impl.conn.PoolingClientConnectionManager] Connection [id: 0][route: {s}->https://ivy-bucket.s3.amazonaws.com:443] can be kept alive indefinitely
13:39:28.528 [DEBUG] [org.apache.http.impl.conn.PoolingClientConnectionManager] Connection released: [id: 0][route: {s}->https://ivy-bucket.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]
13:39:28.528 [DEBUG] [com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser] Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
13:39:28.529 [DEBUG] [com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser] Examining listing for bucket: ivy-bucket
13:39:28.530 [DEBUG] [com.amazonaws.request] Received successful response: 200, AWS Request ID: A3E57B0A1041D3B7
13:39:28.531 [DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.ResourceVersionLister] found 0 resources
13:39:28.532 [DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.ResourceVersionLister] Listing all in s3://ivy-bucket/repo/com.razerzone/library-publishing/[revision]/library-publishing-[revision].jar
13:39:28.533 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.dynamicversions.SingleFileBackedModuleVersionsCache] Caching version list in module versions cache: Using '[]' for 'com.razerzone:library-publishing'
...
...

And then by hand:

curl 'https://ivy-bucket.s3.amazonaws.com:443/?prefix=repo%2Fcom.razerzone%2Flibrary-publishing%2F&delimiter=%2F&max-keys=1000'
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
   <Name>ivy-bucket</Name>
   <Prefix>repo/com.razerzone/library-publishing/</Prefix>
   <Marker />
   <MaxKeys>1000</MaxKeys>
   <Delimiter>/</Delimiter>
   <IsTruncated>false</IsTruncated>
   <CommonPrefixes>
      <Prefix>repo/com.razerzone/library-publishing/1.0.0/</Prefix>
   </CommonPrefixes>
   <CommonPrefixes>
      <Prefix>repo/com.razerzone/library-publishing/1.0.2/</Prefix>
   </CommonPrefixes>
   <CommonPrefixes>
      <Prefix>repo/com.razerzone/library-publishing/1.0.3/</Prefix>
   </CommonPrefixes>
   <CommonPrefixes>
      <Prefix>repo/com.razerzone/library-publishing/1.0/</Prefix>
   </CommonPrefixes>
</ListBucketResult>

mark_vieira · April 23, 2016, 3:21am

I’m wondering if it’s getting the object summaries that’s failing. Perhaps this requires more permissions?

github.com

gradle/gradle/blob/5436ea4a17968e385bc5c55950ecbf0ad8d6be0e/subprojects/resources-s3/src/main/java/org/gradle/internal/resource/transport/aws/s3/S3Client.java#L145-L158


private List<String> resolveResourceNames(ObjectListing objectListing) {
    List<String> results = new ArrayList<String>();
    List<S3ObjectSummary> objectSummaries = objectListing.getObjectSummaries();
    if (null != objectSummaries) {
        for (S3ObjectSummary objectSummary : objectSummaries) {
            String key = objectSummary.getKey();
            String fileName = extractResourceName(key);
            if (null != fileName) {
                results.add(fileName);
            }
        }
    }
    return results;
}

simtel12 · April 25, 2016, 11:56pm

I’ve tried it with full-access credentials and still get the same result. Later this week or early next I can try building gradle from source and adding some more logging to the S3Client.

simtel12 · April 26, 2016, 6:47pm

Had a quick poke at it - interestingly the problem seems to be in the amazonS3Client.listObjects call. It’s returning no summaries at all. If I instead use the non ListObjectsRequest call (listObjects(bucketName, prefix)) then it correctly gets the objects - though it fails to determine which version to use. Continuing investigation.

github.com

gradle/gradle/blob/5436ea4a17968e385bc5c55950ecbf0ad8d6be0e/subprojects/resources-s3/src/main/java/org/gradle/internal/resource/transport/aws/s3/S3Client.java#L135


    S3RegionalResource s3RegionalResource = new S3RegionalResource(parent);
    String bucketName = s3RegionalResource.getBucketName();
    String s3BucketKey = s3RegionalResource.getKey();
    configureClient(s3RegionalResource);


    ListObjectsRequest listObjectsRequest = new ListObjectsRequest()
            .withBucketName(bucketName)
            .withPrefix(s3BucketKey)
            .withMaxKeys(1000)
            .withDelimiter("/");
    ObjectListing objectListing = amazonS3Client.listObjects(listObjectsRequest);
    results.addAll(resolveResourceNames(objectListing));


    while (objectListing.isTruncated()) {
        objectListing = amazonS3Client.listNextBatchOfObjects(objectListing);
        results.addAll(resolveResourceNames(objectListing));
    }
    return results;
}


private List<String> resolveResourceNames(ObjectListing objectListing) {

Incidentally, the gradle daemon seems to have issues building subproject tasks. The following command does not result in a new gradle being built and/or installed properly on subsequent builds: ./gradlew resourcesS3:build -x resourcesS3:test -x resourcesS3:integTest install -Pgradle_installPath=/Users/gleach/gradle-source-build when the daemon is in charge. When the daemon is not enabled it works. Should probably be a new topic though, unless it’s a known bug already.

Edit:

Okay, I think I’ve found the problem.

No summaries are returned because the default ivy structure does not keep files in the module directory - it keeps version directories.

The listObjects call is returning the version directories (eg. common prefixes). This explains why it works when an explicit version is specified - because there ARE files in that directory.

mark_vieira · April 30, 2016, 7:49pm

So is listObjects not returning directories? Does it only return results for files inside a directory?

simtel12 · May 4, 2016, 11:42pm

Apologies, I could have worded that better.

No, listObjects is returning all the items in the directory, including the directories. resolveResourceNames is ignoring the the directories. This happens because resolveResourceNames is only considering object summaries - but directories are returned as common prefixes.

It’s possible to fix this use-case by pushing objectListing.getCommonPrefixes through some directory-name-extraction function similar to extractResourceName. This breaks a few tests, but this is what I used.

private static final Pattern DIRNAME_PATTERN = Pattern.compile("([^\\/]+)[\\/]$");
private List<String> resolveResourceNames(ObjectListing objectListing) {
    List<String> results = new ArrayList<String>();
    List<S3ObjectSummary> objectSummaries = objectListing.getObjectSummaries();
    if (null != objectSummaries) {
        for (S3ObjectSummary objectSummary : objectSummaries) {
            String key = objectSummary.getKey();
            String fileName = extractResourceName(key);
            if (null != fileName) {
                results.add(fileName);
            }
        }
    }

    List<String> commonPrefixes = objectListing.getCommonPrefixes();
    if(null != commonPrefixes) {
        for(String prefix : commonPrefixes) {
            String dirName = extractDirectoryName(prefix);
            if(null != dirName) {
                results.add(dirName);
            }
        }
    }

    return results;
}

private String extractDirectoryName(String key) {
    Matcher matcher = DIRNAME_PATTERN.matcher(key);
    if (matcher.find()) {
        return matcher.group(1);
    }
    return null;
}

On the subject of extractResourceName, it seems fundamentally flawed to only consider files that have a . in the name as resources. I’m guessing that it’s legacy support and/or matching the way that other ResourceConnectors work, but shouldn’t that ignoring of files happen in the consumer of the file-listing, instead of the S3Client?

Edit: Fixed the tests and am putting together a pull request. Will update with the link to the PR later today.

Edit 2: PR: https://github.com/gradle/gradle/pull/636