S3 AwsCredentials should be able to load from an AWS profile file

Help/Discuss
1

I’m really excited about the S3 repository support in the 2.4!

The AWS Java SDK supports loading credentials from a common profiles file, typically stored in .aws/credentials, which can then be shared between SDKs and projects. I think loading them this way should be an alternative to explicitly specifying the public and secret keys.

Right now, the credentials block is required for an s3 repository. Gradle will generate an error if it is missing.

I think an ideal solution would to instead fetch credentials from the profile file if the credentials block is missing. The AWS SDKs default to loading credentials from a profile file if not explicitly told, so I think it makes sense for this too.

If you don’t think the implicit profile detection is acceptable, or you wanted to support some of the additional profile options, we could make it explicit, like so:

maven {
  url "s3://myCompanyBucket/maven2"
  credentials(AwsCredentials) {
    useProfilesFile()
  }
}

More information: ProfileCredentialsProvider docs

I would be more than happy to make a pull request for this, but I wanted to propose the idea here first.

1 Like
2

@daz did this come up in the development of this feature to date?

3

Yes was some discussion with Adrian Kelly (original S3 contributor) regarding improving our Credentials infrastructure to better support different sources and formats for credentials:

In addition, we have an outstanding PR that would allow the forcing of preauthentication of HTTP requests. Adam thought this should be integrated with the Credentials DSL:

4

@Danny_Kirchmeier if you’re interested in progressing this, a good place to start would be to get your concerns added into that spec that Daz linked to.

5

Did anything happen with this? I was just looking for this feature today. The linked discussions don’t seem to address S3 credentials from the profile.

6

Just realized it was trivial to do locally until supported by the framework.

7

And in case someone else needs this:


ext {
    awsCredentials = new ProfileCredentialsProvider().credentials
}
publishing {
    repositories {
        maven {
            url "s3://foo"
            credentials(AwsCredentials) {
                accessKey awsCredentials.AWSAccessKeyId
                secretKey awsCredentials.AWSSecretKey
            }
        }
    }
}
8

@axl Did you use an import to make ProfileCredentialsProvider available in your code? For it work, did you simply add a dependency on AWS SDK to your buildscript? Or maybe something else?

9

Here what I did:

import com.amazonaws.auth.*
import com.amazonaws.auth.profile.*

buildscript {
  repositories { jcenter() }
  dependencies { classpath 'com.amazonaws:aws-java-sdk-core:1.11.5' }
}

plugins { id 'ivy-publish' }

def fetchAwsCredentials = {
  try {
    return new ProfileCredentialsProvider().credentials
  } catch (Exception exception) {
    logger.debug('Unable to retrieve AWS credentials from profile, publishing to S3 will not be available.')
    return null
  }
}

AWSCredentials awsCredentials = fetchAwsCredentials()
if (awsCredentials != null) {
  publishing {
    repositories {
      ivy {
        name 's3'
        // Ensures you have the right region sub-domain and bucket here!
        url 's3://<bucket>.s3-<region>.amazonaws.com'
        credentials(AwsCredentials) {
          accessKey awsCredentials.AWSAccessKeyId
          secretKey awsCredentials.AWSSecretKey
        }
      }
    }
  }
}

Doing only the import was not working correctly (I guess because Gradle S3 subproject classpath was not accessible to my plugin somehow) so I explicitly added a dependency on AWS Java SDK client code. If that not required, I could check why it’s not working.

All in all, thanks @axl for the clean solution!

Regards,
Matt

10

I’m glad you got it sorted, my script is more or less the same as what you’ve done.

11

Some notes about my previous post and lesson learned.

I was using url s3://s3-us-west-1.amazonaws.com/your-bucket-here at first and it seems that was not the format expected by Gradle because it was giving me Access Denied everytime. After a bit of debugging, I found that Gradle expects a specific uri format to extract region, bucketandkey` correctly.

The correct format to use is url s3://<bucket>.s3-<region>.amazonaws.com (and simply s3://<bucket>.s3.amazonaws.com for us-east-1 region). As an example, s3://my-special-s3-bucket.s3-us-west-1.amazonaws.com.

This scheme will correctly extract all information for a correct upload to S3. Check http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region for list of endpoints and simply do s3://<bucket> + region endpoint in documentation

I edited my previous post to show accurate information.

1 Like
12

If you use new DefaultAWSCredentialsProviderChain() instead of new ProfileCredentialsProvider() it will respect the AWS credential provider chain (http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html) and get environment variables first, followed by java system properties, etc.

13

How would one reference DefaultAWSCredentialsProviderChain() from buildscript{} or pluginRepositories{} block?

2 Likes