Cannot access repositories via basic auth proxy

[m_m]

If you can provide a mechanism to specify the preferred authentication scheme via gradle.properties ( some thing like systemProp.http.auth.preference=basic or some other system property ) and if this value is set in httpclient , then it will be very useful.

List authpref = new ArrayList();
 authpref.add(AuthPolicy.BASIC);
 client.getParams().setParameter(AuthPNames.PROXYAUTHPREF, authpref);

It will also help avoiding checking for unnecessary schemes ( NTLM ,NEGOTIATE ) in my case and try for only BASIC authentication for every fetch

[daz]

While providing an explicit override is an option (I’ve now raised GRADLE-2589 for this), we really want gradle to work out of the box without requiring extra configuration like this. So I’d rather get to the bottom of what’s causing your issue. If one scheme fails, we should move onto the next.

Inspecting your debug logs, it looks like this happens: * Your proxy is responds with 407 (proxy auth required), declaring that it supports NEGOTIATE, NTLM and BASIC authentication * Negotiate is not available on the client side, so is ignored (with a warning) * NTLM authentication is attempted using the username/password supplied. * NTML authentication fails and responds with another 407, but the proxy server no longer lists BASIC in the set of available authentication schemes (only NTLM is listed). * For some reason, HttpClient does not attempt the request with BASIC auth. Possibly the subsequent 407 request is parsed and since BASIC is no longer listed, it is not attempted.

I suspect a bug in Apache HttpClient, but without access to your environment it’s very hard to debug this issue. You could greatly help us resolve this problem, by experimenting with your direct HttpClient sample using NTMLCredentials. When NTML authentication fails, an attempt should be made to authenticate with Basic auth, I think. If you can prove this isn’t happening in a simple sample using HttpClient, we could file a bug report.

It could be that you’re hitting a bug similar to https://issues.apache.org/jira/browse/HTTPCLIENT-1107, but with proxy authentication instead of regular http authentication. But I’m certainly no expert in proxy authentication.

[m_m]

Thanks Daz for the response.

Find the log below for direct http client without NT Credentials

https://gist.github.com/d08654e3d7c905dc0628

Find below the log for direct http client after setting org.apache.http.auth.NTCredentials in http Client

https://gist.github.com/f5b902d614d01fd0cf0d

[daz]

Hi again, and thanks for the data. It looks like your proxy does indeed support NTLM authentication, and you’re successfully authenticating using that method (in your second example). The 2 questions I have now are:

1- Why is Gradle rejecting your NTLM credentials? Can you please verify the values that are being sent by Gradle against the values you’re using in your test app? You can see the values Gradle is using by looking for a line like this in your debug logs:

08:32:13.809 [DEBUG] [org.gradle.api.internal.externalresource.transport.http.HttpClientConfigurer] Using Credentials [username: MY_USER_ID] and NTLM Credentials [user: MY_USER_ID, domain: MY_DOMAIN, workstation: MY_WORK_STATION] for authenticating against 'MYPROXY:8080'

2- What happens in your direct HTTPClient test if you supply the incorrect NTLM credentials? You could try setting the invalid value for Domain or Workstation. The goal would be to see if HTTPClient will attempt Basic Auth if NTLM fails when NTML Credentials are supplied.

If you don’t mind working through this with me, I’d love to get to the bottom of it. Proxy authentication is the source of a lot of problems for new users and experienced users alike.

[daz]

There is a undocumented system property for setting the NTML workstation value: ‘http.auth.ntlm.workstation’. You can use this property to tweak the value that Gradle uses in NTLM authentication.

[m_m]

1.This is how I set org.apache.http.auth.NTCredentials in my test program.( Similar to what is done in gradle )

NTCredentials credentials = new NTCredentials("MY_USER_NAME", "MY_PASSWORD", "MY_WORKSTATION", "MY_DOMAIN");
httpclient.getCredentialsProvider().setCredentials(new AuthScope(
"MY_PROXY", MY_PROXY_PORT, AuthScope.ANY_REALM, AuthPolicy.NTLM), credentials);

Since the above values are not getting printed when i test the default http client , I am not

able to verify what are the values actually getting used by httpclient. I verified manually and they are same.

Below are the values that get printed when i use gradle

09:37:55.720 [DEBUG] [org.gradle.api.internal.externalresource.transport.http.HttpClientConfigurer]
 Using Credentials [username: MY_DOMAIN/MY_USER_NAME]
 and NTLM Credentials [user: MY_USER_NAME, domain: MY_DOMAIN, workstation: MY_WORKSTATION] for authenticating against 'MY_PROXY:MY_PROXY_PORT'

2. You are right. When I provide wrong NT Credentials to httpclient , it fails and does not try for other authentication mechanisms.

Looks like the values passed from gradle are different from the ones used by httpclient.I tried specifying the http.auth.ntlm.workstation in gradle.properties but did

not help.

Few differences which I observed were gradle uses org.apache.http.impl.client.SystemDefaultHttpClient and in the

test program I have used org.apache.http.impl.client.DefaultHttpClient.

From the logs , I observed the below

Gradle - > First attempt for Negotiate Fails

Second attempt results in “HTTP/1.1 407 Proxy Authentication Required”

Third attmept results again in “HTTP/1.1 407 Proxy Authentication Required”

Test Client -> First attempt for Negotiate Fails

Second attempt for NTLM results in “HTTP/1.1 407 Proxy Authentication Required” .

Third attempt results in “HTTP/1.1 302 Found”

[daz]

Looks like the values passed from gradle are different from the ones used by httpclient.I tried specifying the http.auth.ntlm.workstation in gradle.properties but did

not help.

It looks to me like the values are the same: which values do you see as different?

Few differences which I observed were gradle uses org.apache.http.impl.client.SystemDefaultHttpClient and in the test program I have used org.apache.http.impl.client.DefaultHttpClient.

Can you re-run your test using the SystemDefaultHttpClient? We need to track down exactly what’s causing Gradle to fail where your test code succeeds.

One other difference is that we’re registering a NTLM scheme factory that uses the ‘jcifs’ library from the ‘samba’ project. You can see the code at ‘org.gradle.api.internal.externalresource.transport.http.ntlm.NTLMSchemeFactory’, with code borrowed from [http://hc.apache.org/httpcomponents-client-ga/ntlm.html].

The reason we do this is that jcifs has proved to handle more NTLM servers than the built-in NTML support. So this is another thing you could try in your direct http test (or try hacking Gradle to remove this custom NTLM Scheme Factory).

From the logs , I observed the below >

Gradle - > First attempt for Negotiate Fails

Second attempt results in “HTTP/1.1 407 Proxy Authentication Required”

Third attmept results again in “HTTP/1.1 407 Proxy Authentication Required” >

Test Client → First attempt for Negotiate Fails

Second attempt for NTLM results in “HTTP/1.1 407 Proxy Authentication Required” .

Third attempt results in “HTTP/1.1 302 Found”

The ‘second’ and ‘third’ attempts are actually part of the NTLM authentication protocol - you will always see 2 request/response interactions. The server responds with a challenge in the first 407 response, and the client uses this challenge to form the next request.

Again, thanks for your patience in helping track this down. Hopefully we can solve your issues and make Gradle better for everyone at the same time!

[daz]

I’ve raised HTTPCLIENT-1272 for the issue where HttpClient does not attempt BASIC auth after NTLM fails against a proxy.

Can you please add any relavent information to that ticket?

[m_m]

when i changed my test program to use SystemDefaultHttpClient and passed the system properties ( proxyHost , proxyPort … etc ), it failed.

The difference i observed in the logs was that SystemDefaultHttpClient uses PoolingClientConnectionManager .

One another interesting issue i observed from the logs was the difference in response

Using DefaultHttpClient -> “Your credentials could not be authenticated: “Another round of authentication required.”. You will not be permitted access until your credentials can be verified.[\r][\n]” Using SystemDefaultHttpClient -> “Your credentials could not be authenticated: “The credentials provided are invalid.”. You will not be permitted access until your credentials can be verified.[\r][\n]”

I am using the same credentials for both DefaultHttpClient and SystemDefaultHttpClient

[m_m]

Let me know if you want to conduct any further tests . Thanks a lot for your time since it will help us in taking a decision to migrate from maven.

[daz]

So you’re saying that with everything else exactly the same, you get authentication success using DefaultHttpClient and failure using SystemDefaultHttpClient?

This seems like it would be a bug in HttpClient. Would you be able to provide the full debug output, as well as the code to your sample? (Maybe put it in a GitHub repository).

We could then raise this as an issue with the HttpClient project: they are very responsive to bug reports.

[daz]

Another difference between DefaultHttpClient and SystemDefaultHttpClient is the use of the ProxySelectorRoutePlanner, which enables all of the http.proxy* system properties.

[m_m]

Below is the source code and logs

Using org.apache.http.impl.client.DefaultHttpClient

Test :

https://gist.github.com/77f7f4278612d38cb554 Logs : https://gist.github.com/42057219f8903f3b92d2

Using org.apache.http.impl.client.SystemDefaultHttpClient

Test : https://gist.github.com/99f166b73f60176bab39 Logs : https://gist.github.com/7f2c4af8b6532f8e9d0c

The input proxy credentials are same for both the tests.

[daz]

Hi. I’ve forwarded your results to the HttpClient user list. The response is 2-fold: 1) SystemDefaultHttpClient will be soon deprecated, so Gradle will need to switch to use DefaultHttpClient 2) It’s possible that the “http.keepAlive” system property is somehow involved. Could you please rerun your SystemDefaultHttpClient test with this property explicitly set to “true”? Also, check if this property has been set on your system to a different value.

Please again post the log output. If you want to participate directly in the conversation with the HttpClient devs, it’s here: you’ll need to subscribe to the mailing list to post.

[m_m]

Thanks Daz - Once I set the http.keepAlive as true, my test and also the gradle sample worked . Thanks for the continuus help provided.

[daz]

Great to hear! So I assume the http.keepAlive property allows NTLM authentication to function correctly?

I don’t think there is any solution to allow you to use Basic Auth credentials against a server that supports NTLM (as per your original request). If your server supports NTLM+Basic, then you need to provide correct NTLM credentials. This isn’t ideal: Gradle needs to provide a way for you to specify if you want to provide these NTLM credentials.

[m_m]

Yes. NTLM authentication works after setting the http.keepAlive to true Agree.It might not be appropriate to try basic when the proxy supports a a more stronger authentication ( NTLM ). But I am not sure whether there might be a need to specify the authentication preference for some other scenario.

It would help new users who hit this issue if the usage of this property is documented as a reference in gradle documentation ( since this is not related to gradle may be a link to http client would help )

[daz]

Yep, agreed about the documentation. Feel free to submit a patch to the user guide :).

Apparently this has been recognised as a bug in HttpClient and will be fixed in 4.2.3. Thanks for reporting and helping us get to the bottom of it.

[m_m]

Sure. I will try to submit the patch. Thanks.

[Sparmboy (Spencer Loveridge)]

Hi

I’ve read through this thread and the HTTPCLIENT-1272 Jira.

The Jira is not being fixed and there wasnt a clear resolution from a Gradle perspective. This thread has also reached a dead end so could someone point out the solution.

Maven works fine, so its quite frustrating that we cant use Gradle, especially when certain spring projects (e.g. spring-data-rest) are Gradle only.

Thanks SL