Com.google.appengine sdk jar not available in compilation

Help/Discuss
1

Introduction

I am modifying zuul java project from netflix to make it run in the appengine standard environment. To achieve this, some patches need to be done to the core of zuul, mainly replacing java Threads with google appengine’s thread methods.

The problem is that while I can more or less efficiently patch the code, I cannot get the compiler to find ‘com.google.appengine’ package, which is included. I am using the oficial google appengine gradle plugin.

Patch

I have created the minimum set of changes to reproduce this problem, and it’s available in github. I have left out of the patch the xml and the deployment part for appengine to avoid having noise on it. The patch does the following things:

  • Specify appengine-api-1.0-sdk as a compile dependency (this is where com.google.appengine is specified)
  • Specify the appengine gradle plugin as an script dependency
  • Apply the plugin on zuul-core although I don’t know if this is really needed (I just need the thread manager calls from appengine-api-1.0-sdk)
  • Specify to automatically download the sdk

Root build.gradle

This is the final state of build.gradle, adding the dependency and repositories as specified in the gradle appengine plugin docs

buildscript {
    repositories {
        jcenter()
        mavenCentral()
    }
    dependencies {
        classpath 'com.netflix.nebula:gradle-extra-configurations-plugin:2.2.0'
        classpath 'com.google.appengine:gradle-appengine-plugin:1.9.34'
    }
}

plugins {
    id 'nebula.netflixoss' version '3.1.2'
}

ext.githubProjectName = rootProject.name

idea {
    project {
        languageLevel = '1.7'
    }
}

subprojects {
    apply plugin: 'nebula.netflixoss'
    apply plugin: 'java'
    apply plugin: 'nebula.provided-base'

    repositories {
        jcenter()
    }

    group = "com.netflix.${githubProjectName}"

    sourceCompatibility = '1.7'
    targetCompatibility = '1.7'

    sourceSets.test.java.srcDir 'src/main/java'

    tasks.withType(Javadoc).each {
        it.classpath = sourceSets.main.compileClasspath
    }

    test {
        forkEvery = 1
        maxParallelForks = 1
    }
}

Zuul core build.gradle

The Zuul core has been patched applying the plugin, and specifying it to download the sdk. I have checked that it successfully downloads the sdk

apply plugin: 'groovy'
apply plugin: 'war'
apply plugin: 'appengine'

dependencies {
    appengineSdk 'com.google.appengine:appengine-java-sdk:1.9.34'

    compile 'com.google.appengine:appengine-api-1.0-sdk:1.9.34'
    compile 'commons-io:commons-io:2.4'
    compile 'org.codehaus.groovy:groovy-all:2.3.10'
    compile 'org.mockito:mockito-all:1.9.5'
    compile 'org.slf4j:slf4j-api:1.7.6'

    provided 'junit:junit-dep:4.10'
    provided 'javax.servlet:servlet-api:2.5'

    compile 'com.netflix.archaius:archaius-core:0.6.0'
    compile 'com.netflix.servo:servo-core:0.7.2'
    compile 'com.netflix.netflix-commons:netflix-commons-util:0.1.1'
}


javadoc {
    options {
        doclet = "org.benjchristensen.doclet.DocletExclude"
        docletpath = [rootProject.file('./gradle/doclet-exclude.jar')]
        stylesheetFile = rootProject.file('./gradle/javadocStyleSheet.css')
        windowTitle = "Zuul Javadoc ${project.version}"
    }
}

eclipse {
    classpath {
        downloadSources = true
        downloadJavadoc = true
    }
}

appengine {
    downloadSdk = true
    update {
        useJava7 = true
    }
}

File manager

I wont paste all the code, but basically, I have done the really minimum patch to demonstrate that it cannot use the appengine-api-1.0-sdk.jar, an import statement:

diff --git a/zuul-core/src/main/java/com/netflix/zuul/FilterFileManager.java b/zuul-core/src/main/java/com/netflix/zuul/FilterFileManager.java
index a3afec3..7a9c415 100644
--- a/zuul-core/src/main/java/com/netflix/zuul/FilterFileManager.java
+++ b/zuul-core/src/main/java/com/netflix/zuul/FilterFileManager.java
@@ -23,6 +23,7 @@ import org.mockito.MockitoAnnotations;
 import org.mockito.runners.MockitoJUnitRunner;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
+import com.google.appengine;
 
 import java.io.File;
 import java.io.FilenameFilter;

Build output

The build output in debug mode doesn’t fit here, so I have uploaded it to gist.github.com

As you might have noticed, in this line, we can see how among the compiler arguments there is a /home/javier/.gradle/caches/modules-2/files-2.1/com.google.appengine/appengine-api-1.0-sdk/1.9.34/7c15c22fd362478e9758081d28e51590304d5ff4/appengine-api-1.0-sdk-1.9.34.jar. I have checked the contents of the file and it contains everything it says. The only strange thing is that those zip files don’t have 755 in some internal folders (I decompressed them just in case).

16:00:01.795 [DEBUG] [org.gradle.api.internal.tasks.compile.NormalizingJavaCompiler] Compiler arguments: -source 1.7 -target 1.7 -d /home/javier/projects/spinoffs/zuul/zuul-core/build/classes/main -g -sourcepath /home/javier/projects/spinoffs/zuul/zuul-core/build/tmp/compileJava/emptySourcePathRef -classpath /home/javier/.gradle/caches/modules-2/files-2.1/com.google.appengine/appengine-api-1.0-sdk/1.9.34/7c15c22fd362478e9758081d28e51590304d5ff4/appengine-api-1.0-sdk-1.9.34.jar:/home/javier/.gradle/caches/modules-2/files-2.1/commons-io/commons-io/2.4/b1b6ea3b7e4aa4f492509a4952029cd8e48019ad/commons-io-2.4.jar:/home/javier/.gradle/caches/modules-2/files-2.1/org.codehaus.groovy/groovy-all/2.3.10/e6fc2dc8fe2512efa0e78a2c0ae52ea1d724106d/groovy-all-2.3.10.jar:/home/javier/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-all/1.9.5/79a8984096fc6591c1e3690e07d41be506356fa5/mockito-all-1.9.5.jar:/home/javier/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.6/562424e36df3d2327e8e9301a76027fca17d54ea/slf4j-api-1.7.6.jar:/home/javier/.gradle/caches/modules-2/files-2.1/com.netflix.archaius/archaius-core/0.6.0/1ecb29ef5d4c0c98cae35d1038fd980688eab5f9/archaius-core-0.6.0.jar:/home/javier/.gradle/caches/modules-2/files-2.1/com.netflix.servo/servo-core/0.7.2/b940f73ac9ddb440b79e801c8b936228dc0cc142/servo-core-0.7.2.jar:/home/javier/.gradle/caches/modules-2/files-2.1/com.netflix.netflix-commons/netflix-commons-util/0.1.1/39e67061780476f207b31465baaed84a91ff659f/netflix-commons-util-0.1.1.jar:/home/javier/.gradle/caches/modules-2/files-2.1/junit/junit-dep/4.10/64417b3bafdecd366afa514bd5beeae6c1f85ece/junit-dep-4.10.jar:/home/javier/.gradle/caches/modules-2/files-2.1/javax.servlet/servlet-api/2.5/5959582d97d8b61f4d154ca9e495aafd16726e34/servlet-api-2.5.jar:/home/javier/.gradle/caches/modules-2/files-2.1/org.codehaus.jackson/jackson-mapper-asl/1.9.11/45d70862fa016993193075a1e8e32a01dcf438e8/jackson-mapper-asl-1.9.11.jar:/home/javier/.gradle/caches/modules-2/files-2.1/commons-configuration/commons-configuration/1.8/6cce40435bcd8018018f16898de01976b319941a/commons-configuration-1.8.jar:/home/javier/.gradle/caches/modules-2/files-2.1/org.codehaus.jackson/jackson-core-asl/1.9.11/e32303ef8bd18a5c9272780d49b81c95e05ddf43/jackson-core-asl-1.9.11.jar:/home/javier/.gradle/caches/modules-2/files-2.1/com.google.code.findbugs/annotations/2.0.0/d8dff1d83a79f0c0609c360f02bcd2f2fc1f1369/annotations-2.0.0.jar:/home/javier/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest-core/1.1/860340562250678d1a344907ac75754e259cdb14/hamcrest-core-1.1.jar:/home/javier/.gradle/caches/modules-2/files-2.1/commons-lang/commons-lang/2.6/ce1edb914c94ebc388f086c6827e8bdeec71ac2/commons-lang-2.6.jar:/home/javier/.gradle/caches/modules-2/files-2.1/commons-logging/commons-logging/1.1.1/5043bfebc3db072ed80fbd362e7caf00e885d8ae/commons-logging-1.1.1.jar:/home/javier/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/14.0.1/69e12f4c6aeac392555f1ea86fab82b5e5e31ad4/guava-14.0.1.jar /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/FilterProcessor.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/DynamicCodeCompiler.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/ZuulFilterResult.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/IZuulFilter.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/ZuulFilter.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/ZuulRunner.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/FilterUsageNotifier.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/DefaultFilterFactory.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/FilterFactory.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/ExecutionStatus.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/FilterLoader.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/FilterFileManager.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/util/HTTPRequestUtils.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/util/DeepCopy.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/monitoring/MonitoringHelper.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/monitoring/Tracer.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/monitoring/TracerFactory.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/monitoring/CounterFactory.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/groovy/GroovyCompiler.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/groovy/GroovyFileFilter.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/exception/ZuulException.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/constants/ZuulHeaders.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/constants/ZuulConstants.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/context/RequestContext.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/context/ContextLifecycleFilter.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/context/Debug.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/filters/FilterRegistry.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/filters/ZuulServletFilter.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/http/ZuulServlet.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/http/HttpServletResponseWrapper.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/http/HttpServletRequestWrapper.java /home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/http/ServletInputStreamWrapper.java -XDuseUnsharedTable=true

Miscellaneous

Just in case, I need to have zuul-core patched because if I don’t, making a request to the deployed module returns 500 on thread spawn.

Uncaught exception from servlet
java.lang.ExceptionInInitializerError
	at com.google.appengine.runtime.Request.process-fb6cba390016def3(Request.java)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at java.lang.Class.newInstance(Class.java:588)
	at com.netflix.zuul.DefaultFilterFactory.newInstance(DefaultFilterFactory.java:17)
	at com.netflix.zuul.FilterLoader.putFilter(FilterLoader.java:156)
	at com.netflix.zuul.FilterFileManager.processGroovyFiles(FilterFileManager.java:170)
	at com.netflix.zuul.FilterFileManager.manageFiles(FilterFileManager.java:176)
	at com.netflix.zuul.FilterFileManager.init(FilterFileManager.java:81)
	at com.netflix.zuul.StartServer.initGroovyFilterManager(StartServer.java:60)
	at com.netflix.zuul.StartServer.contextInitialized(StartServer.java:42)
	at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:548)
	at org.mortbay.jetty.servlet.Context.startContext(Context.java:136)
	at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1250)
	at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
	at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:439)
	at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:446)
	at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:256)
	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:310)
	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:302)
	at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:443)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "modifyThreadGroup")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:382)
	at java.security.AccessController.checkPermission(AccessController.java:572)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
	at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
	at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:57)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190)
	at filters.route.SimpleHostRoutingFilter.<clinit>(SimpleHostRoutingFilter.groovy:73)
	... 25 more

The non-verbose build just in case:

$ ./gradlew build 
Inferred project: zuul, version: 1.3.0-SNAPSHOT
Publication nebula not found in project :.
[buildinfo] Not using buildInfo properties file for this build.
Publication named 'nebula' does not exist for project ':' in task ':artifactoryPublish'.
None of the specified publications matched for project ':' - nothing to publish.
:zuul-core:appengineDownloadSdk
:zuul-core:compileJavawarning: [options] bootstrap class path not set in conjunction with -source 1.7
/home/javier/projects/spinoffs/zuul/zuul-core/src/main/java/com/netflix/zuul/FilterFileManager.java:26: error: package com.google does not exist
import com.google.appengine;
                 ^
1 error
 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':zuul-core:compileJava'.
> Compilation failed; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 9.557 secs

I really hope someone can help me through this because I have really run out of ideas.

(This is a cross post, same thread in stackoverflow, will post answer too when available)

Also, about the category, I would post it as a bug, but I am maybe missing something, so I decided to leave it like this.