It's subprocesses all the way down - project.exec, wrapper scripts, and argument-handling


(Jason Griffith) #1

Hello!

I’m trying to figure out whether it’s possible to nest a series of wrappers together and invoke them through project.exec without things getting a little confused.

The program I’m trying to invoke is this:
script --quiet --flush --return /dev/null --command “/usr/bin/xbuild /property:Configuration=CodeAnalysis “/property:Platform=Any CPU” /verbosity:minimal /target:Build /p:CommandLineDefineConstants=Linux PathToASolutionFile

When run on the commandline, “script” will invoke its --command as if that command was run in a terminal, and will therefore preserve color codes. Fantastic.

What is less fantastic is trying to figure out how to pass the right stuff in the right order and the right way to project.exec()

// First, what we actually care about...
def args = [
    "/usr/bin/xbuild",
    "/property:Configuration=${configuration}",
    "/property:Platform=Any CPU",
    "/verbosity:minimal",
    "/target:Build",
    "/p:CommandLineDefineConstants=Linux"
]
args << solution.absolutePath

// Now, because we must, wrap in 'script'.  Somehow...
// (Note: obviously I didnt do all these at the same time, they're just written this way
//  for forum-expedience)
//
//// Tried stuffing the xbuild command into a single element of the args array wrapped 
//// in double-quotes, and wrapping long elements (the 'Any CPU' part) in 
//// backslashes-and-double-quotes which is required when running 
//// this on the commandline directly.
////    Result - Commandline: [script, --flush, --quiet, --return, /dev/null, --command, "/usr/bin/xbuild /property:Configuration=CodeAnalysis \"/property:Platform=Any CPU\" /verbosity:minimal /target:Build /p:CommandLineDefineConstants=Linux thing.sln"]
////    bash: /usr/bin/xbuild /property:Configuration=CodeAnalysis "/property:Platform=Any CPU" /verbosity:minimal /target:Build /p:CommandLineDefineConstants=Linux thing.sln: No such file or directory
args = ["script", "--flush", "--quiet", "--return", "/dev/null", "--command"] + ("\"" + args.collect({it.contains(" ") ? "\\\"$it\\\"" : it}).join(" ") + "\"")

//// Tried putting everything into the arg list individually.  
////    Result: Commandline: [script, --flush, --quiet, --return, /dev/null, --command, /usr/bin/xbuild, /property:Configuration=CodeAnalysis, /property:Platform=Any CPU, /verbosity:minimal, /target:Build, /p:CommandLineDefineConstants=Linux, thing.sln]
////    xbuild ran but as if no other arguments were supplied to it.
args = ["script", "--flush", "--quiet", "--return", "/dev/null", "--command"] + args 

//// Tried only putting xbuild's arguments into a single element, but adding the executable to the list.
////    Result: Commandline: [script, --flush, --quiet, --return, /dev/null, --command, /usr/bin/xbuild, /property:Configuration=CodeAnalysis \"/property:Platform=Any CPU\" /verbosity:minimal /target:Build /p:CommandLineDefineConstants=Linux thing.sln]
////    Same as above...  Xbuild ran but as if it had no arguments.
args = ["script", "--flush", "--quiet", "--return", "/dev/null", "--command", args[0]] +  args[1..-1].collect({it.contains(" ") ? "\\\"$it\\\"" : it}).join(" ")

// Execute...  Hopefully.
logger.quiet("Commandline: $args")
result = project.exec {
    errorOutput = errorOut
    standardOutput = standardOut
    commandLine(args)
}

Any thoughts on the correct way to get a script-within-a-script to execute under Gradle? Preferably we could find a way to get color to come through without using ‘script’, but that’s another forum post: Project.exec, Linux, and color terminals

Thank you!


(Schalk Cronjé) #2

There is no direct support for this in the Gradle Build Tool itself. The best is to roll this on in a task type (which can setup via buildSrc or a full plugin).

The following plugins implements this kind of pattern and can act as examples.

The JRuby case is the closest to wahat you have described. It has a JRubyExec tak type and a project.jrubyexec extension. Both of these take parameters which can be passed to the JVM. parameters to JRuby and then parameters to be passed to the script itself. It handles these as seperate DSL keywords, which are then constructed in the correct order. (In this case it will ventuially call project.javaexec underneath).

The NPM plugin allows for parameters to be passed to node, parameters to be passed to npm itself and then parameters which are NPM command-specific. Once again these are handled via separate keyword, which are constructed in the correct order and then evenutally passed to project.exec

In addition to the above the Grolifant library provides support for building execution specifications and task types for wrapping tools and script-like tools.

HTH


(Jason Griffith) #3

Hi, Schalk!

Thanks for your response! I’ve checked out your links, read through the Grolifant docs, and looked through the available source in those plugins. I may be misreading, but unfortunately it doesn’t seem like we can do what we’re shooting for using those techniques or libraries.

Looked through the source for the JRuby and NodeJS plugins, and while both do lots of nifty things, at the end of the day they both build up a fairly regular(-looking) list of arguments to pass in.

Do they invoke scripts that then invoke their own subscripts?

Using the Grofiliant library, we’d extend AbstractCommandExecSpec with our own class, but we’d still need to setExecutable(‘script’) and still supply a List to cmdArgs, correct? Does that handle execution or CLI parsing differently than Project.exec()?

Cheers!
Jason


(Schalk Cronjé) #4

They actually call project.exec or project.javaexec below the hood, but they ensure that the arguments are in the correct order when calling the executable in question.

For instance there is an npmexec that will end up calling node <node..args> npm-cli.js <npm_command> <args..to..npm..command>. The script author does not see that. Script authors only be exposed to setting the command and the arguments to the command.

What they also do is build their own execution specificatons that extend BaseExecSpec. As project.exec does not take an ExecSpec as parameter, but an Action or Closure you need to do something extra to manipulate it. Something like the following

        Closure runner = { <? extends ExecSpec> fromSpec, ExecSpec toSpec ->
            fromSpec.copyToExecSpec(toSpec)
        }
        project.exec runner.curry(execSpec)

In the end this becomes the easiest as you can still reuse the functionality that project.exec offers, but concentrate on adding the richer functionality you need.

If you need to handle execution differently then you will need to roll a new solution using Process from the JDK or for Java execution you could look at the Worker API that is coming in 4.1.

HTH

P.S. If you want we can discuss this over Hangouts and maybe you can walk me through the exact details.


(Jason Griffith) #5

Thanks for the info! It’s the “ensure that the arguments are in the correct order” that I’m tripping over… And I can’t articulate the issue in words very well.

I’m trying to create a stripped-down recreation to make the problem clearer and so far haven’t managed a simplified case in Windows (it’s all working as expected, darn it! :scream:). Going to try under Linux… Perhaps it’s an OS-y issue?

I’ll get back to ya :slight_smile:


(Schalk Cronjé) #6

It’s could be a too long command-line under Windows (don’t know if that is still an issue in Windows 10).

Also I noted that you are trying to run xbuild. There is actually a Gradle plugin to run xbuild and msbuild.


(Stefan Oehme) #7

Your first attempt looks almost correct, just leave out the quotes around the concatenation of arguments.

If that doesn’t work, a stripped down example using some dummy script would be useful for others to reproduce.


(Jason Griffith) #8

Afraid I need to abort the effort to create a stripped-down example of my problem here… Been having a number of other Linux-related problems, plus we’ve decided to wait until we replace XBuild with MSBuild to attack the real thing we’re trying to achieve (ANSI colors from the subprocess)

Thanks for the help!