Collecting output from Worker API tasks

(JF Bibeau) #1

Hello!

Is there an idiomatic way of collecting output from a set of tasks run using the Worker API? In other words, if I have a workerExecutor fire off a Runnable task to a bunch of workers to operate on files in parallel, what’s the best way to ‘collect’ the output of the calculation?

To give a more concrete example, say we re-use the Worker API tutorial example that computes a hash per file, if instead of writing the hash to a file in each Runnable, you wanted to simply return the hash string, then have the initial task collect them all and dump them to a single file?

I’ve found a way using a ConcurrentHashMap as a static variable on the task, and having the Runnable update that, but it’s a bit hacky, and prevents the tasks from safely being re-used in parallel. I’ve tried passing the Map as an argument to the runnable via params, but it appears to copy the object, not pass the reference to it, hence each Runnable gets its own copy :frowning_face:

Thoughts?

1 Like
(Gary Hale) #2

Unfortunately, there is not a way to gather computed values from work items. It’s an important use case, though. We eventually want to support a Callable as the item of work implementation and ship the result back to the caller, but at the moment, we only support Runnable.

(JF Bibeau) #3

Thanks, that’s along the lines of what I was thinking, Callable as a work implementation would be great.