Different stages of the build and delivery pipeline produce different artifacts, at different points in time (the pipeline is sequential). Other than the end product artifact, there are quality records that the team wishes to keep. At first I though I could upload the artifacts to maven as they were being produced by the different stages, and use the maven classifier to name them after the pipeline stage that produced them, so they would have a unique name and coexist under the same maven coordinates (groupId:artifactId:version). But at the Gradle Summit, @sterling said this would not work well with the maven metadata. He proposed instead to have a different repository for each stage of the pipeline. This seems like a lot of repositories to manage.
I was wondering if anyone had ideas on where and how to store pipeline produced artifacts.
At the Summit, I talked about how we had initially tried to use classifiers and then different repositories and classifiers for keeping platform specific artifacts separate. Both of those strategies didn’t work out well.
If you’re talking about just uploading the different quality records along with the final artifact (and not producing different variants of the same artifact), I think you can accomplish that with a single pipeline and publishing step. e.g., with Jenkins, you can either share a workspace or copy artifacts from one workspace to another in your pipeline. Then your final build step can publish everything at once. You could use classifiers for that.
One of the issues with using classifiers to publish different artifacts has to do with when you publish them. If you publish them as they come available, you’ll necessarily publish the metadata file multiple times (which is one kind of headache) each time you add an artifact or you’ll publish the metadata file with a list of artifacts that do not exist yet. If you go the first way, you might run into problems on the artifact repository side (the wrong metadata files overwriting one another) or just managing the jobs since you can’t parallelize them. If you go the second way, you’ll run into problems in other projects if they try to pull in artifacts that have metadata but have not been published yet.
My recommendation at the Summit for publishing the same artifact built for multiple platforms was to encode the platform into the artifact id or version number. That’s what my project ended up doing (using the artifact id). It sounds like your problem might be different and you might be able to avoid most of the problems by just having a single publish step, if I understand you correctly.
GoCD can store and manage the artifacts for you (this is one possible solution). We’ve found that solution to be too slow when dealing with binaries. Storing some binaries with GoCD’s built-in artifacts handling, and some binaries with a BRM (Nexus, Artifactory) was explored and is typically what we’re using (and I’m seeing at other places).
The pattern we used the most for what you’re describing is to upload the binaries to Artifactory (Pro) and let the Build meta-data in Artifactory “remember” which run of which pipeline step produced a certain artifact.
That said (and done), I always visioned a maven repo manager that incrementally built-up repositories as you moved down the pipeline:
Pipeline Upstream
Step 1: curl -X PUT http://maven-repo/pipeline_name/pipeline_run/ - something like /upstream/1/
Step 2: create gradle.properties with this repository, ie set gradle up for artifact publishing
Step 3: gradle ... - uploads the artifacts to this repository
Pipeline Downstream
Step 1: curl -X PUT http://maven-repo/downstream/1/ -d inherit=/upstream/1/ - creates a new repo with all content uploaded from run 1 of the pipeline upstream
Step 2: create gradle.properties with this repository, ie set gradle up for artifact fetching, and publishing
Step 3: updated dependencies, based on what’s in /downstream/1/ (which is all from /upstream/1/)
Step 4: gradle ... - uploads artifacts to /downstream/1/