artifact download (and shasum) only search for finished uploads.

[pda]

The Artifact Search API endpoint supports filtering the search to finished artifacts, so that incomplete uploads are not included.

buildkite-agent artifact download and buildkite-agent artifact shasum probably want to apply that filter, otherwise they're likely to get stuck trying to download artifacts that don't exist.

• more testing and/or local validation.
• sanity-check the decision to not attempt downloading unfinished artifacts.

[waferbaby]

Yesss, this looks like exactly what we need!

[pda]

Given this command:

buildkite-agent artifact download --debug hello.txt .

Before this PR, for an artifact in new state (not finished), it was downloaded:
(in this case the download was successful because I reverted an uploaded artifact back to new state)

2022-03-18 11:35:32 INFO   Searching for artifacts: "hello.txt"
2022-03-18 11:35:32 DEBUG  GET http://agent.buildkite.localhost/v3/builds/8faced79-94fc-4e61-bc99-49162bf0b275/artifacts/search?query=hello.txt
2022-03-18 11:35:32 DEBUG  ↳ GET http://agent.buildkite.localhost/v3/builds/8faced79-94fc-4e61-bc99-49162bf0b275/artifacts/search?query=hello.txt proto=HTTP/1.1 status=200 Δ=61.300692ms
2022-03-18 11:35:32 INFO   Found 1 artifacts. Starting to download to: /redacted/local/dir
2022-03-18 11:35:32 DEBUG  Downloading http://redacted/hello.txt?redacted to /redacted/local/dir/hello.txt
2022-03-18 11:35:32 INFO   Successfully downloaded "hello.txt" 6 bytes

After this PR for that same new (unfinished) artifact:
Note the /artifacts/search?query=hello.txt&state=finished API call now passes state parameter.
And the unfinished artifact is not found.

2022-03-18 11:35:28 INFO   Searching for artifacts: "hello.txt"
2022-03-18 11:35:28 DEBUG  GET http://agent.buildkite.localhost/v3/builds/8faced79-94fc-4e61-bc99-49162bf0b275/artifacts/search?query=hello.txt&state=finished
2022-03-18 11:35:28 DEBUG  ↳ GET http://agent.buildkite.localhost/v3/builds/8faced79-94fc-4e61-bc99-49162bf0b275/artifacts/search?query=hello.txt&state=finished proto=HTTP/1.1 status=200 Δ=88.67795ms
2022-03-18 11:35:28 FATAL  Failed to download artifacts: No artifacts found for downloading

After this PR, once the artifact is finished, just to make sure it still works:
Note the state=finished is still in the API call.
And again, the download succeeds because this was a successful upload that I reverted to new state.

2022-03-18 11:38:56 INFO   Searching for artifacts: "hello.txt"
2022-03-18 11:38:56 DEBUG  GET http://agent.buildkite.localhost/v3/builds/8faced79-94fc-4e61-bc99-49162bf0b275/artifacts/search?query=hello.txt&state=finished
2022-03-18 11:38:56 DEBUG  ↳ GET http://agent.buildkite.localhost/v3/builds/8faced79-94fc-4e61-bc99-49162bf0b275/artifacts/search?query=hello.txt&state=finished proto=HTTP/1.1 status=200 Δ=66.417817ms
2022-03-18 11:38:56 INFO   Found 1 artifacts. Starting to download to: /Users/pda/bk/test-repo
2022-03-18 11:38:56 DEBUG  Downloading http://redacted/hello.txt?redacted to /Users/pda/bk/test-repo/hello.txt
2022-03-18 11:38:56 INFO   Successfully downloaded "hello.txt" 6 bytes

[pda]

Also a quick buildkite-agent artifact shasum test, because this PR also changes that to only look at finished uploads:

For an unfinished artifact:

pda@paulbookpro ~/bk/test-repo $ ~/bk/agent/buildkite-agent artifact shasum hello.txt
2022-03-18 11:44:08 INFO   Searching for artifacts: "hello.txt"
2022-03-18 11:44:08 FATAL  No artifacts found for downloading

For a finished artifact:

pda@paulbookpro ~/bk/test-repo $ ~/bk/agent/buildkite-agent artifact shasum hello.txt
2022-03-18 11:43:58 INFO   Searching for artifacts: "hello.txt"
f572d396fae9206628714fb2ce00f72e94f2258f

[pda]

Here's a previous report that @pzeballos pointed out which validates that this change should be good: #1511

[pda]

Related:

• Only search for finished artifacts #1728