Normalise upload path to Unix/URI form on Windows

[ticky]

This is in an effort to address failures to download on Google Cloud due to the mix of path separators being unsupported.

I have updated the related specs to always expect Unix/URI style paths for the relative file path (Path field) on all platforms.

Given that the uploaders all use the AbsolutePath, which is left unchanged here, this should allow uploads to continue to function, but with more correct URLs on upload. Currently, this will also change the path shown to the user in the console output, API and Buildkite UI, which may or may not be desirable. Further discussion of the implications here is likely needed, and in either case this may be considered a breaking change!

This is now behind an experiment flag, as it would be a breaking change for some people, but likely will become the default in the next major release.

potentially fixes #354, fixes #1159, follows #638 (which did almost exactly the same thing!) and #1113.

See also this post in the forum

[ticky]

Given the change in paths used for display, perhaps this change requires a new normalised-path field, which is used for computing the upload URL, but not the URL shown during upload or in the UI. I think this would require the API to accept this field, or some way to filter it out, because the api.Agent struct used to pass around this information is built around that purpose. Alternately, we could shift the path normalisation to the steps which compute the URL, but that requires changes to every upload strategy we implement, whereas this nips it in the bud early.

Happy to discuss, though!

[ticky]

Additionally, should this bit of Artifact Downloader logic also do the opposite, and be using filepath.FromSlash / filepath.ToSlash?

agent/agent/artifact_downloader.go

Lines 93 to 97 in 57f9045

	// Convert windows paths to slashes, otherwise we get a literal
	// download of "dir/dir/file" vs sub-directories on non-windows agents
	if runtime.GOOS != `windows` {
	path = strings.Replace(path, `\`, `/`, -1)
	}

[petemounce]

This looks good to me. The behaviour that causes difficulty for us is uploading from Windows to then download on not-Windows. It makes sense to me to defend against that at the edge, which here is the upload and download (your follow up comment for on Windows). cc @jamiebrynes7

[sj26]

This looks really good, and I think is a great direction.

I'm going to remove the golang team review request because it looks like @pda is assigned.

[pda]

Great work @ticky 🙌

[jayco]

This is awesome @ticky 👌