HTCondor logging has unexpected behaviour after introduction of transfer_output_remaps.

[tvoigtlaender]

Bug description

1. The HTCondor options for stream_output and stream_error seem broken due to the way that the output remapping is handled
2. The handling for similar paths between output and error can be iffy.

Version: v0.1.20

Output streaming

Previously it was possible to have the logfiles at a custom location and also stream to that location.

The output and error arguments now only use the basename of the provided path.
While the remapping approach works when the files are transferred out of the job, it does not work with the file streaming.
The streamed files end up in the same directory as the *.jdl file instead.

The option to stream the logs is great for longer jobs/tasks to check their status, so it would be nice to somehow have this option again.

---

Example for wrong location of stderr and stdout when streaming:

...
config.custom_content.append(("stream_error", "True"))
config.custom_content.append(("stream_output", "True"))
config.stdout = os.path.join("Output", "Output_$(JobId).txt")
config.stderr = os.path.join("Error", "Error_$(JobId).txt")
...

For the purpose of streaming, this leads to an effective path of <htcondor_create_job_file_factory().dir>/Output_$(JobId).txt instead of the expected <htcondor_create_job_file_factory().dir>/Output/Output_$(JobId).txt.

Similar outputs

In addition, the reduction of both output and error to their basename can lead to unexpected issues if the basename of both is the same and only the rest of the provided path differs.

---

Example for mix up between stderr and stdout:

...
config.stdout = os.path.join("Output", "$(JobId).txt")
config.stderr = os.path.join("Error", "$(JobId).txt")
...

leads to

transfer_output_remaps = "$(JobId)$(law_job_postfix).txt = <htcondor_log_directory>/Error/$(JobId)$(law_job_postfix).txt

and there is no entry for Output.