Add Fusion support to HTCondor

[JosephLalli]

Hi there,

Feel free to use this code or not, but I want to let you know that I have come back to this project (#3697) and have Nextflow successfully running on UWisc CHTC's HTCondor implementation using Fusion and a connection to a local s3 storage. I have a separate Dockerfile and launch script [here] that allows for running Nextflow in an Apptainer container off a submit node with the ancillary files necessary to establish a connection to the scheduler.

I have a feeling that this code would require some changes to meet your house style. First, I do not know how Nextflow typically handles the kind of issues surrounding scheduler access that I described above. I know you don't have a separate Dockerfile for each grid computation software package. I'm willing to work to implement scheduler access in a Nextflow-y manner (even if it's just "make sure the user environment is set up correctly").

Second, I could not figure out how to directly submit both a server configuration request (called a submit file in Condor) and the script to run over stdin. Condor requires both a submit file (which can be submitted via stdin) and a separate execution file, which is specified in the submit file and automatically transferred to the compute server. For now, I have created a generateFusionBashWrapperCommand function in the CondorTaskHandler class that reports the output of FusionHelper.runWithContainer with a bash shebang. fusionStdinWrapper writes the output of generateFusionBashWrapperCommand to FileHelper.getLocalTempPath().resolve(".condor.${task.id}.${task.hash}.sh"), which is then included in the submit file that is passed to condor_submit. I copied write0 from GridTaskHandler, since write0 is private and I couldn't figure out how to run it from CondorTaskHandler. I imagine there is a better file writing function to use, as well as a file location and file name that better comports with Nextflow style.

Finally, Condor allows for the "condor_history" command to parse a local log file that is automatically updated by multiple jobs and report the job status (amongst other attributes) of each job. The advantage of this method over "condor_q" is that it does not make a submission to the scheduler server, and can be repeatedly and easily checked. I give each job the same local log file, saving it to the $PWD with the run uuid as file name. I imagine there is a logical place for Nextflow to store this file, but I do not know where.

It has only been tested with Apptainer process containers, but both nextflow hello and nextflow rnaseq-nf run to completion. Root access is not an option on remote servers, so docker cannot be used, but most other containerization software should function well.

Notably, condor will handle file transfer of small files, including container image files. While I used a shared drive for my apptainer cache location, it would be straightforward to tell condor to transfer a locally stored container image to the execution server before running Fusion's apptainer command, or potentially to download it from Tower.

I'm happy to work with you all to integrate this into Nextflow as you see fit.

Signed-off-by: Joseph L Lalli [email protected]

[netlify]

✅ Deploy Preview for nextflow-docs-staging ready!

Name	Link
🔨 Latest commit	ae98186
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/67ce292899068d0008ecddbb
😎 Deploy Preview	https://deploy-preview-5865--nextflow-docs-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.