The DIVCFG framework is a natural way to run commands in a container, for example, using docker or singularity. All we need to do is 1) design a template that will run the job in the container, instead of natively; and 2) create a new compute package that will use that template.
The divcfg repository includes templates for the following scenarios:
If you need a different system, looking at those examples should get you started toward making your own. To take a quick example, using singularity on SLURM combines the basic SLURM script template with these lines to execute the run in container:
singularity instance.start {SINGULARITY_ARGS} {SINGULARITY_IMAGE} {JOBNAME}_image
srun singularity exec instance://{JOBNAME}_image {CODE}
singularity instance.stop {JOBNAME}_image
This template uses a few of the automatic variables defined earlier (JOBNAME and CODE) but adds two more: {SINGULARITY_ARGS} and {SINGULARITY_IMAGE}. For the image, this should point to a singularity image that could vary by pipeline, so it makes most sense to define this variable in the pipeline_interface.yaml file. So, any pipeline that provides a container should probably include a singularity_image attribute providing a place to point to the appropriate container image.
Of course, you will also need to make sure that you have access to singularity command from the compute nodes; on some clusters, you may need to add a module load singularity (or some variation) to enable it.
The {SINGULARITY_ARGS} variable comes just right after the instance.start command, and can be used to pass any command-line arguments to singularity. We use these, for example, to bind host disk paths into the container. It is critical that you explicitly bind any file systems with data necessary for the pipeline so the running container can see those files. The singularity documentation explains this, and you can find other arguments detailed there. Because this setting describes something about the computing environment (rather than an individual pipeline or sample), it makes most sense to put it in the DIVCFG environment configuration file. The next section includes examples of how to use singularity_args.
To add a package for these templates to a DIVCFG file, we just add a new section. There are a few examples in this repository. A singularity example we use at UVA looks like this:
singularity_slurm:
submission_template: templates/slurm_singularity_template.sub
submission_command: sbatch
singularity_args: --bind /sfs/lustre:/sfs/lustre,/nm/t1:/nm/t1
singularity_local:
submission_template: templates/localhost_singularity_template.sub
submission_command: sh
singularity_args: --bind /ext:/ext
These singularity compute packages look just like the typical ones, but just change the submission_template to point to the new containerized templates described in the previous section, and then they add the singularity_args variable, which is what will populate the {SINGULARITY_ARGS} variable in the template. Here we've used these to bind (mount) particular file systems the container will need. You can use these to pass along any environment-specific settings to your singularity container.
With this setup, if you want to run a singularity container, just specify --compute singularity_slurm or --compute singularity_local and it will use the appropriate template.
For another example, take a look at the basic localhost_container.yaml DIVCFG file, which describes a possible setup for running docker on a local computer:
compute:
default:
submission_template: templates/localhost_template.sub
submission_command: sh
singularity:
submission_template: templates/localhost_singularity_template.sub
submission_command: sh
singularity_args: --bind /ext:/ext
docker:
submission_template: templates/localhost_docker_template.sub
submission_command: sh
docker_args: |
--user=$(id -u) \
--env="DISPLAY" \
--volume ${HOME}:${HOME} \
--volume="/etc/group:/etc/group:ro" \
--volume="/etc/passwd:/etc/passwd:ro" \
--volume="/etc/shadow:/etc/shadow:ro" \
--volume="/etc/sudoers.d:/etc/sudoers.d:ro" \
--volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" \
--workdir="`pwd`" \
Notice the --volume arguments, which mount disk volumes from the host into the container. This should work out of the box for most docker users.