QuickStart¶

This QuickStart section is inspired by the online QuickStart option shown on the HTCondor web pages.

This section assumes that:¶

Assumption	Check / Set-up on BCC
HTCondor is running	HTCondor is installed on BCC
You have access to a machine within the pool that may submit jobs, termed a submit machine.	IP Address: `submit.biochem.wisc.edu`
You are logged in to and working on the submit machine	Your username is your UW-Madison `NetID` and password
Your program executable, your submit description file, and any needed input files are all on the file system of the submit machine
You should work in the `/scratch` directory
Your job (the program executable) is able to run without any interactive input. Standard input (from the keyboard), standard output (seen on the display), and standard
error (seen on the display) may still be used, but their contents will be redi-rected from/to files.	We will run a test file that complies with these requirements.

It is also assumed that you know how to converse within the line command and edit simple text files e.g. with text editor nano.

Connect and set-up a working space¶

We’ll follow the above table process.

First, we connect and create a directory in the shared data space.

Use YOUR NetID to connect – represented here as myname

You should open a text-based terminal from your local machine and then issue the connect command: (replace myname with your login name.)

Bash

    $ ssh myname@submit.biochem.wisc.edu

Then move to /scratch and create a directory with your name and another directory within to work with as your project e.g. quickstart.

Bash

    $ cd /scratch
    $ mkdir -p myname/quickstart  
    #replace myname with e.g. YOUR ID
    $ cd myname/quickstart

Create a simple executable test file¶

Using the nano word processor on the cluster or a copy/paste method we now create a file to be executed by HTCondor.

Bash

    $ nano hello.sh

Within the file enter the following:

Text Only

    #!/bin/sh
    echo "Hello World"

If you are using nano, use Ctrl-X to exit from edit mode to save the file.

Now we make sure that the file is executable:

Bash

    $ chmod u+x hello.sh

Create a simple submit file¶

The submit file contains information for running a job and is passed to HTCondor. The following file is a minimal file, more information could be provided to HTCondor for more job control.

Bash

    $ nano hello.sub

Then enter the following text within the file and save:

Text Only

    executable = hello.sh
    should_transfer_files = Yes

    output = hello.out
    error  = hello.err
    log    = hello.log

    queue

Blank lines are ignored.

Upper/Lower case is ignore on the left side of the equal sign.

Line 1 specifies the file to run

Line 2 requests necessary files to be transferred

Line 3 – 5 specify the name of the standard output files

Line 6 places the job in the queue so it can be run.

Tip

The following implicit assumptions are made:
- general-use bash commands are available, *e.g. echo.
- the process will use standard input, standard output and standard error (stdin, stdout, stderr.)

Submit the job¶

We now follow the example with the cascade steps of submitting a file (hello.sub) containing information about an executable (hello.sh) that is calling on a software (echo) that will create some output files (hello.out, etc.) that will be transferred to the local directory when the job is done.

The submit command is as follows:

Bash

    $ condor_submit hello.sub

Check output¶

The job will be queued and executed rather rapidly, transferring output files to the local directory when done:

Bash

    $ ls

Text Only

    hello.err  hello.out  hello.sub
    hello.log  hello.sh

We can verify that the job executed correctly:

Bash

    $ cat hello.out

Text Only

    Hello World

Syntax¶

This is a very simple example.

Clearly, if we were to run this example again the files that we just created would be overwritten (clobbered) by the files created by the new run.

This and many other aspects of job control can be overcome by specific HTCondor command syntax.

For example, unique numbers could be created for the output files with syntax such as:

Text Only

    Error           = logs/err.$(cluster)
    Output          = logs/out.$(cluster)
    Log             = logs/log.$(cluster)

Or

Text Only

    error   = err.$(Process)                                                
    input   = in.$(Process)                                                 
    output  = out.$(Process)

See examples in the manual under the “Submitting a Job” section for details.

For version 8.6 the link is:

http://research.cs.wisc.edu/htcondor/manual/v8.6/2_5Submitting_Job.html

HTCondor version¶

The current version of the HTCondor software running is obtained with the command condor_version:

Bash

    $ condor_version
    $CondorVersion: 8.6.13 Oct 30 2018 BuildID: 453497 $
    $CondorPlatform: x86_64_RedHat7 $

(This is the installed version on BCC as of December 28, 2021.

The most recent version of the manual is always available at http://research.cs.wisc.edu/htcondor/manual/

Conclusion¶

Info

The complete HTCondor manual for version 8.6 is 1128 pages long! Therefore, the virtue of patience needs to be called upon to tackle and master using a cluster running HTCondor!