{ "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Cluster & HTCondor\n", ">Biochemistry laboratories - 201\n", ">\n", ">Jean-Yves Sgro -jsgro@wisc.edu\n", "\n", "\n", "\n", "
\n",
    "\n",
    "\n",
    "
\n", "\n", "\n", "*Note*: To see as slides click on **\"Gift icon\"** at the top right. To revert back to page view click on **\"book icon\"** when on the first slide.\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Set-up\n", "\n", "- check your name on the attendees list\n", "- choose an iMac\n", "- login with your NetID ( @wisc.edu is not necessary)\n", "- If this is the first time this Mac \"sees\" you it will go through a quick set-up: simply skip using \"Apple ID\"" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Finding Terminal on the iMac\n", "\n", "
\n", "\n", "## Method 1\n", "\n", "- Click on the \"Finder\" at the bottom left of the **\"Dock\"** on the bottom of the screen. This will open a new window.\n", "- On the left click on **\"Applications\"**\n", "- In the alphabetical list open folder **Utilities**\n", "- double click to open **Terminal**\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Method 2\n", "\n", "- *Spotlight Search*: Click the magnifying glass at the top right corner of your screen \n", "- type `Terminal` \n", "- press the `return` key\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Cluster & HTCondor\n", "\n", "The Biochemistry Computational Cluster (BCC) is a High Throughput Computing (HTC) environment within the UW-Madison Biochemistry Department.\n", "```\n", " submit_node\n", " |\n", " |-[2TB disk]\n", " -------------------------\n", " | | | | | | | | |\n", " N1 N2 N3 N4 N5 N6 N7 N8 N9 \n", "```" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# HTCondor\n", "\n", "* HTCondor is a “scheduler” system that dispatches compute jobs to 1 or more \"compute nodes.\"\n", "\n", "* Using HTCondor is the only approved method for performing high throughput computing on the BCC Linux cluster. \n", "\n", "* Jobs have to be ready to be processed by HTCondor as jobs cannot be interactive on the cluster. \n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Cluster access overview\n", "\n", "1.\tText-based access only\n", "1.\tNo graphical interface\n", " 1.\tLogin info `ssh myname@submit.biochem.wisc.edu`\n", "1. Linux OS: Scientific Linux\n", "1.\tVPN access\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Process\n", "\n", "Submit a \"job file\" that contains information on how to run the software\n", "\n", "* submit file: `myjob.sub`\n", "* executable: `myjob.sh`\n", "\n", "Information about software and file transfer\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Getting ready\n", "\n", "Invert the process:\n", "\n", "* define software and files (with dependecies)\n", "* create script to run \"hands-off\"\n", "* create submit file" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true, "slideshow": { "slide_type": "slide" } }, "source": [ "# QuickStart\n", "\n", "Login - [replace *myname* with your login name.]\n", "\n", "```\n", "ssh myname@submit.biochem.wisc.edu\n", "```\n", "Then move to `/scratch` and create a directory with your name and another directory within to work with.\n", "\n", "```\n", "$ cd /scratch\n", "$ mkdir myname #replace myname with e.g. YOUR ID\n", "$ mkdir quickstart\n", "$ cd myname/quickstart\n", "```\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Create an simple executable test file\n", "Create execute file (*e.g.* with `nano`)\n", "\n", "```\n", "$ nano hello.sh\n", "```\n", "\n", "File content:\n", "\n", "```\n", "#!/bin/sh\n", "echo \"Hello World\"\n", "```\n", "\n", "Make file executable:\n", "\n", "```\n", "$ chmod u+x hello.sh\n", "```" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Create a simple submit file\n", "\n", "```\n", "$ nano hello.sub\n", "```\n", "File content:\n", "\n", "```\n", "executable = hello.sh\n", "should_transfer_files = Yes\n", "\n", "output = hello.out\n", "error = hello.err\n", "log = hello.log\n", "\n", "queue \n", "```\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Submit job\n", "\n", "```\n", "$ condor_submit hello.sub\n", "```\n", "\n", "# Check output\n", "```\n", "$ ls\n", "```\n", "```\n", "hello.err hello.log hello.out hello.sh hello.sub\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# On-line resources\n", "\n", "| Resource | Link | \n", "| -------- | ---- |\n", "| HTCondor Quick Start Guide | http://research.cs.wisc.edu/htcondor/manual/quickstart.html |\n", "| Complete manual\\* | http://research.cs.wisc.edu/htcondor/manual/ |\n", "\n", "\\*You can check which manual you need by checking which version of HTCondor is installed with command: **`condor_version`**\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] } ], "metadata": { "anaconda-cloud": {}, "celltoolbar": "Slideshow", "kernelspec": { "display_name": "Bash", "language": "bash", "name": "bash" } }, "nbformat": 4, "nbformat_minor": 0 }