Due Dates

Due: before 11:59pm, Tues Nov. 9

I’m giving you a long time to complete it because of your project proposal. Also, runs may take a long time, but you can start them or submit them and come back later to see the results.

Overview and Goals

This is a continuation of your Lab 4 assignment that involves running a few some large runs of your program on different systems. The goal is to give you some practice running on different systems, and practice running some large size runs.

You should spend no more than 4 hours total of your time to complete this part (really, do not spend much more than that on this…​just stop, it is not worth it).

With your Lab 4 partner, you will run some large runs of your MPI odd-even sort on the following systems and submit the timed results of a few of the largest runs:

  1. CS lab machines

  2. the Swarthmore Stelka cluster. Submit the results of a few large runs, be careful to not allocate sizes larger than total available RAM (i.e., don’t trigger VM paging)

  3. the XSEDE Bridges-2 cluster. Try out some small tests first to see that it runs correctly, and do not do any runs that have debug printf stmts. Your runs should have no output other than timing and possibly printing out initial size and number Pis).

I encourage you try out some runs on Strelka before you do on XSEDE to get some practice with sbatch and {slum}[slurm batch scripts].

Goals:

  • Remind yourself of the vim editor that you can use when ssh’ed into cs, Strelka, and XSEDE systems.

  • Try out some large-size runs on CS lab machines and two clusters.

  • Practice with slurm batch scheduler, and using sbatch and writing and using slurm batch scripts.

  • Practice with scp to transfer files between remote hosts and CS lab machines.

Unresolved directive in assn_webpage.adoc - include::./details.adoc[]

"Huge" runs on the CS system

Run these late at night or during other times when the CS machines are mostly idle. Do not run during times when classes, labs, or ninja sessions are in the labs, and also avoid times in the evenings when labs tend to be full.

Run some large runs with lots of hosts, large -np values, and large problem sizes (N) on CS lab machines. In particular, do some runs that distribute processes over a large number of nodes and sort large size N.

  • First test out some large size data (N) runs (-np doesn’t have to be that big), and make sure your program doesn’t have deadlock whe the MPI message buffer is smaller than the data size you send.

    • Try your largest sized run (in terms of N and P on a single node) and make sure that its execution does not exceed physical RAM size (run htop on a node running an MPI process(s) and make sure it doesn’t use the swap partition). If your sizes are too big to fit in memory, scale them back a good bit.

    • Also, it is very important that you are certain that your runs finish. Make sure there is no infinite loop in your code that makes your program steal all the cpu cycles on all the lab machines!

  • Next, write a run script(s) to run a few big experiments.

    • Test out your script before running (you can comment out the actual calls to time mpirun …​ to just check the script variable values and its output). Then uncommet to actually run.

    • You do not need to run a whole bunch of runs, just a few different large runs is fine.

  • Finally, start your experiments running (ssh in to start, use screen or tmux, and script, or use at cron to run them), and check back after they should be done to check that they are. This should be a quick 10 minutes of your time (ssh in, check that hosts are up, start your script in a screen or tmux session, then ssh in later to see that it finished).

See running OpenMPI on our system for more information about ssh-agent, mpirun, creating hostfiles, finding machines on our system, and using autoMPIgen.

Use the check_up.sh script to test that all machines are reachable

./check_up.sh hostfilename

You can copy a version of the check_up.sh script that works with hostfiles with or without slot=X annotations with each host here:

cp ~newhall/public/cs87/check_up.sh .

Here is more information about hostfiles in openMPI: about hosts and hostfiles

If you want to limit the number of MPI processes spawned per node specify the number of slots after each host in the hostfile. I have some large hostfiles with different numbers of slots you are welcome to use as starting points:

cp ~newhall/public/cs87/hostfileHUGE* .

Once you have a large hostfile, you can create experiment script of some large size runs using this hostfile.

IMPORTANT: before running a huge set of experiments, make sure your largest size run completes (make sure it does not deadlock and never finishes). If your oddeven sort deadlocks, then see the Lab 4 write-up for some hints about this and how to fix it.

See the Experiment Tools page from Lab 1 for a reminder about useful tools for running experiments, writing scripts, some useful commands/utilities: screen, script, tmux, basch scripts, cron, …​

Using scp, ssh, vim on Bridges-2 and/or Strelka

You will need to use vim to edit files on Bridges-2 and Strelka, and you will need to use scp to copy files between the CS system and Bridges-2 and Strelka. For example, for your XSEDE odd-even sort big runs, you need to scp your oddevensort.c file from your repo on our systemn to Bridges-2. And you may want to scp result and slurm script files from Bridges-2 back to our system. You can scp each file one by one or make a tar file containing all files you want to copy over, and just scp the single tar file. (more info on scp, more info on tar)

Some scp basics (there are specific examples of scp commands in later sections about running on Strelka and Bridges-2):

# scp a file from your CS account onto Bridges-2 from Bridges-2:
on_cs$ ssh you@bridges2.psc.edu
bridges2$ scp you_on_cs@cs.swarthmore.edu:/oddevensort.c .

# remember, you can us mv to move file to a different location if you
# scp it into the wrong directory, for example:
bridges2$ mv oddevensort.c cs87/oddeven/.

# scp a file from Bridges-2 to CS from Bridges-2
bridges2$ scp RESULTSXSEDE you_on_cs@cs.swarthmore.edu:./cs87/labs/Lab04-repo/.

You can also copy from CS either way:

# scp a file from your CS account onto Bridges-2 from CS
on_cs$ scp filename you_at_xsede@bridges2.psc.edu:./cs87/oddeven/.

# scp a file from your Bridges-2 onto CS from CS
on_cs$ scp  you_at_xsede@bridges2.psc.edu:./cs87/oddeven/XSEDERESULTS .

Big Runs on Strelka

Try runs on Strelka before trying some runs on XSEDE. You will need to use vim to edit files on Strelka.

Swarthmore’s Strelka cluster is set up to use the slurm batch scheculer like XSEDE systems. It is also a resource you can use for your course project. You should try out some runs on Strelka before trying some runs on XSEDE. On both Strelka and Bridges-2, you will need to use vim to edit files, and you will use sbatch to submit jobs to slurm.

To get a Strelka account, complete this form, which asks for your SSH public key: Strelka account request

Once you have an account, try out your oddeven sort on Strelka, first with a smaller sized run to see that it works, and then try a couple larger runs.

Strelka (like Bridges-2 uses the slurm scheduler, so the same process you use to compile and run on the XSEDE system, you will use on this system.

Basically, you will use scp to copy over your oddevensort.c file onto Strelka, copy over a Makefile and sample slurm batch scritp. Once this is set up you just compile your code and then you can submit oddevensort job(s) to slurm using sbatch.

Here are the steps:

# ssh in:
$ ssh you@strelka@swarthmore.edu

# create some subdirectory structure the first time:
strelka$ mkdir cs87
strelka$ chmod 700 cs87
strelka$ cd cs87
strelka$ mkdir oddeven
strelka$ cd oddeven

strelka$ pwd           # should be in ~/cs87/oddeven

strelka$ scp you@cs.swarthmore.edu:./cs87/Labs/Lab4-repo/oddevensort.c .

I have a Makefile and an example slurm batch script on strelka that you can copy over and try out (note that the last two characters in my user name are the letter l followed by the number 1, they look very similar):

strelka$ cd ~/cs87/oddeven
strelka$ cd ~/cs87/oddeven
strelka$ cp ~tnewhal1/Makefile .
strelka$ cp ~tnewhal1/oddeven.sb .

# open oddeven.sb slurm batch script in vim to see what the settings
# are (you can change these for larger or smaller sized runs
# and change this to a path in your directory not mine:
# #SBATCH --output=/home/tnewhal1/cs87/oddeven/oddeven.%j.%N.out
strelka$ vim oddeven.sb

Then compile, and submit your job to slurm:

strelka$ make
strelka$ sbatch oddeven.sb

# see queue state (or just your jobs in the queue)
strelka$ squeue
strelka$ squeue -u yourusername

# scancel: delete a job from the queue using its job id (squeue lists job ids)
strelka$ scancel jobid

# sinfo: to see how resources are currently allocated, unallocated, totals:
sinfo -o "%n %T %C"

You may need to add the mpi module into your environment to run mpicc and mpirun. I belive the default version of openMPI is the one you want to use, so you likely DO NOT need to explicitly load this version, but here is some info just in case:

# list available modules
module avail
------------------------- /opt/modulefiles -----------------------
  list of modules
  ...
(L) are loaded

# to load an mpi version into your environment:
module load  openmpi/4.0.2-gcc-8.2.1

If you echo your PATH environment variable you will see the path to this module added to the fromt of your path:

echo $PATH
/opt/apps/mpi/openmpi-4.0.2_intel-19.0.5.281/bin: ...

You can also add this to your PATH in your .bashrc file and then avoid running module load each time you log in.

For more information on Strelka: Strelka cluster

Runs on Bridges-2

Run some large runs of your Lab 4 solution on Bridges-2. You should do this after running some large runs on the CS machines to make sure your solution does not have any deadlock or other errors that would make it run forever. Also, try out sbatch first on Strelka.

Also, you will want to estimate a reasonably accurate upper bound for its runtime in your slurm script. Do not make the time super long, as if your application has deadlock it will use up a lot of XSEDE CUs for no good reason. Pick a reasonable upper-bound estimate (you want to pad it a bit so that it is long enough for your application to finish, but you don’t want a deadlocked process to continue to use CUs for a huge over-estimate of total runtime). Do some experimentation with times on XSEDE and use times on our system to help you pick a good upper bound estimate. Your time doesn’t have to be super close, but if you expect it to easily complete within 15 minutes, submit a slurm script with a time of a few minutes beyond this, maybe 17, and don’t submit one with a runtime of 1 day, for example).

What to do

This is a high-level overview of an order in which to try out some things on XSEDE (details follow):

  • First make sure to set up your XSEDE and your Bridges-2 account (follow all the directions under "XSEDE and Bridges-2 Account Set-up"): XSEDE and Bridges-2 accounts

  • Next, try ssh’ing into Bridges-2, and try out my mpi hello world example. You can scp from CS to your Bridges-2 account.

    bridges2$ cd ~/cs87
bridges2$ scp you_on_cs@cs.swarthmore.edu:/home/newhall/public/XSEDE_MPI.tar .
bridges2$ tar xvf XSEDE_MPI.tar

# then cd into the XSEDE_MPI subdirectory:
bridges2$ cd XSEDE_MPI

    Then open the README file in vim and (follow the directions for how to submit jobs. Also look here: using XSEDE Bridges-2

  • Once you have tried out the helloworld, then practice with some small runs of oddevensort before submitting larger runs.

    • Once you figured out slurm and how to submit jobs with helloworld, then scp over your Lab04 solution (details: scp, slurm and Makefile ), and build on Bridges-2.

    • Then, try out my slurm batch script, or write your own for some small runs and try running. You can try a very few very small runs with some debugging (don’t do all) printing to make sure it runs on Bridges-2.

    • Finally, try a small run with printing disabled in preparation for larger runs (remove all debug output from your program, comment out #define DEBUG, except that you can keep in printing out timing information.

    • Next, submit some larger runs.

scp, slurm and Makefile

On {xseddsys} you will need to scp over your oddevensort.c file, and copy over a slurm batch scritp and Makefile from my account on Bridges-2.

First, scp over your oddevensort.c file:

on_cs$ pwd       # get path to your Lab 4 repo
on_cs$ ssh you@bridges2.psc.edu

bridges2$ mkdir cs87
bridges2$ chmod 700 cs87
bridges2$ cd cs87
bridges2$ mkdir oddeven
bridges2$ cd oddeven
bridges2$ scp you_on_cs@cs.swarthmore.edu:/oddevensort.c .
# example:
       scp newhall@cs.swarthmore.edu:./cs87/labs/Lab04-tia/oddevensort.c .

On Bridges-2, I have a Makefile and a slurm script (for a small run) you can copy into your directory to use to compile and run oddevensort on Bridges-2, and then try submitting an oddevensort job by running sbatch with the slurm script you copied over: submit to the run queue:

bridges2$ cp ~newhall/cs87/README .
bridges2$ cp ~newhall/cs87/Makefile .
bridges2$ cp ~newhall/cs87/oddeven.sb .

Or you can scp these from my CS machines:

bridges2$ scp you@cs.swarthmore.edu:/home/newhall/public/cs87/lab4/README_bridges2 .
bridges2$ scp you@cs.swarthmore.edu:/home/newhall/public/cs87/lab4/Makefile_bridges Makefile
bridges2$ scp you@cs.swarthmore.edu:/home/newhall/public/cs87/lab4/oddeven_bridges2 oddeven.sb

The README file includes directions for compling and running. Open it and follow the directions:

vim README

These include

  • edit the path name in oddeven.sb to be the pathname to your oddevensort executable file.

  • load the openmpi module (once), and then make to compile:

    bridges2$ module load openmpi/3.1.6-gcc10.2.0

bridges2$ make

  • run your oddevensort by submitting your oddeven.sb batch script to slurm by running sbatch:

    bridges2$ sbatch oddeven.sb

  • you can see the status of your job in the run queue:

    bridges2$ squeue -u yourusername

On Bridges-2, you will need to edit the oddeven.sb slurm file using vi (and create additional .sb files) to submit large runs of your oddeven sort to sbatch:

bridges2$ vim oddeven.sb   # change something like number of nodes
bridges2$ sbatch oddeven.sb

Assignment: try some long runs of your solution on Bridges-2

Try out at least two large runs of your Lab 4 solution on Bridges-2.

  • Make sure to disable or remove all debug output from your program (comment out #define DEBUG).

  • Copy over your oddevensort.c and oddeven.sb (if you didn’t already do it with practicing scp above): Using scp, ssh, vim on Bridges-2 and/or Strelka

    On Bridges-2, I have a Makefile you can copy to compile oddevensort on Bridges-2, and then try submitting an oddevensort job to the default queue by running sbatch with the slurm script you copied over: submit to the run queue:

    bridges2$ cp ~newhall/Makefile .
bridges2$ make
bridges2$ sbatch oddeven.sb

  • Try out some small runs using example slurm script. With the starting point code is an example slurm script for running on Bridges-2 (it is for a small run on the debug queue). Only try running this with small sized problems. To submit to the debug queue on Bridges-2:

sbatch oddeven.sb

  • write a couple slurm submission scripts for long runs (large sizes), submit them. In your slurm script, you will want to modify at least these four lines (use the compute queue instead of the debug queue for your experiment runs):

#SBATCH --partition=RM         # which queue
#SBATCH --nodes=2              # Total number of nodes
#SBATCH --ntasks-per-node=24   # Total number of mpi tasks
#SBATCH -t 00:30:00            # Run time (hh:mm:ss) - 30 mins

  • You should choose way more than 2 nodes and 24 mpi tasks in your runs.

  • You can also try large sized arrays for each process to sort via command line args to your executable (add them to the command in the slurm script).

  • You may need to adjust the estimated runtime (30 mins in this example). If your estimate is too small and your program runs longer than your estimate, it will be killed before it completes. If your estimate is too long, it will wait in the job queue for much longer than it should.

  • You should also submit to a regular job queue (e.g. <tt>compute</tt>) for the big runs (don’t use the debug queue).

Submit

You will submit the following via git:

  1. BIGRESULTS: add this file (like RESULTS) that contains results of your large run tests on the CS machines. make sure to edit this file so that it is clear what run sizes the results you show are from

  2. Two Strelka output files from two large runs. The only program output should be that the process with rank 0 should print out the size of the problem: N and P (make sure you have no debug printing output in these runs…​these files should be very small). If you forget to include this printing of the problem size N and P, then edit the output file to include these values (copy in the slurm script corresponding to the run, or just edit the output file to add this in).

  3. Two Bridges-2 output files from two large runs. The only program output should be that the process with rank 0 should print out the size of the problem: N and P (make sure you have no debug printing output in these runs…​these files should be very small). If you forget to include this printing of the problem size N and P, then edit the output file to include these values (copy in the slurm script corresponding to the run, or just edit the output file to add this in).

    vim XSEDEResults
# in vim can import file contents using :r  in ESC mode
# put cursor where you want file inserted:
:r output.jobnum....
:r oddeven.sb        # this job's slurm script
:r output.jobnum....
:r oddeven2.sb       # this job's slurm script
...

    You can scp over the XSEDEResults file to your cs account and add them to your repo:

    bridges2$ scp filename you@cs.swarthmore.edu:./cs87/Labs/Lab04-you-partner/.

# or scp to home and then on CS just mv file from your
# home directory into your Lab04 repo:
bridges2$ scp filename you@cs.swarthmore.edu:.
cs$ cd ~/cs87/labs/Lab04-you-partner
cs$ mv ~/filename .

  4. A couple output file from running on on the Swarthmore Strelka cluster. Similar to XSEDE result ouput.

Then just add these to your git repo and commit and push:

$ cd ~/cs87/labs/Lab04-you-partner
$ git add BIGRESULTS
$ git add XSEDEResults
$ git add StrelkaResults
$ git commit -m "lab 4b results"
$ git push

Before the Due Date, one of you or your partner should push Bridges-2 output files from two large runs to your Lab04 repo. (ssh them over to your cs account to git push them to your Lab04 repo).

If you have git problems, take a look at the "Troubleshooting" section of the Using git page.

Handy Resources