The following is information, tutorials, and help links for tools that we will use in for remote course activities and course work. More course-specific information is available off the CPSC 66 webpage, and from individual assignment pages.

Remote Access and Code Environment Tools

SSH and SCP

ssh is useful for remotely logging into the CS lab machines. scp is useful for copying files between your local system and the CS system.

ssh

You can remotely connect to any machine by sshing in as you to lab.cs.swarthmore.edu or to a particular machine by logging in as you to a specific machinename.cs.swarthmore.edu. For example:

# log into some lab machine:
ssh username@cslab.cs.swarthmore.edu

# log into owl:
ssh username@owl.cs.swarthmore.edu

You can find the names of lab machines here: cs lab machines

scp

scp is like the cp command, but for copying files between two different machines.

# copy a file named "bio.txt" from the "{labpath}/lab01" directory on a lab machine to your machine
scp yourusername@cs.swarthmore.edu:{labpath}/lab01/bio.txt .

# copy a file named "bio2.txt" from current directory on your machine onto a lab machine
# Note: by default, this will place the file in your home directory; i.e.,  "/home/yourusername"
scp bio2.txt yourusername@cs.swarthmore.edu:

# copy a file named "bio3.txt" from current directory on your machine onto a lab machine in the "{labpath}/lab01" directory
scp bio3.txt yourusername@cs.swarthmore.edu:{labpath}/lab01

Additional Links

Finding Good/Available Machines

If you are ssh’ed into a lab machine and notice that it is running slowly or not responding well, you will want to find a better machine to use.

Here are few utilities to see what is running on a machine and to help you find another machine that will respond better and ssh into that one instead.

Getting Usage Info on a Machine

First, you can run who to see if there are a lot of other users logged into the same machine as you are:

who

If there are a lot of users logged into a machine, it may be one that could become overloaded. If all the users are idle, however, it will not be.

A better way to see real cpu and memory load on a machine is to run top and htop. htop has nicer output, so we recommend that:

htop

htop shows CPU usage on each core, memory usage, and swap usage. If the CPU and/or memory usage is high, and particularly if the machine is Swapping (Swap is active), you will want to pick another lab machine to use. Here is some more information about using top and htop

Tools for Finding a Good Machine(s)

smarterSSH can be used to ssh into a machine that has a lot more available RAM and CPU resources, and should perform well. Running it directly will ssh you in to a good machine. Alternatively, you can run it using the -i command line option to list the top machines. Then, log out of your ssh directly ssh into one of the machines listed.

Here is the way you may want to run smarterSSH:

smarterSSH -i  -v -n 20

This lists the best 20 machines in terms of available RAM and CPU. Choose one of the machines listed to ssh into (you don’t always want to pick the top one, as maybe others are also choosing it, but if you select one in the top 10 or so, it should perform well)

Here are some other sample line options for using smarterSSH:

# directly ssh's you into one of the best machines
smarterSSH

# list the command line options
smarterSSH -h

# lists top 50 machines based on their available RAM
# probably want the default that orders based on a function of RAM and CPU
smarterSSH -i  -v -n 50 -m

# lists top 50 machines based on their available cpu
smarterSSH -i  -v -n 50 -c

System Dashboard for Machine Stats

You could also look at the status dashboard off the CS department machines page to try to find a machine that looks less loaded.

Tools for Lab Work

EdStem and GitHub

We will use EdStem (an alternative to Piazza) for Q&A and Github for lab assignments. Here are the links to the CPSC 66 EdStem and Github orgs:

Git

Git and the college’s Enterprise GitHub are used for lab work.

  • Using Git for CS labs contains some basic (and verbose) git instructions for using git for CS course lab work. It is written using CS31 as an example, but replace with your own course number.

  • Git on our system and git help. This is a more complete reference for using git on our system.

Using Jupyter Labs

We will be using Jupyter Labs (similar to Jupyter Notebooks) for lab assignments this semester. Before getting started, please familiarize yourself with the Jupyter Lab interface.

Pick a Machine

First, select a machine to log in to. You can ssh into cslab.swarthmore.edu or use smarterSSH to have a machine picked for you. However, you may want to pick a machine based on resources, particularly if you have compute-intensive final projects. You can view machines on the CS network and their resources here. You can sort by the attributes (e.g., sort by Graphics card if you are looking for high-powered GPUs for your final project).

For the labs we design, you should not need to worry about resources except to make sure you are not using a machine currently in heavy use. Use tools like htop to see if a machine is busy.

Running Jupyter Labs Remote (CS) to Local (Your Home Machine)

Here we describe a set of steps to run Jupyter Labs on your machine that is being hosted on a remote CS machine. That is, Jupyter Labs will run on your local machine but the program will be executing on a CS machine. Aside from getting Jupyter going you can rely on the stable, fully supported environments we create so that you do not have to install specialized packages or rely on your machine’s computing resources.

  1. One partner should log into a machine. Below, we assume that you are using cslab.cs.swarthmore.edu to pick a machine for you. Use your normal ssh process, but add -L localhost:XXXX:localhost:XXXX here so that you create a connection and create a port tunnel. For example:

    ssh -L localhost:XXXX:localhost:XXXX yourUserID@machineName.cs.swarthmore.edu

# for example, if a student asas wants to use port 6600 on cslab
ssh -L localhost:6600:localhost:6600 asas@cslab.cs.swarthmore.edu

    Be sure to use your assigned port to prevent conflicts with another lab group.

  2. Now that you are logged into a CS machine, run Jupyter Lab on the remote (CS) machine you have logged into. Use the 4 digit port ID assigned you (this is posted with the lab writeup). For example, you would use --port=6021 if you were assigned port 6021:

    $ source /usr/swat/bin/ML # loads python environment for this course
$ jupyter lab --no-browser --port=XXXX # start jupyter lab, but don't have it open a browser yet.  Use your 4 digit port ID

    It is important to read the output messages to see if the port XXXX did not work. This can happen if another application is already using the port (e.g., another group entered your ID by accident, or you have two instances of Jupyter Lab running). The script will still work but you will get a message that it used the next available port (likely 1 higher). If this happens, the current set up will not work and you should restart or, alternatively, ssh in using a second terminal but replacing the port number with the one actually assigned.

    Look for something like the following in the output (you will need to copy and paste these lines in a few steps):

    [I 2021-01-24 16:42:22.147 ServerApp] Jupyter Server 1.2.2 is running at:
[I 2021-01-24 16:42:22.148 ServerApp] http://localhost:XXXX/lab?token=...

    The line with http://localhost:XXXX/lab?…​ is the key line. It has the port number (should be the same one you were assigned unless it was unavailable) and the url you’ll use later. You have now set up your remote connection.

  3. Finally, go back to the first step where we told you to note the server address (http://localhost:XXXX/lab?token…​). Copy and paste that whole URL (including the token, which is a required passcode generated for security purposes) into the browser on your local machine. You should now see your files as they appear on your CS machine and you can start using Jupyter Labs.

  4. If you need a second terminal, you can open one up in Jupyter Labs directly. Go to File > New Launcher and then select Terminal under Other. You can use this terminal to run your git commands (git pull to start, git add, git commit, and git push when you want to checkpoint your code). *Remember: only add the files you want to commit. Do not use git add * or git add .; this will cause lots of hidden, temporary, or binary files to be committed which is bad practice and likely to lead to slower git pulls and more merge conflicts.

  5. If you are working with a partner, we recommend using a screenshare (e.g., on Zoom). Your lab partner can also tunnel to the Jupyter Lab server. Follow step 1, but use your own userID instead of the your partners. Also, make sure you use the specific machine your partner was assigned (not cslab.cs.swarthmore.edu, this is an alias that assigns a machine). Note that both cannot edit at the same time; one partner needs to save, then wait a few seconds, then the other partner needs to do a refresh for the most recent changes to show up. Use it to quickly switch who’s "driving," but don’t use it as an alternative to screen sharing with Zoom.

  6. Do not share your token with anyone outside your lab group (other than an instructor), as it is equivalent to giving them direct access to your code.

  7. When you are done, be sure to kill the server on your remote machine. You can enter CTRL+C to kill the process.

  8. Note that you may need to adapt this slightly if you’re not using the command line version of ssh; for example, if you’re using PuTTY on Windows, you’ll need to go to Connection→SSH→Tunnels, where you’ll put your port number (XXXX in the example above) into both the "source port" and "destination" fields, then click the "Add" button.

(Deprecated) Jupyter setup with separate ssh logins

You may want to use the setup below if you want to pick your machine after sshing in or if you want to connect to your partner’s machine.

  1. Run Jupyter Lab on the remote (CS) machine you have logged into. Use the 4 digit port ID assigned you (this is probably posted with the lab writeup). For example, you would use --port=6021 if you were assigned port 6021:

    $ source /usr/swat/bin/ML # loads packages for this course
$ jupyter lab --no-browser --port=XXXX # start jupyter lab, but don't have it open a browser yet.  Use your 4 digit port ID

    It is important to read the output messages to see if the port XXXX did not work. This can happen if another application is already using the port (e.g., another group entered your ID by accident, or you have two instances of jupyter lab running). The script will still work but you will get a message that it used the next available port (likely 1 higher). You will need to know this port number for the next step.

    Look for something like the following in the output (you will need to copy and paste these lines in a few steps):

    [I 2021-01-24 16:42:22.147 ServerApp] Jupyter Server 1.2.2 is running at:
[I 2021-01-24 16:42:22.148 ServerApp] http://localhost:XXXX/lab?token=...

    The line with http://localhost:XXXX/lab?…​ is the key line. It has the port number (should be the same one you were assigned unless it was unavailable) and the url you’ll use later. You have now set up your remote connection.

  2. On your local machine (i.e., your laptop) you will now need to make a connection to the server you set up in the previous step. Open up a terminal and enter the following command:

    ssh -N -f -L localhost:XXXX:localhost:XXXX yourUserName@machine.cs.swarthmore.edu

    The items in bold may need to be changed:

    • XXXX: this is the port you used above. Our command defaulted to the one assigned to you, but use the one actually picked in the previous step if there was a conflict.

    • yourUserName: your CS login ID.

    • machine: the name of the machine you logged into.

      This is the most error prone step. Double-check you are using the correct machine name on the CS side (e.g., olive). If you are logging in to lab.cs.swarthmore.edu or cslab.cs.swarthmore.edu you will get a random machine and not the one your server is running on.

      If you get an error that the port is already in use, it probably means your computer still has the port assigned from a previous time you ran the above process. You can find and kill that process, though this depends on your machine. For MacOS and Linux, run:

      $ kill $(lsof -ti:XXXX) # lsof lists the processes using port XXXX. kill will end that process

# to see a description/make sure this worked, you can list the process first
$ lsof -i tcp:6000
COMMAND   PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
ssh     23134 ....
# ssh with process ID 23134 is holding on to port 6000

      Here are details for a Windows machine.

  3. Finally, go back to the first step where we told you to note the server address (http://localhost:XXXX/lab?token…​). Copy and paste that whole URL (including the token, which is a required passcode generated for security purposes) into the browser on your local machine. You should now see your files as they appear on your CS machine and you can start using Jupyter Labs.

Running a Local Installation of Jupyter Labs

Alternatively, you can install Jupyter Labs on your local machine and run your labs locally so that you are not dependent on a connection with a CS machine. For most of our labs, your local machine will have enough compute power. However, we can only provide minimal troubleshooting support. Use Ed to post questions so we and your classmates can help you if you have a problem.

Here are different ways to install Jupyter Labs. We highly recommend using Anaconda (conda). You’ll also need to install Git tools on your machine so that you can clone and push to your Git repo. You’ll also need to install all dependencies - Python, numpy, matplotlib, etc. So only go down this path if you have a decent chunk of time and patience to spare.

Unix Basics

For course work you should set up a course-specific subdirectory in your home directory. First read the resource about ssh about remotely connction to the CS system, then create ssh into our system and create a subdirectory for {coursepath}:

# from home ssh into a cs lab machine:
ssh <yourusername>@labs.cs.swarthmore.edu

# on the CS system:
$  cd                 # change current working directory to your home directory
$  pwd                # print current working directory
$  mkdir {coursepath}         # create a new subdirectory named {coursepath}
$  ls                 # list contents of current working directory (your home directory)
$  cd {coursepath}            # change current working directory to {coursepath}
$  pwd                # print current working directory

Using Unix and Useful Commands and Resources:

Remote Lecture, Lab, and Office hours

These are tools that we will use for lecture, lab mettings and office hours in CPSC 66:

Zoom

You will receive an email with information about how to find our class zoom link before the start of class.

There are instructions for using zoom off the college’s Working Remotely page.

Try it out:

Set-up and try out a zoom meeting with your lab partner or project group members

  1. one of you should create a zoom meeting for your lab work:

    • Name the meeting something like labXmeeting where X is the number of your lab assignment

    • Make it a Recurring meeting with the Recurrence→No Fixed Time option.

    • Enable join before host in the Meeting options

    • Save and send your lab partner the zoom link (Join URL associated with your meeting or copy the invitation link)

  2. contact each other to find a time for a 10 minute practice zoom meeting.

  3. at the time you arranged to meet, join your zoom meeting (because "join before host" is selected any of you can start the meeting).

Slack

Joining Slack

You will receive an invitation to join the class slack page prior to the start of the semester.

Navigating Slack

Slack breaks up workspace into channels, which are listed in the sidebar on the left. Some channels we will use in this course are:

  • '#general' for general announcements about the class, you can select pinned announcements to find links to the course zoom page

  • '#office-hours' for class office hours. You can send a quick message to the professor here and start a zoom session to which s/he will attach.

  • '#lab-X' we will often move to slack after starting each lab session on the course zoom page. Here you can get individual help on your lab work, and we will check in with each lab group. You can type in short questions, but typically we will have you start a /zoom session to which we will attach.

To start a zoom session in slack:

First choose the slack channel for the type of session (lab or office hours), then you can start a zoom session to which the professor/lab instructor/TA/ninja will attach and help you with your specific question(s) or just check in with you. To start a zoom session just type '/zoom'. If you are working on a partnered lab one of you and your partner can start, and the other can join.

In zoom you can share your screen to display and run your code, or display your homework to your instructor.

Accessibility

The course staff is committed to making all course material accessible to you. To access Slack’s accessibility settings, start by clicking on the workspace name at the top of the navigation pane on a laptop or desktop.

Then from the dropdown menu choose “Preferences”, then select “Accessibility” from the navigation bar on the left.

Options include:

  • Content magnification (“Zoom”). Allows you to enlarge content shown in Slack

  • Animation. Please uncheck this box if you are sensitive to animated images.

  • Keyboard. Options for non-mouse navigation.

To direct message someone:

By default, messages posted on Slack are visible to all members of the workspace. But for smaller conversations, you can use Slack’s direct messaging (DM) feature to start a private conversation with one person or a small group.

Code of Conduct

The course Slack workspace is accessed by a wide variety of people all working in different environments and timezones. We ask that you be respectful of the staff as well as your fellow students. We expect you to follow these general rules:

  • Do not use Slack to harass faculty, staff, or other students.

  • Do not spam any channels or direct messages with garbage. Try to make posts in academic channels concise.

  • Follow channel guidelines. E.g. do not post idle conversations in the #lab-X channel.

  • Respect others’ time. When you are in lab, or in office hours, try to respond promptly to messages directed to you.

  • Note that any messages you post in a channel will be visible to everyone in that channel. Think before you post!

If there is a problem where someone is violating the code of conduct or otherwise failing to respect others, please raise the issue to one of the class professors or instructors.

Slack Links

Here is some documentation on using slack

Here is some documentation on using zoom in slack (scroll past the how to install zoom to "Use Zoom in Slack" part)

Gradescope For Homeworks and Exams

For class Homework assignments and Midterms, we will be using Gradescope. We have posted a video tutorial. to explain the interface. Note that the video is specific to a different course, but the same ideas apply here. In the video, we briefly run through a few features and demonstrate how you will use it for an assignment. Here are the complete instructions for reference later:

  1. Log in to Gradescope

  2. Under Your Courses select CPSC 066. Any open homeworks and exams will be viewable here.

  3. Select the current homework/exam you want to complete by clicking on the Name field (e.g., Homework 1: KNN and Decision Trees).

  4. You may now enter your answers using the provided text boxes. For certain questions, you will have an upload option if you want to write/draw your response and upload an image or PDF. Your submission must be legible to be graded. This is a perfectly acceptable option if it is more effective and/or efficient for you.

  5. Be sure to save answers periodically by clicking Save next to questions as you complete/update your responses. You can also click Save All Answers at the bottom of the page. Don’t worry about this being final - you can edit all answers up until the deadline.

  6. When you are done, click on Submit & View Submission. You may edit and resubmit up until the deadline by going to the assignment and clicking Resubmit in the bottom right corner. View Submission History if you want to revert to an older version.

Some additional features:

  • Gradescope allows the use of Markdown and LaTeX (using double dollar signs to surround items in Math mode, e.g., $$e=mc^2$$). This is not required.

  • For homeworks, you will be able to work in groups. One person will control the group submission - pick this person before beginning. All members of the group are expected to have worked on all problems together - do not "divide and conquer". For Homework 1, you can work in groups up to size 4.

    1. To select group members, you will first need to click on Submit & View Submission at the bottom to get to the screen. You can do this at any time (remember, submitting is not final, you can still work on your responses).

    2. After hitting Submit & View Submission, you will see information about the submission to the right. Click on View or edit group under GROUP. Alternatively, you can click Group Members at the bottom of the submission screen to edit your list of group members.

    3. Select all group members under the Add Student dropdown menu. Click Add when done.

      • It is an academic integrity violation to work with other groups or members of other groups to come up with solutions. If you decide not to work in a group anymore after solving a few problems, you will need to work on your own or with a subset of the original group.

  • We will grade and post grades on Gradescope. In addition, solutions for Homeworks will be viewable there after the deadline.

  • We are new to Gradescope, so please let us know if something is not working as expected.

  • Exams will use Gradescope in a similar format, except that you cannot communicate with anyone other than the instructors, and the exam will be timed.

(specific links for each tool are included above)