Lab 2: Exploring Chatbots
Due Wed. Feb28th in class

Introduction

Using Open AI's ChatGPT version 3.5, we asked it to respond to the following prompt:

Assume you are current a student at the small liberal arts institution Swarthmore College [who is interested in X]. Write a short paragraph explaining why you chose to attend Swarthmore College.

The portion in square brackets was an optional addition we might include where X was for example athletics, the arts, social justice, etc.

We also asked several current Swarthmore students to explain why they chose to attend Swarthmore College.

Consider the five options below. At least one of them is from ChatGPT and at least one of them is from a current student. Your goal is to figure out which one(s) were written by actual students and which one(s) were written by ChatGPT. NOTE: You should not try testing ChatGPT again on a similar prompt. Just use the responses provided to make your determinations.

Responses

A: I chose to go to Swarthmore because I was broadly looking for a small liberal arts college on the east coast and it best checked the boxes most important to me. It had both strong humanities and STEM (as evidenced by the focus placed on its Engineering Department), it had a focus on Quaker values, and it had a community enthusiastic about working hard on the things they are passionate about. Most importantly it had this passion without seeming to have a competitive spirit. Also the campus seemed just as lovely as the people.

B: I picked Swarthmore College mainly because it's got this awesome mix of challenging academics, a tight-knit community, and a real focus on making a difference in the world. The small classes mean I can really get to know my professors and classmates, and the interdisciplinary approach means I can explore all sorts of interests. Plus, Swarthmore's commitment to social justice is something I totally vibe with. It just feels like the right place for me to grow both academically and personally while making a positive impact.

C: I chose to go to Swarthmore College because my college counselor told me to. I had no specific goal or "dream school." I only knew that I had to get into a "good school," whatever that means. My college counselor recommended me to look into Swarthmore College, as I had mentioned that I like smaller schools that are close to nature yet not too far from cities. I visited campus to check it out and I really liked it!

D: Being a recruited athlete, Swarthmore gave me an opportunity few other schools I was recruited by did: to go to a rigorous institution that prioritizes my status as a student without giving up the sport I love. As I continue at Swarthmore, I am astounded by the ways I have found it to be an amazing fit. Through my two majors of Computer Science and English, I feel I have been able to fully take advantage of the liberal arts education in a way I would not be able to at many other institutions. Swarthmore College has and will continue to be formative to who I am.

E: I decided to attend Swarthmore College because it provides an ideal environment for balancing my passion for athletics with a strong academic program. As a soccer player, I was drawn to Swarthmore's commitment to excellence both on and off the field. The college's Division III athletics program offers the opportunity to compete at a high level while still maintaining a focus on academics and personal development. Additionally, the supportive community at Swarthmore ensures that student-athletes have the resources and encouragement needed to succeed in both their sport and their studies. Whether it's through rigorous training sessions, exciting game days, or engaging in team bonding activities, I know that Swarthmore will help me grow as an athlete and as a student.

Your analysis

Once you've read and considered the responses above, fill out this survey to select which ones were written by ChatGPT and which ones were written by a human.

Write a one-page to the following questions:

  1. Of the five responses, pick the one that seems most likely to be human. Explain your reasoning.
  2. Of the five responses, pick the one that seems the most likely to be ChatGPT. Explain your reasoning.
  3. In your own words, and at a high level, explain how large laguage models produce responses to prompts. Be sure to explain how they manage to generate different responses to the same prompt.

Bring a printed copy of your answers to these questions to class next week.

The results are in!

To reveal which responses were written by humans and which were written by ChatGPT check out the results.