Write a function called get_words that takes a string s as its only argument. The function should return a list of the words in the same order as they appeared in s. Note that in this question a "word" is defined as a "space-separated item". For example:

Hint: If you don’t know how to approach this problem, read about str.split().

Write a function called count_words that takes a list of words as its only argument and returns a dictionary that maps a word to the frequency that it occurred in s. Use the output of the get_words function as the input to this function.

Notice that this is somewhat unsatisfying because the is counted separately from The. We can easily fix this by lower-casing all of the words before counting them using str.lower:

Write a function called words_by_frequency that takes a list of words as its only argument. The function should return a list of (word, count) tuples sorted by count such that the first item in the list is the most frequent item. The order of items with the same frequency does not matter (but you could try to sort such items alphabetically if you wanted to try that). You should use your count_words function to get the counts first.

In the /data/cs91r-s25/gutenberg/trimmed/ directory there are a number of .txt files containing texts found in the Project Gutenberg collection.

In python, read the contents of Lewis Carroll’s "Alice’s Adventures in Wonderland", which is stored in the file carroll-alice.txt. Use your words_by_frequency and count_words functions from Section 1 to explore the text. Assuming that you wrote your code properly in Section 1, and assuming that you lower-cased all of the words in the text, you should find that the five most frequent words in the text are:

We now see that there is a deficiency in how we implemented the get_words function. When we are counting words, we probably don’t care whether the word was adjacent to a punctuation mark. For example, the word hatter appears in the text 57 times, but if we queried the count_words dictionary, we would see it only appeared 24 times. However, it also appeared numerous times adjacent to a punctuation mark so those instances got counted separately:

Our get_words function would be better if it separated punctuation from words. We can accomplish this by using the re.split function. We used the re library last lab to create our first regular expressions. The re.split function is more powerful than python’s str.split function because it allows you to split a string based on a regular expression match rather just a literal string.

Be sure to import re at the top of your file to make re.split work. Below is a small example that demonstrates how str.split works on a small text and compares it to using re.split:

Note that this is not exactly what we want, but it is a lot closer. In the resulting list, we find empty strings and spaces, but we have also successfully separated the punctuation from the words.

Using the above example as a guide, write and test a function called tokenize that takes a string as an input and returns a list of words and punctuation, but not extraneous spaces and empty strings. You don’t need to modify the re.split line we provided: just process the resulting list to remove spaces and empty strings.

Use your tokenize function in conjunction with your words_by_frequency function to list the top 5 most frequent words in carroll-alice.txt. You should get this:

Write a function called filter_nonwords that takes a list of strings as input and returns a new list of strings that excludes anything that isn’t entirely alphabetic. You can use the str.isalpha method to determine is a string is comprised of only alphabetic characters. If you are more daring, try using a regular expression containing the special alphanumeric set represented by \w (and explained above).

Use this function to list the top 5 most frequent words in carroll-alice.txt:

Iterate through all of the files in the /data/cs91r-s25/gutenberg/trimmed/ directory and print out the top 5 words for each. To get a list of all the files in a directory, use the os.listdirfunction:

This example also uses the pathlib library that you might want to read about.

The goal of this part of the lab is to summarize those compressed files into the output below, where we have only saved the word and the total number of times the word occurred across all 24 files. (You do not need to keep track of how many volumes each word occurred in, only the total counts.)

Write your solution to this part of the lab in ngrams.py. Hints and requirements are below.

Your goal is to read these file in python without decompressing the files first. To do that, see the python gzip library documentation. Try to print out the first 3 lines of the file 1-00023-of-00024.gz in python using the gzip library to open the file. In the documentation, note that one of the options when you open the file is the mode. Be sure to set the mode to rt (read, text); otherwise python will interpret the data incorrectly.

Note that these compressed files are very large and they take a while to read over the file system and decompress. When your program is complete, it may take up to 30 minutes to process all 24 files. Therefore, you should definitely be testing your code step-by-step to be sure things are working!

Your program should have the following interface:

That means that when you run the program, you will specify at least one parameter (that will be stored in sys.argv[1]) but you can specify as many files as you’d like on the command-line. You can use the length of sys.argv to figure out how many parameters you’ve passed to your program.

Try out your interface using just one file and output the first 3 lines of the file as you did before. Next, provide multiple files as parameters and output the first 3 lines of each file you provide.

You will notice that there is a lot of junk in these files. By junk, we mean things that aren’t words. You can view some of "words" Google has counted by looking at the first column of one of the files:

Many of these are not "words" that we would care about (e.g. strings with characters that aren’t English letters or strings that have numbers and punctuation in them). You only want to count words that meet the following criteria:

For every word that meets these criteria, you will remove all characters after (and including) the first period or underscore. So, from the examples above, third-order_NOUN will become third-order, unresolved.64 will become unresolved, and tracts.78_NUM will become tracts; the word yellowness remains unchanged.

Run your program using the 1-00023-of-00024.gz file as the only parameter and print out all of the words in the order you find them. Do not print anything except for the word (the first column of text in the file). Only include the words that meet the criteria above and remove extraneous characters (e.g. .78_NUM) as described above. You should get the following output for file 00023:

It’s important to notice that before you removed the extraneous characters, each word only appeared once in the file. But now, words like yellowness appear multiple times:

This is because the word yellowness appears in 1-00023-of-00024.gz as yellowness ,yellowness_ADJ, yellowness_X, yellowness_VERB, and yellowness_NOUN. This will be important in the next step.

For each line that meets the criteria, you want to count the total number of times that the word occurred across all years. Do not count the number of volumes that the word appeared in. If you total up each line and print it out, you should get this output:

However, remember that words can appear multiple times as we saw with yellowness above. If we grep for the appearances of yellow after calculating the totals, we’d see:

So, rather than printing the totals out as soon as you calculate them, save the counts in a dictionary (or collections.Counter). Then, after you’ve gone through all of the files on the command-line, print out the results. Here’s a sample of my output where you’ll notice we are now summing all the appearances of yellowness:

Note: Most words appear in only one file, but a very small number of words, once you’ve removed the extraneous characters, appear in multiple files. Once such example is Kids that appears in file 00010 and file 00011. You can test to be sure your program handles those cases correctly by verifying that if you run the program only on file 00010 you should count 3643441 appearances of Kids; if you run only on file 00011 you should count 3639068 appearances of Kids; and if you run it on both files you get the 7282509, the sum of both files.

Remember, each of these files has a lot of words, and decompressing them over the file system and then processing them will take a fair amount of time. Do not run on all 24 files until you are fairly certain your code is working. You should expect this step to take about 30 minutes. Of course, your particular implementation may make this slower or faster, but be patient!

Don’t forget to use tqdm!

You can run on all 24 files by using file globbing to grab all gzip files in the directory. Notice below that the output is being directed to a file in the /scratch directory. You can store it anywhere you’d like in /scratch but you’ll need to replace userid1 with your own user id.

You’ll notice that the output file is quite large (nearly 300MB). You should compress that file, too, using gzip:

Or you could gzip it directly as you run your program (shown without the /scratch directory to prevent word-wrapping):