In the first part of the lab, you will write a simple sentence segmenter.

The /data/cs91r-s25/brown/ directory includes three text files taken from the Brown Corpus:

The files do not indicate where one sentence ends and the next begins. In the data set you are working with, sentences can only end with one of 5 characters: period, colon, semi-colon, exclamation point and question mark.

However, there is a catch: not every period represents the end of a sentence since there are many abbreviations (U.S.A., Dr., Mon., etc., etc.) that can appear in the middle of a sentence where the periods do not indicate the end of a sentence. The text also has many examples where colon is not the end of the sentence. The other three punctuation marks are all nearly unambiguously the ends of a sentence. Yes, even semi-colons.

For each of the above files, we have also provided a file containing the line number (counting from 0) containing the actual locations of the ends of sentences:

Your job is to write a sentence segmenter. We will add that segmenter to spacy’s processing pipeline.

We have provided you with some starter code in segmenter.py. Don’t worry about what the existing code does just yet.

Eventually, we’re going to want to run this program, which means we’re going to need main to do something other than pass. (Note: pass is a python statement that does nothing, but can sometimes be useful when python expects you to have an indented block of code and you don’t want to put anything there. In that case, using pass as a placeholder is useful.) You’ll want to remove pass once you have something better to put there.

We’re going to want to be able to provide the segmenter.py program with command-line arguments, taking one required argument (a textfile) and one optional argument (the OUTPUT_FILE specified using ‑o or ‑‑outfile). If the user does not specify an OUTPUT_FILE, the program should output to sys.stdout.

Write the command-line interface for segmenter.py using the argparse module in python for processing command-line arguments. In addition to the module documentation, you may also find the argparse tutorial useful. We’ve already included import argparse at the top of the file to get you started.

Both arguments to segmenter.py will be files. The first file will be one that you read and the second file will be one that you write. The type of both arguments should be FileType arguments.

You can print out the arguments you get to verify that things are working, but you don’t need to do anything with the arguments just yet.

As in Lab 2, all print statements should be in your main() function, which should only be called if segmenter.py is run from the command line.

Now that your segmenter.py function takes command-line arguments, you can use them to create a spacy Doc from the textfile. Pass the textfile argument into the create_doc function we’ve provided. The create_doc function returns a spacy Doc.

The tokenizer.py file contains some code that you do not need to modify in any way. However, it would be helpful if you understood what it does at a high-level, even if the specifics aren’t clear.

The tokenizer.py file defines two things:

The MyTokenizer class allows you to define a piece of spacy’s processing pipeline. In spacy, a pipeline is defined as the steps that spacy takes to take raw text and convert it into a format that is ready for various text processing tasks.

In this step, we are replacing spacy’s default English tokenizer — a component of spacy’s processing pipeline — with our tokenizer. We are writing our own tokenizer because otherwise spacy would do much of the sentence segmentation for us. Since the goal of this lab is for you to write the segmenter, we are replacing the default tokenizer with a tokenizer that simply separates the non-alphabetic text (punctuation, dollar signs, etc.) from the letters and numbers in the text.

In the segmenter.py file, write a function called baseline_segmenter that takes a spacy Doc as its only argument. This function will use a naive algorithm for determining when we’ve reached the end of each sentence.

This is called a baseline segmenter, because we are expecting that this very simple segmenter can get pretty good results without much work on our part. By setting the bar pretty high, we’re expecting our improvements to be even better than this baseline. We will be able to compare our improved algorithm against this baseline to see how much better it performs.

The baseline segmenter will iterate through all of the tokens in the Doc (using for token in doc:) and predict which ones are the ends of sentences. Confusingly, instead of keeping track of the ends of sentences, spacy keeps track of the beginning of sentences. In particular, the first token in each sentence in a spacy Doc has its is_sent_start attribute set to True. This means that for every token that you predict corresponds to the end of a sentence, you should set the is_sent_start attribute to True for the next token.

Recall that every sentence in our data set ends with one of the five tokens ['.', ':', ';', '!', '?']. Our baseline_segmenter will predict that every instance of one of these characters is the end of a sentence.

As shown below, you can access the text content of a Token in spacy through its .text attribute to determine if it matches one of these five tokens:

Next, add your baseline_segmenter function to the pipeline of tools that will be called on every Doc that is created with your create_doc function. To do that, add the following line to the create_doc function just below where you set up your tokenizer:

Finally, update your main() function to write out the token numbers corresponding to the predicted line breaks to outfile, if it was specified as a command-line option, or print the token numbers to the screen if the outfile argument was omitted. You should write the token numbers one per line. You can view /data/cs91r-s25/brown/editorial-eos.txt as an example.

Be sure to write out the last token number as a sentence boundary. Since spacy is keeping track of the starts of new sentences, the final sentence is never explicitly marked. You can access a list of the sentences in a spacy Doc with its sents attribute:

Confirm that when you run your baseline_segmenter on the file editorial.txt, it predicts 3278 sentence boundaries.

To evaluate your system, we are providing you a program called evaluate.py that compares your hypothesized sentence boundaries with the ground truth boundaries. This program will report to you the true positives, true negatives, false positives and false negatives (as well as precision, recall and F-measure, which we haven’t talked about in class just yet). You can run evaluate.py with the -h option to see all of the command-line options that it supports.

A sample run with the output of your baseline segmenter from above stored in bl_editorial-eos.txt would be:

Run the evaluation on your baseline system’s output for the editorial category, and confirm that you get the following before moving on:

Now it’s time to improve the baseline sentence segmenter. We don’t have any false negatives (since we’re predicting that every instance of the possibly-end-of-sentence punctuation marks is, in fact, the end of a sentence), but we have quite a few false positives.

Make a copy of your baseline_segmenter function called my_best_segmenter. Be sure to copy the line that says @Language.component("baseline_segmenter") and change it to say @Language.component("my_best_segmenter"). In the create_doc function, change the line that says add_pipe to use my_best_segmenter instead of the baseline_segmenter to use this new segmenter instead of the baseline one.

You can see the tokens that your system is mis-characterizing by setting the verbosity of evaluate.py to something greater than 0. Setting it to 1 will print out all of the false positives and false negatives so you can work to improve your my_best_segmenter function.

To test your segmenter, we will run it on a hidden text you haven’t seen yet. It may not make sense for you to spend a lot of time trying to fix obscure cases that show up in the three texts we are providing you because these cases may never show up in the hidden text that you haven’t seen. But it is important to handle cases that occur multiple times and even some cases that appear only once if you suspect they could appear in the hidden text. You want to write your code to be as general as possible so that it works well on the hidden text without leading to too many false positives.

The dataset we will be using is in /data/cs91r-s25/semeval-19-04/. The file articles.xml file contains all of the articles we’ll look at this week. The file labels.xml contains the labeling of each article (is it hyperpartisan or not, and what is its bias). These files have 600,000 articles and 600,000 labels, respectively, so they are pretty large and hard to work with while you’re developing. Therefore, we’ve provided small_articles.xml and small_labels.xml which only contains 12 articles. You can work with them while you work out the bugs.

Once you switch to the full data set, you’ll have to deal with the fact that the data file is big (around 2.6G) so we won’t store the whole thing in memory at once if we can help it. Fortunately, the lxml library gives us a way to iteratively parse through an xml file, dealing with one node (in our case, article) at a time. Here’s sample code that opens a file called myfile.xml and calls a function called my_func on every article node:

As needed, refer back to the lxml tutorial we saw in clab. You won’t need to use the event variable, but the element variable will be a node in the XML tree. You will want to get the .items() of the element (which contains the attributes of the node such as id and title). And you will need to use the .itertext() method to get the text of the node.

Do not use findall or find. There are two reasons: (1) the dataset is very large and findall will be slow, and (2) this data has a lot of embedded HTML tags, so finding all of the text that is nested instead of <p> and <a> tags is doable but non-trivial.

Before we continue, take a look at the contents of the articles.xml file and the labels.xml file. Look at the structure of the documents, especially at the start of each <article>.

Write two functions, one called get_labels() and one called get_articles(). The get_labels function should take a filename (e.g. /data/cs91r-s25/semeval-19-04/labels.xml) and it should return a dictionary mapping article IDs to their bias ("left", "right", etc). The get_articles function should take a filename (e.g. /data/cs91r-s25/semeval-19-04/articles.xml) and should return a dictionary that maps article IDs to a spacy Doc containing the news article.

When creating spacy documents, don’t use your MyTokenizer from Part 1. We’ll be ignoring sentence boundaries for this part, so you don’t need to add your my_best_segmenter either. To initialize the spacy pipeline, you can just use this line:

One problem we will need to worry about is that xml files contain HTML entities. These files in particular include a lot of the entity  , a non-breaking space. In order to clean these up, we can use a function in the html library before creating our spacy document:

Once you have both the dictionary of labels and the dictionary articles, create another new dictionary whose keys are labels (e.g. "left", "right", etc). Each key should map to a list of all the articles (as spacy documents) labeled with that key.

A bigram is a pair of words that occur together. For example, in the sentence "The cat sat on the mat", the bigrams are: "The cat", "cat sat", "sat on", "on the", "the mat". A trigram is a triplet of words that occur together. The trigrams are: "The cat sat", "cat sat on", "sat on the", "on the mat".

The table shown below is included in your README.md file. When a question asks you to fill in the table, fill in the matching table in the README.md file. Be sure to put your answers to Questions 8, 9 and 10 in the table.

Instead of collecting statistics on one set of articles ("training") and seeing how closely these statistics match another set of articles ("testing"), what if you trained your tool on two of the sets (e.g. 'left' and 'least') and tested it on the third category (e.g. 'right')?