1 Good approaches to information retrieval we will neglect

1.1 Metadata

The oldest approach is to have people create data about the data, metadata, to make it easier to find relevant items. Library catalogues are like this (Figure 1): people devise detailed category schemes for books, magazines, etc. For instance, a book has a title, one or more authors (possibly “anonymous” or pseudonyms), a publisher, a place of publication, a date of publication, an ISBN (if it’s recent enough), and its contents are noted as belonging to one or more subject topics, with a lot of work going in to designing the set of subject-matter topics. A magazine doesn’t have an author, it has multiple volumes with multiple dates, and it has an ISSN instead of an ISBN, if it has a number like that at all. (Nowadays some people call this sort of scheme an ontology.) Having fixed on a scheme like this, people would actually examine the objects, decide which categories they belong to, and write down that information along with their location on the shelves, and then copy this information so that it appeared in multiple places — by title, by author, by subject, etc. This worked OK for a few thousand years (better for the last few hundred, once people got really serious about it), but it needs people, who are slow and expensive and don’t scale. It’s not feasible for searching census records, or purchase histories at an online store, or the Web.

Figure 1: Example of metadata (from [search.library.cmu.edu]). This is information about the book, which is not part of its actual contents. Notice that this is structured: every book in the catalogue will have a title, author(s), subject(s), a location, etc. All of this information is humanly-generated.

2 Similarity search

This is where searching by similarity comes in. Suppose you can find one Web page which is about problems in 2009-model Mustangs. We’re going to see, first, how you can tell your computer “find me more pages like this”. We will see later how you can avoid the step of initially lucking into the first page, and how you can use the same sort of trick to search other kinds of data — say images on the Web, or hospital patient records, or retail transactions, or telephone-call records.

To illustrate these ideas concretely, we’re going to use a part of the New York Times Annotated Corpus (Sandhaus 2008). This consists of $1.8 \times {10}^6$ stories from the Times, from 1987 to 2007, which have been hand-annotated with metadata about their contents. (See Figure 2.) We are going to look at methods for information retrieval which do not use the metadata, but having it there will help us determine how well those methods are working. You’ve gotten to use a small part of this data set in the homework¹.

Figure 2: Example of the structured meta-data provided with the New York Times Annotated Corpus

3 Representation

The crucial first step is to chose a representation of the data. There are multiple considerations here.

• The representation ought to be something our methods can work with easily.
• The representation ought to be something we can easily generate from the raw data.
• The representation ought to highlight the important, helpful aspects of the data, and suppress others. (If the representation didn’t ignore some aspects of the data, it would be the data again.)

One of the more important bits of craft lore in data mining is that getting a good representation — trading these concerns off against each other — is at least as important as picking the right algorithms.

For today, we are searching for documents by similarity, so our data are natural-language documents.

We could try to represent the meaning of the documents. Look at Figure 2 again. The abstract reads “Chairs of the 1920s and 30s are featured at Johnson & Hicks, new home furnishing store in TriBeCa”. We could try to represent the meaning here in something like mathematical-logical² notation:

exhibit.of(chairs(age-1920--1930),Johnson&Hicks,now)
is.a(Johnson&Hicks,store,type="home furnishing")
location.of(Johnson&Hicks,TriBeCa)
begin.date(Johnson&Hicks,now)

and then we’d probably want to go on to tell the system that chairs are a kind of furniture, where TriBeCa is, that chairs that old are a kind of vintage product, etc. If we could do this, there are tons of methods for automated deduction which we could use to find stories about displays of contemporary chairs, or vintage lamps, etc. The snag is that extracting this sort of meaning from the text alone, without using human readers, turns out to be insanely hard. People in artificial intelligence have been saying it’ll happen within the next twenty years since the 1950s; so we’ll wish them good luck and leave them to their work.

A less ambitious sort of representation which still tries to get more or less explicitly at meanings would be to draw up a big list of categories, and then go over the documents and check of which categories they fall in to. This is basically what the annotating meta-data does for the Times corpus. Social scientists sometimes do this and call it coding the documents (Franzosi 2004); they’ll often code for the emotional tone or valence as well. (This tends to be done with smaller, more focused sets of documents, so they can get away with smaller sets of categories.) Once we have this, our representation is a bunch of discrete variables, and there are lots of methods for analyzing discrete variables. But, again, extracting this sort of information automatically from the text is too hard for this to be useful to us³. In fact, the best automatic methods for this sort of thing use the bag-of-words representation which is coming up next.

           #        1920s        1930s            a        above 
           9            1            3            3            1 
         and          are           at        backs     bergdorf 
           4            3            3            1            1 
       buyer       chairs checkerboard        clean   discovered 
           1            4            1            1            1 
  especially        ferry      folding          for     franklin 
           1            1            1            2            1 
      french         from  furnishings      goodman          her 
           2            2            2            1            2 
       hicks         home       hudson            i           in 
           2            2            1            1            2 
 information           is      johnson        lines         lisa 
           1            1            2            1            1 
        love           ms           of           on         once 
           1            1            2            1            1 
      opened   originally         pair      passion     richness 
           1            1            1            1            1 
       right         said    september       simple        steel 
           2            1            1            1            1 
       store       street       tables          the        there 
           1            2            1            3            1 
       those      tribeca         used       walnut          was 
           1            1            1            1            1 
      weimer          who         with        woods 
           2            1            1            1 

# Load functions for turning NYT Annotated Corpus documents into bag-of-word data
# frames Having a separate source file does go against the spirit of complete
# reproducibility somewhat, but this makes it easier for you to re-use the code,
# and I don't feel like writing a full-blown R package
source("http://www.stat.cmu.edu/~cshalizi/dm/19/hw/01/nytac-and-bow.R")

# read.doc() reads a document in and extracts the body of the story as a vector
# of words in order strip.text() does some tidying up (e.g. removes punctuation
# marks) table() turns the vector of strings into a table of _counts_ of words
# table() is built-in to R, the others are from the provided code
music.1 <- table(strip.text(read.doc("nyt_corpus_small/music-0416371.xml")))
music.2 <- table(strip.text(read.doc("nyt_corpus_small/music-0430743.xml")))
music.3 <- table(strip.text(read.doc("nyt_corpus_small/music-0702928.xml")))
music.4 <- table(strip.text(read.doc("nyt_corpus_small/music-1343561.xml")))
music.5 <- table(strip.text(read.doc("nyt_corpus_small/music-1439011.xml")))
art.1 <- table(strip.text(read.doc("nyt_corpus_small/art-0437155.xml")))
art.2 <- table(strip.text(read.doc("nyt_corpus_small/art-1573733.xml")))
art.3 <- table(strip.text(read.doc("nyt_corpus_small/art-1590454.xml")))
art.4 <- table(strip.text(read.doc("nyt_corpus_small/art-1657584.xml")))
art.5 <- table(strip.text(read.doc("nyt_corpus_small/art-1812955.xml")))
small.bow <- make.BoW.frame(list(music.1, music.2, music.3, music.4, music.5, art.1, 
    art.2, art.3, art.4, art.5), row.names = c(paste("music", 1:5, sep = "."), paste("art", 
    1:5, sep = ".")))

kable(small.bow[, c("a", "film", "members", "new", "old", "program", "programs", 
    "sale", "series", "said", "space", "time", "tuesday", "wife")])

4 Measuring Similarity

Right now, we are interested in saying which documents are similar to each other because we want to do search by content. But measuring similarity — or equivalently measuring dissimilarity or distance — is fundamental to data mining. Most of what we will do will rely on having a sensible way of saying how similar to each other different objects are, or how close they are in some geometric setting. Getting the right measure of closeness will have a huge impact on our results.

This is where representing the data as vectors comes in so handy. We already know a nice way of saying how far apart two vectors are, the ordinary or Euclidean distance, which we can calculate with the Pythagorean formula: $$\| \vec{x} - \vec{y} \| \equiv \sqrt{\sum_{i=1}^{p}{{\left(x_i - y_i\right)}^2}}$$ where $x_i$, $y_i$ are the $i^{\mathrm{th}}$ components of $\vec{x}$ and $\vec{y}$. Remember that for bag-of-words vectors, each distinct word — each entry in the lexicon — is a component or feature.

(The Euclidean length or Euclidean norm of any vector is $$\| \vec{x} \| \equiv \sqrt{\sum_{i=1}^{p}{{x_i^2}}}$$ so the distance between two vectors is the norm of their difference $\vec{x} - \vec{y}$. Equivalently, the norm of a vector is the distance from it to the origin, $\vec{0}$.)

Now, there are other ways of measuring distance between vectors. Another possibility is the taxicab or Manhattan distance $$\sum_{i=1}^{p}{|x_i - y_i|}$$ It’s a perfectly good distance metric; it just doesn’t happen to work so well for our applications.

rownames(small.bow)[nearest.points(small.bow)$which]

##  [1] "music.5" "music.5" "music.1" "music.5" "music.1" "art.5"   "art.5"  
##  [8] "music.2" "music.4" "art.2"

5 Abstraction

Let’s step back and summarize how we’ve set ourselves up to search documents by content.

1. We gather a large collection of potentially-relevant documents.
2. We calculate a bag-of-words vector from each document.
   1. At this step, we may do some cleaning or pre-processing, like removing punctuation marks or capitalization.
   2. We gather the vectors into an $n\times p$ matrix.
3. We normalize the vectors and/or weight their components, if we want.
4. We calculate the $n\times n$ matrix of distances between documents.
5. We find the closest matching document(s) to any document of interest.

It’s important to note here that only steps (1) and (2) actually use the documents. Everything else after that uses only the feature vectors. In programming terms, we’d say that the feature vectors are an abstraction of the documents, and thereafter we’re working at the abstract level.

There are two reasons programmers use abstraction: convenience, and generalization. “Convenience” here means that we don’t have to keep dealing with the actual documents, just their representations, and we’re mathematically trained enough to find vectors convenient to work with. But “generalization” means that we can use the same tools for different objects, so long as they have the same kind of abstraction.

What that in turn means is that we haven’t just seen how to do similarity search for text documents. We’ve seen how to similarity search for images, music, social-network nodes, credit card transactions, phone calls, etc., etc. — for anything where we can represent in terms of feature vectors. This is the great power and attraction of abstraction. This is in fact why we can have a course on data mining in general, rather than many courses on data mining for text (what kind of text?), images (what kind of images?), music, credit-card transactions, etc., etc.

There are two big drawbacks to abstraction: the representation you’re using may not capture much (or any) useful information about the underlying items, and you can’t tell whether it is working well from the abstraction alone. Intuitively, the bag-of-words representation of documents feels like it should work better than the bag-of-letters representation, though the later would have only 26 dimensions and would in many ways be much more convenient. But they’re equally good vector-space representations, and nothing you do to the vectors will be able to tell you that words are a useful level of abstracting text for search, but letters are not. Nor is anything we do to the vectors alone going to tell us that we want to use, say, normalization by word-count and inverse document frequency weighting. To decide on the right representation, we need to be able to go back and connect the abstract feature vectors, or things we’ve calculated from them, to either the items themselves, or to other information about those items. Looking at whether the closest match to a story about art is another story about art is a first step towards that sort of checking.

6 Text search: queries are documents

Let’s summarize where we left similarity searching for documents. We represent each document as a bag of words, i.e., a vector giving the number of times each word occurred in the document. This abstracts away all the grammatical structure, context, etc., leaving us with a matrix whose rows are feature vectors, a data frame. To find documents which are similar to a given document $Q$, we calculate the distance between $Q$ and all the other documents, i.e., the distance between their feature vectors.

7 Evaluating Search

When someone uses a search engine, they have some idea of which of the results are what they were looking for. In the jargon, we say that the good results were relevant to the query. There are actually two aspects to finding relevant documents, both of which are important:

• Most of the results should be relevant; that is, the precision of the search should be high.
• Most of the relevant items should be returned as results; that is, the recall should be high, too.

In symbols, if the search returns $k$ items, $r$ of which are relevant, and there are $R$ relevant items in the whole corpus of $N$ items, the precision is the ratio $r/k$, and the recall is the ratio $r/R$. (This is for one query, but we can average over queries.) Notice that $r \leq k$, so there are limits on how high the recall can be when $k$ is small. As we change $k$ for a given query, we get different values for the precision and the recall. Generally, we expect that increasing $k$ will increase recall (more relevant things can come in) but lower precision (more irrelevant things can come in, too). A good search method is one where the trade-off between precision and recall is not very sharp, where we can gain a lot of recall while losing on a little precision.

A visual way of representing the precision-recall trade-off is to plot precision (on the vertical axis) against recall (on the horizontal axis) for multiple values of $k$. If the method is working well, when $k$ is small the precision should be high, though the recall will be limited by $k$; as $k$ grows, the recall should increase, moving us to the right, but the recall will fall, moving us down. So the precision-recall curve should going from somewhere near $(1,0)$ to somewhere near $(0,1)$. The total area under the curve is often used as a measure of how good the search method is.

Of course, there is nothing magic about changing $k$; if we have a different search technique with a tunable setting, we can make a precision-recall curve for it, too.

7.3 Practice in Evaluating Searches

In practice, the only way to tell whether your search-engine’s results are relevant is to ask actual people (Figure 3). The major services do this with lab experiments, with special browsers given to testers to ask quiz them on whether the results were relevant, and by taking random samples of their query log and having testers repeat the queries to see whether the results were relevant. Naturally, this use of human beings is slow and expensive, especially because the raters have to be trained, so the amount of this data is limited — and they are very reluctant to share it.

Figure 3: Search-engine evaluation in practice. (Source)

Notice, by the way, that when dealing with something like the web, or indeed any large collection where users give arbitrary queries, it is a lot easier to estimate precision than recall (how do you find $R$, the number of genuinely relevant documents in the whole corpus?).

8 Feedback

People are much better at telling whether you’ve found what they’re looking for than they are at explaining what it is that they’re looking for. (They know it when they see it.) Queries are users trying to explain what they’re looking for (to a computer, no less), so they’re often pretty bad. An important idea in data mining is that people should do things at which they are better than computers and vice versa: here they should be deciders, not explainers. In information retrieval, Rocchio’s algorithm takes feedback from the user, about which documents were relevant, and then refines the search, giving more weight to what they like, and less to what they don’t like.

The user gives the system some query, whose bag-of-words vector is $Q_t$. The system responses with various documents, some of which the user marks as relevant ($R$) and others as not-relevant ($NR$). (See Figure 3 again.) The system then modifies the query vector: $$Q_{t+1} = \alpha Q_t + \frac{\beta}{|R|}\sum_{\mathrm{doc} \in R}{\mathrm{doc}} - \frac{\gamma}{|NR|}\sum_{\mathrm{doc}\in NR}{\mathrm{doc}}$$ where $|R|$ and $|NR|$ are the number of relevant and non-relevant documents, and $\alpha$, $\beta$ and $\gamma$ are positive constants. $\alpha$ says how much continuity there is between the old search and the new one; $\beta$ and $\gamma$ gauge our preference for recall (we find more relevant items) versus precision (more of what we find is relevant). The system then runs another search with $Q_{t+1}$, and cycle starts over. As this repeats, $Q_t$ gets closer to the bag-of-words vector which best represents what the user has in mind, assuming they have something definite and consistent in mind.

(A word can’t appear in a document a negative number of times, so ordinarily bag-of-words vectors have non-negative components. $Q_t$, however, can easily come to have negative components, indicating the words whose presence is evidence that the document isn’t relevant. Recalling the example of problems with used 2010 Mustang cars, we probably don’t want anything which contains “baskets” or “herds”, since it’s either about sports or animals, and giving our query negative components for those words suppresses those documents.)

The exact same principle can be used for all sorts of implicit feedback. If the user opens a document, spends time with it, “engages” with it, etc., those are all signals that the document is relevant, and so the query should be refined to give more like that, and less like rejected items, or ones which are left after only a brief engagement. In fact, the same principle can be, and is, used in recommendation engines, where we represent the user as a “query” in the space of items to be recommended, and adjust that vector to be more like the items the user engages with, and less like the items the user rejects. Going still further, if the user is deemed similar to other users, their explicit or implicit feedback can be used to shape the original user’s $Q$ vector.

9 Exercises

Do not turn these in, but do think them through.

1. Why is it called the “Manhattan metric” or “taxicab metric”? (Think before looking this up.)
2. Why is it called the “cosine similarity”? (Again, think before looking this up.)
3. Why do we need to normalize bag-of-word vectors to compare documents with different sizes?
4. Why does normalization by Euclidean length de-emphasize a document’s rare words more strongly than normalization by word count? Hint: think about the relationship between $\sum_{i}{|x_i|}$ and $\| \vec{x}\|$.
5. Why is the cosine similarity “the cosine of the angle between the two vectors”?
6. Explain how the cosine similarity is related to the Euclidean-length normalized distance between two vectors.

10 Further reading

Manning, Raghavan, and Schütze (2008) is still a very good introduction to information retrieval.