Welcome!

Agenda for today

• Course mechanics
  • All of the details are in the syllabus
• General orientation to the course

What is data mining?

• Finding useful patterns in large collections of data
  • a.k.a. “knowledge discovery in data bases”
• Two sides:
  • Methods for coming up with candidate patterns
  • Methods for checking candidate patterns
• The most useful and most checkable patterns are ones which let us predict

What are we going to learn about

So many things!

Nearest neighbors

• “This new case will do what similar cases did” is surprisingly powerful
• Need good ways to define “similar”
• Need good ways to find similar cases in big data sets

Prediction and decision trees

• Simple, binary-choice models for prediction
• Plus ways of combining many trees to get “forests”

Nonlinear features and kernels

• Recycle everything we know about linear models by using new, nonlinear features of the data
• Avoid having to actually calculate the transformations by using tailored similarity measures (“kernels”)

Information measures

• How much information does one feature give us about another?
• How much information does the model give us about predictions?
• How much information does the data give us about the models?

Dimension reduction

• More than 3D is very hard for us to grasp
  • Maybe 5 if you use color and animation well
• Somehow reduce the huge number of features to something more manageable but still intelligible

Clustering

• Divide the data into groups, without knowing what the right groups are
• Using probability models so this isn’t totally arbitrary

Checking our guesses

• Cross-validation for seeing how well our models actually predict
  • Divide the data, fit the model to one part, evaluate predictions on the other
• Bootstrap for seeing how much uncertain we should have about the model
  • Sub-divide the data, fit the model to only part, see how much things change over many random sub-samples
• How do we account for the fact that we’ve chosen our model to fit the data?

Applications

• Could have looked at astronomy, biology, marketing, medicine, supply-chain management, social welfare services…
• Will instead focus on three:
  • Information retrieval
  • Recommendation engines
  • Criminal risk/recidivism prediction
  • You live with the first two and hopefully will never be on the receiving end of the third

Information retrieval

• Everything important about data mining in a small package
  • Finding similar items by content
  • Using features to define “similar content”
  • Making predictions about patterns
  • Evaluating predictions about patterns

Recommendation engines

• “You may also like”
• Uses: nearest neighbors, clustering, dimension reduction, classification, …
• Abuses/unintended consequences…

Fairness in prediction

• How do we keep our models from just reproducing the injustices of the society around us?
  • Can we keep our models from doing this?
  • What counts as an injustice?
• Case study: how risky is it to release this just-arrested person, before their trial?

Waste, fraud and abuse

• Sometimes data mining just won’t work
  • Bad data
  • Overwhelming data and the curse of dimensionality

Waste, fraud and abuse

• Sometimes data mining is just wrong

Where did this come from?

• Statistics
  • Exploratory data analysis
  • Regression, especially regression by matching
  • Principal components and factor analysis
  • Classifiers
  • Clustering
• Computer science
  • Databases
  • Pattern recognition (= classsifiers)
  • Dimension reduction (= PCA and factor analysis)
  • Clustering (= clustering)
  • Nearest neighbors (= matching)
  • Machine learning (= predictive statistical modeling)

Where did this really come from?

• “The state, the coldest of all cold monsters”

Where did this really come from?

• We invented data-processing machines, before computers, because we were already keep tabs on so much about so many people

Where did this really come from?

• Computers made it really easy to create, store and analyze data
  • Create: sensors (phones, cameras, cash registers…), digitization (doing everything via computer)
  • Store: hard disks, disk arrays, data-bases…
  • Analyze: that’s where we come in
  • Organizations create and store the data even when they don’t have particular analyses in mind
• Tension: flexibility to find many different patterns vs. vulnerability to noise and coincidence
  • All the data is great: brilliant ideas from 50–100 years ago become practical!
  • Why we look a lot of different ways of finding patterns
  • All the data is a problem: If you run a gazillion analyses, (gazillion/20) will be significant at the 5% level
  • Why we look at how to avoid fooling ourselves

What will you need to know?

• 36-401, modern regression
• \(=\) Linear statistical models in R
• \(=\) Actual experience with predictive modeling of data
  • \(+\) Mathematical statistics (for notions of inference and error)
  • \(+\) Probability (for notions of distributions and risk)
  • \(+\) Linear algebra through eigenvalues and eigenvectors (essential for multivariate data)
  • \(+\) Calculus (essential for optimization)

Course mechanics

• Class meetings
• Readings
• Homework
• Class homepage: [http://www.stat.cmu.edu/~cshalizi/dm/20]
  • Full syllabus with all the details
  • Links to course assignments, due dates, etc.
  • What to read
• Canvas: submitting work, gradebook, some readings
• Piazza: question-answering

Class meetings

• Lecture: me explaining and demonstrating stuff, you asking questions
• In-class exercises: you checking your understanding
• No electronics

In-class exercises

• Short (\(< 20\)) minute problem-solving exercises
• Pencil-and-paper, not electronics
• Groups of up to 4
• Most if not all class meetings

Reading

• Most class meetings will have required reading: do it!
• Many will have recommended reading: try to do it
• Most will have background reading: if you get interested

Reading: Textbook

Principles of Data Mining

Reading: Textbook

Reading: Textbook

The Ethical Algorithm: The Science of Socially Aware Algorithm Design

Reading: Recommended

Statistical Learning from a Regression Perspective

Reading: Recommended

Elements of Statistical Learning

Homework

• Implementing methods on actual data
• Working out some of the mathematical details
• Practicing interpreting and communicating the results
• One assignment per week, 14 in all
  • Released by Friday each week (sometimes earlier)
  • Usually due Thursdays at 10 pm via Canvas

Homework

• 10% of each homework will be graded on the quality & clarity of your communication
  • There will be a rubric for this on each assignment
• Most (if not all) homeworks will also have an online assignment
  • Easy if you’ve done the reading
  • Usually due Sundays at 10 pm via Canvas
  • Online assignments will be about 10% of your grade

Grading

• Homework: 90% of your grade
  • Lowest 2 homework+online grades dropped automatically
  • No late homework
  • If you do all 14 homeworks with a minimum grade of \(\geq 60\)%, lowest 3 grades get dropped
• In-class exercises: 10% of your grade
  • Lowest 3 dropped automatically
  • If you do all exercises with minimum of \(\geq 60\)%, lowest 5 dropped
• No exams
• Grade boundaries: 90 for an A, 80 for a B, etc.
  • 662: 97 for an A+, etc.

Time expectations

• This is a 9 credit-hour class
• You spend 3 hours in lecture each week
• \(\Rightarrow\) 6 hours working on the class outside of lecture each week
  • averaged over the semester
  • Talk to me if it’s taking much longer than that

Cheating, collaboration & plagiarism

• Don’t
• You can talk to each other, you can read whatever you like, but everything you turn in must be your own work
  • Exception to “read whatever you like”: Don’t read old solutions, or share this year’s
  • Exception to “all work must be your own: Working together is OK for in-class exercises
• Full policy in the syllabus
• You will need to do a HW 0 about the class cheating policy before getting graded

Homework format

• We will use R Markdown to integrate your code directly in to your writing
  • Write a source file that’s mostly ordinary text, plus the R code you want
  • “Knit” to an HTML or PDF with the text plus the output of the code (figures, tables, numbers)
• Ensures computational reproducibility: your results really came from the code that you say/think they came from
• You will turn in both your raw R Markdown file and the knitted PDF or HTML

Switch to R Studio

Specifically welcome.Rmd

Some lessons from the demo

• Using R Markdown is almost no extra work over just using R and copying-and-pasting your results in
  • Eventually it’s less work than copying-and-pasting and it’s more reliable
• Trees are a characteristic data-mining model
  • Very flexible (can fit anything, with a big enough tree…)
  • Good at things linear models are bad at (thresholds, interactions)
  • Almost never realistic (… but when is the linear model right?)

Next time: The truth about linear regression

• A review, without the mythology, to set us up for more powerful prediction methods
• Do the reading!