Name:
Andrew ID:
Collaborated with:
On this homework, you can collaborate with your classmates, but you must identify their names above, and you must submit your own homework as an knitted HTML file on Canvas, by Sunday 10pm, this week.
1a. A data table of dimension 3000 x 11, containing demographic and economic variables measured on individuals living in the mid-Atlantic region, is up at http://www.stat.cmu.edu/~ryantibs/statcomp-S18/data/wage.csv. (This has been adapted from the book An Introduction to Statistical Learning.) Load this data table into your R session with read.csv() and save the resulting data frame as wage.df. Check that wage.df has the right dimensions, and display its first 3 rows. Hint: the first several lines of the linked file just explain the nature of the data; open up the file (either directly in your web browser or after you download it to your computer), and count how many lines must be skipped before getting to the data; then use an appropriate setting for the skip argument to read.csv().
1b. Identify all of the factor variables in wage.df, set up a plotting grid of appropriate dimensions, and then plot each of these factor variables, with appropriate titles. What do you notice about the distributions?
1c. Identify all of the numeric variables in wage.df, set up a plotting grid of appropriate dimensions, and then plot histograms of each these numeric variables, with appropriate titles and x-axis labels. What do you notice about the distributions? In particular, what do you notice about the distribution of the wage column? Does it appear to be unimodal (having a single mode)? Does what you see make sense?
2a. Fit a linear regression model, using lm(), with response variable wage and predictor variables year and age, using the wage.df data frame. Call the result wage.lm. Display the coefficient estimates, using coef(), for year and age. Do they have the signs you would expect, i.e., can you explain their signs? Display a summary, using summary(), of this linear model. Report the standard errors and p-values associated with the coefficient estimates for year and age. Do both of these predictors appear to be significant, based on their p-values?
2b. Save the standard errors of year and age into a vector called wage.se, and print it out to the console. Don’t just type the values in you see from summary(); you need to determine these values programmatically. Hint: define wage.sum to be the result of calling summary() on wage.lm; then figure out what kind of R object wage.sum is, and how you can extract the standard errors.
2c. Plot diagnostics of the linear model fit in the previous question, using plot() on wage.lm. Look at the “Residuals vs Fitted”, “Scale-Location”, and “Residuals vs Leverage” plots—are there any groups of points away from the main bulk of points along the x-axis? Look at the “Normal Q-Q” plot—do the standardized residuals lie along the line \(y=x\)? Note: don’t worry too if you’re generally unsure how to interpret these diagnostic plots; you’ll learn a lot more in your Modern Regression 36-401 course; for now, you can just answer the questions we asked. Challenge: what is causing the discrepancies you are (should be) seeing in these plots? Hint: look back at the histogram of the wage column you plotted above.
2d. Refit a linear regression model with response variable wage and predictor variables year and age, but this time only using observations in the wage.df data frame for which the wage variable is less than or equal to 250 (note, this is measured in thousands of dollars!). Call the result wage.lm.lt250. Display a summary, reporting the coefficient estimates of year and age, their standard errors, and associated p-values. Are these coefficients different than before? Are the predictors year and age still significant? Finally, plot diagnostics. Do the “Residuals vs Fitted”, “Normal Q-Q”, “Scale-location”, and “Residuals vs Leverage” plots still have the same problems as before?
2e. Use your fitted linear model wage.lm.lt250 to predict: (a) what a 30 year old person should be making this year; (b) what President Trump should be making this year; (c) what you should be making 5 years from now. Comment on the results—which do you think is the most accurate prediction?
3a. Fit a logistic regression model, using glm() with family="binomial", with the response variable being the indicator that wage is larger than 250, and the predictor variables being year and age. Call the result wage.glm. Note: you can set this up in two different ways: (i) you can manually define a new column (say) wage.high in the wage.df data frame to be the indicator that the wage column is larger than 250; or (ii) you can define an indicator variable “on-the-fly” in the call to glm() with an appropriate usage of I(). Display a summary, reporting the coefficient estimates for year and age, their standard errors, and associated p-values. Are the predictors year and age both significant?
3b. Refit a logistic regression model with the same response variable as in the last question, but now with predictors year, age, and education. Note that the third predictor is stored as a factor variable, which we call a categorical variable (rather than a continuous variable, like the first two predictors) in the context of regression modeling. Display a summary. What do you notice about the predictor education: how many coefficients are associated with it in the end? Challenge: can you explain why the number of coefficients associated with education makes sense?
3c. In general, one must be careful fitting a logistic regression model on categorial predictors. In order for logistic regression to make sense, for each level of the categorical predictor, we should have observations at this level for which the response is 0, and observations at this level for which the response is 1. In the context of our problem, this means that for each level of the education variable, we should have people at this education level that have a wage less than or equal to 250, and also people at this education level that have a wage above 250. Which levels of education fail to meet this criterion? Let’s call these levels “incomplete”, and the other levels “complete”.
3d. Refit the logistic regression model as in Q3b, with the same response and predictors, but now throwing out all data in wage.df that corresponds to the incomplete education levels (equivalently, using only the data from the complete education levels). Display a summary, and comment on the differences seen to the summary for the logistic regression model fitted in Q3b. Did any predictors become more significant, according to their p-values?
4a. Install the gam package, if you haven’t already, and load it into your R session with library(gam). Fit a generalized additive model, using gam() with family="binomial", with the response variable being the indicator that wage is larger than 250, and the predictor variables being year, age, and education; as in the last question, only use observations in wage.df corresponding to the complete education levels. Also, in the call to gam(), allow for age to have a nonlinear effect by using s() (leave year and education alone, and they will have the default—linear effects). Call the result wage.gam. Display a summary with summary(). Is the age variable more or less significant, in terms of its p-value, to what you saw in the logistic regression model fitted in the last question? Also, plot the fitted effect for each predictor, using plot(). Comment on the plots—does the fitted effect make sense to you? In particular, is there a strong nonlinearity associated with the effect of age, and does this make sense?
4b. Using wage.gam, predict the probability that a 30 year old person, who earned a Ph.D., will make over $250,000 in 2018.
Challenge. For a 32 year old person who earned a Ph.D., how long does he/she have to wait until there is a predicted probability of at least 20% that he/she makes over $250,000 in that year? Plot his/her probability of earning at least $250,000 over the future years—is this strictly increasing?
for() loop or one of the functions from apply family, implement the jackknife procedure to estimate the standard errors for the coefficients attributed to year and age in your linear regression model from (say) Q2d. Display the results. How do these jackknife standard error estimates compare to the ones given by the call to summary()? Again implement the jackknife to estimate the standard errors attributed to each coefficient in your logistic regression model from (say) Q3d, display the results, and compare to those from summary().