========================================================================== Team member names: Bin Yang biny@andrew.cmu.edu William Ouyang wouyang@andrew.cmu.edu Victoria Docherty vdochert@andrew.cmu.edu P. Daphne Tsatsoulis ptsatsous@andrew.cmu.edu ========================================================================== For Team Assignment I.2, please * Turn in a complete, revised draft of your Team Assignment I.1, covering all the points [(a) through (e) for project 1, and (a) through (f) for project 2] below. Please label these points in your revised draft, so they are easy (for me!) to find and read. * The revised draft should include very carefully thought out information about Target Population (item b below), Sampling Frame (item c below) and Method of Data Collection (item d below). * Also make any additional revisions required or suggested in my comments below. Please revise ALL PARTS of the proposal for next Tuesday. Even if I have not commented much on one or more points, all aspects of the proposal can be improved. * Assignment I.2 will be graded using the same scheme below, but with higher standards throughout. ========================================================================== Project Proposal 1: CMU Perceptions of Academic Integrity & Old Course Material Archives OVERALL: * EVERYONE can do a better job of convincing me why each of their projects is interesting enough to actually bother to do. * Very nice topic. Now focus your research questions, and your survey questions, on things that someone could actually do something about if they knew the results of your survey! * Need to think carefully about TARGET POPULATION, SAMPLING FRAME, METHOD OF DATA COLLECTION (and missing data / nonresponse), and VARIBLES OF INTEREST for this proposal, for assignmetn I.2. (a) Interesting research question? Doable? Each team member provides one piece of previous research or results on this question? * great set of previous research citations. if you do this project it will be great to keep these articles in mind as you construct questionnaire, collect data, analyze and interpret results. * seems like a very doable on-campus survey * Make a better argument that this survey would have enough impact to be worth doing: - What could anyone do about the results of this survey? Who would be interested? Deans? Professors? Students on campus? - What kinds of results could someone actually do something about? - if your survey simply finds that students "do care" or "don't care" about academic integrity, there isn't much anyone can do except give speeches. - if your survey finds specific results that impact student learning, fairness, etc., *AND* that someone can do something about, that would be interesting. For example (BUT NOT LIMITED TO): + x% of students who studied old course materials found the exact same questions on exams/hw/etc they took in class + x% of students thought old course materials helped them get a better grade, but only y% thought it made them understand the material better + z% of faculty members change questions regularly (how often? every semester? once a year? each time they teach? etc.) to make old course archives less useful [note that the target population is very difffernt for this sort of result, than for the results above! you'll have to choose a single target population for your work!] + z% of faculty members post all their old exams to negate the effects of old course archives [you might do this with either a student target population or a faculty one!] etc. Develop research questions like this that you can answer with a survey and that someone might do something about, if they knew your results. - this topic is closely related to the topic of "how widespread is cheating...". That has been done a bunch of times, but I think you have a different angle that is interesting and potentially more impactful. Find ways to argue that this is so! (b) What population do you want to make inferences about? * pls answer carefully for I.2: TARGET POPULATION * Since you answered ABC from the shorter outline you haven't answered this yet. * Please think carefuly about this. - as you can see above, different sorts of research questions on the same topic could lead to very different target populations! - please specify the target population very clearly. "CMU students" is not clear enough. "Students at the main CMU campus who are currently freshman, sophomores, juniors, or seniors (4th or 5th year)" probably is clear enough. (c) What population will you sample? Is this different from (b)? * pls answer carefully for I.2: SAMPLING FRAME * Since you answered ABC from the shorter outline you haven't answered this yet. * It is not enough to repeat your answer to part (b). Who can you actualy contact? students with on-campus telephone numbers? those with email addresses? anyone who passes by the fence during the hours of X to Y on daya A, B and C? Etc. (d) How do you plan to carry out the survey & why? * pls answer carefully for I.2 MODE OF DATA COLLECTION * Since you answered ABC from the shorter outline you haven't answered this yet. Here are some important things to think about: - Face to face interviews make for great response rates but would be at least very time consuming to carry out. - if you could get a student phone book from each campus, you could conduct a phone survey. this might be effective at getting better response rates but not use up a ton of your time. - if you could get a list of email addresses for each campus, you could conduct an email or email-and-web survey. - for something like academic integrity / cheating, which mode of data collection (face to face, telephone, email, web, etc.) would get you the most accurate responses from each person in the survey? Do you have to worry more about guaranteeing confidentiality? - for email/web surveys nonresponse will be a bigger issue. - for email and web surveys, get only about 25% or less response rate. - how can you improve this with followup and/or prenotification (e.g. postcards)? how many times, and how, will you followup? - who responds, who doesn't? how would this affect biases in your results? - if you are stuck with low response rate, what would you need to know or be able to do, to argue either 1. the sample you got is still like a representative probability sample; or 2. you know what adjustments to make so that the sample can be treated as a representative probability sample (e) What variables will you measure? * please carefully list these variables, and why you would collect them (what would you do with the information, how might they be related to each other, etc.) for I.2 "As previously stated, our questions revolve around studying student attitudes toward old academic material being used (old tests, notes, etc). As a follow up, we would also study student reactions to potential/proposed solutions." * these are fine variables. Note that they are student-focused, so your target population is probably some version of "students". * reactions to proposed solutions are good, because they may provide evidence or impetus for someone to decide to implement, or not implement, that solution. - straw man solutions are not good to propose. get reactions to things that actually have a chance of being implemented - other questions about attitudes, past behavior, and background information, are also good to get, to get a better understanding of your survey results. "To study trends in the data, we would ask about background information such as Student Year Level, Major, Gender, Citizenship status. We would also seek to measure basic facts such as the overall percentage of students with personal access to old course materials, if they have ever utilized this access, and through what means they have such access." * these are good background questions. What about membership in greek houses and other organizations that might save this information? (actually it might be good to compile a list of such organizations, before you do the survey!) "Finally, one of the questions we would ask would be about a campus-wide, official university-sanctioned system modeled after that of the one recently proposed at Case Western Reserve University (see Part C)." * sorry, in a quick skim I didn't see what system was proposed at CW. * it would be good to provide evidence [in your proposal to me] that such a system could have a chance of being implemented at CMU ========================================================================== Project Proposal 2: Bar Workers' Perspective of Law Requiring Establishments to Close at 2AM OVERALL: * EVERYONE can do a better job of convincing me why each of their projects is interesting enough to actually bother to do. * I did not think this topic was as well thought-out as proposal #1. You have a potentially interesting policy question but a survey of bar workers doesn't seem to be the right way to gather information that could cause action on the question. Moreover, it may be implausible to change the closing law (as some of the articles you cite suggest), and so working on a related question might make more sense. * For assignment I.2, I will want you to (i) think about modifying the main topic or research question to something that might actually be acted upon. (ii) change the target population, if need be, to focus on people whose opinions will matter for acting on the results of the survey. NOTE: I wrote most of this before class last Thursday, where Victoria clarified that you didn't mean to survey only bar workers. A general voter survey would be interesting. Separately identifying stakeholders (bar workers, bar owners, bar patrons, bar neighbors, etc.) would also be interesting. (iii) carefully answer the other questions below about TARGET POPULATION, SAMPLING FRAME, METHOD OF DATA COLLECTION and VARIABLES. (a) Interesting research question? Doable? Each team member provides one piece of previous research or results on this question? * I think this is a potentially interesting question but surveying bar workers seems likely to produce and answer that no one but the bar workers themselves cares about: - only one of the bar workers' answers is really interesting: - if the bar workers don't want the bars to be open later, that's somewhat interesting, and the reasons might be more interesting, even though no action would be taken. - if the bar workers do want bars open later it's a bit "ho-hum", since it would be in their economic interest to stay open. - there is not much bar workers can do about the closing law. bar owners set closing hours within the limits of the law, and city council sets the law, responding to the general voting public [not just bar workers] who elect them. * The citations you provided are interesting and relevant. However, they mostly point to reasons to not change to a later closing law, and in one case implicitly makes an argument for making closing time even earlier. So again it may be implausible that anyone could actually do anything with your survey results. * Is there a version of this research question that relates to something someone can, and would be interested in, doing something about? A different target population? A different set of questions on a related topic? Please modify this proposal accordingly for I.2. (b) What population do you want to make inferences about? * Since you answered ABC from the shorter outline you haven't answered this yet. * pls answer carefully for I.2: TARGET POPULATION I've argued above that "Bar workers" is probably not the right target population for the questions you are raisin.g Since you will probably be changing the research question somewhat anyway, it is worth thinking through exactly what target population will make the most sense for your modified research question. (c) What population will you sample? Is this different from (b)? * Since you answered ABC from the shorter outline you haven't answered this yet. * pls answer carefully for I.2: SAMPLING FRAME * Since you answered ABC from the shorter outline you haven't answered this yet. For example, if you were to make "bar workers" your population, how would you get a list of bar workers from which to choose a random sample? [that list would be your sampling frame] Or would you have some other way of getting to bar workers? That defines your sampling frame. This will obviously also be an issue if you change the target population. (d) How do you plan to carry out the survey & why? * pls answer carefully for I.2: MODE OF DATA COLLECTION * Since you answered ABC from the shorter outline you haven't answered this yet. Here are some important things to think about: - Face to face interviews make for great response rates but would be at least very time consuming to carry out. - if you could get a student phone book from each campus, you could conduct a phone survey. this might be effective at getting better response rates but not use up a ton of your time. - if you could get a list of email addresses for each campus, you could conduct an email or email-and-web survey. - for something like academic integrity / cheating, which mode of data collection (face to face, telephone, email, web, etc.) would get you the most accurate responses from each person in the survey? Do you have to worry more about guaranteeing confidentiality? - for email/web surveys nonresponse will be a bigger issue. - for email and web surveys, get only about 25% or less response rate. - how can you improve this with followup and/or prenotification (e.g. postcards)? how many times, and how, will you followup? - who responds, who doesn't? how would this affect biases in your results? - if you are stuck with low response rate, what would you need to know or be able to do, to argue either 1. the sample you got is still like a representative probability sample; or 2. you know what adjustments to make so that the sample can be treated as a representative probability sample (e) What variables will you measure? * your main variables arise from the question "How would additional hours of operation benefit or disable regular bar-attending clientele, bar employees, and general residents of Pittsburgh?" are sensible, but I do not see how you could get this information from bar workers. Would you not have to survey those populations to find out how they react, or what costs/benefits they perceive? * pls answer carefully for I.2 (f) Sample questions for off-campus project? * pls answer carefully for I.2 ================================================================== Total grade: Project Proposal 1: (out of 48; 8 pts/part) Project Proposal 2: (out of 56; 8 pts/part) ----------------------- Total 104 NOTES: On I.1, everyone gets 100 points, regardless of the comments I made above. On I.2, I will grade each part about out of 8 pts/part. This come out to a bit more than 100 pts, which will just be extra credit for a team that really nails I.2. Each person on the team gets the team score recorded in the gradebook. If your team gets 84 pts on I.2, then each team member gets 84 pts in the gradebook. ==================================================================