1. Experiments and Observational Studies

Task: Farm Plots

      The farm plot exercise will be completed as an introduction and conclusion to experimental design.  The first day will be a chance to freely try different approaches to maximizing farm profits.  The second day will use a more rigorous, experiment-minded approach to drive long-term learning and follow the rubric below:
      • Team shares approach doc with teammates and teacher and calls it [Name, Name, Name] Farm Plots
      • 1pt: Team lists all planned experiments for the study.
      • 3pts: Results are tracked and a conclusion is made after each experiment.
      • 2pts: Summary of overall approach (at the top of the document) clearly communicates an understanding of the experiment design process and its benefits.


      Mastery Quiz Prep

      Not all hypotheses can be tested with a proper experiment.  Explain why each of the following would be very difficult / impossible to test.
      1. Jason wants to determine how gender is related to political party preference in his town. 
      2. The military wants to study the long term effects of a new biological weapon on humans. 
      3. A psychology group wants to study the effect of making poor people millionaires 
      4. The math department wants to study the differences in academic performance of students learning a course primarily through lecture and students learning primarily with reverse-classroom. 
      5. A student wants to understand how complete social isolation affects people for a year.

      For the problems below, you need to prove each of the statements with an experiment.  Answer a-j for each:

      • a)  What/who are the experimental units/subjects? 
      • b) What is the explanatory variable? 
      • c) What should the treatments be (the options for the explanatory variable)? Is a placebo / control group needed? 
      • d) What is the response variable? Is it quantitative or categorical? 
      • e) Without proper experimental design, what lurking variables could be confounded with the explanatory variable? 
      • f) Explain how you would do a randomized comparative experiment with this situation. Is blinding necessary? 
      • g) Can you ethically ask volunteers to participate in this experiment? Is there any information you would need to hide from the volunteers? 
      • h) If you have human subjects, is it likely that your subjects could follow the directions of the study and carry it out properly to its completion?
      6. Premium gasoline (89 octane) gives cars better gas mileage than regular gasoline (87 octane).

      7. A server who suggests the most popular appetizer to customers at Restaurant A will make more appetizer sales than a server who asks “can I start you out with an appetizer?” 

      8. A new hip replacement procedure, when compared to the existing common procedure and to no procedure at all, will lead to more natural walking (as rated 1 to 10 by physical therapists) 2 months after surgery. 

      9. Taking a recently developed pill each day will reduce the number of headaches experienced over the next 3 months. 

      10. A person who smokes is more likely to get lung cancer than someone who does not smoke. (Hint: if your experiment is nearly impossible to do well with humans, use rats). 

      11. A 2” diameter tree (measured at the trunk) has a greater chance of surviving a transplant (being replanted in a new location) if you water every day for 20 minutes than if you water every other day for 40 minutes.

      Block the subjects in the experiment below by their math grades.

      12.    An education study wants to compare student performance in a problem-set-only course and a course infused with multiple large projects. The students in the table are available for the study:

       Name Last math grade
       Bill 67
       Mary 74
       Suzy 83
       James 92
       Josh 89
       T.J. 93
       Missy 78
       Kirsten 98

      >> 14 and 16 on attached PDF below

      Below you are given an experiment or observational study.  Answer the following for each:

      • a) Is this an observational study or an experiment? How can you tell? 
      • b) If it is an experiment, explain how randomization, replication, and a control are used. 
      • c) What population do the individuals/experimental units/subjects come from? 
      • d) The explanatory variable is the type of treatment. What are the treatments? How many are there? 
      • e) What is the response variable? Is it quantitative or categorical? 
      • f) In this study, imagine that your test had statistical significance. What conclusions CAN you make? List a few things you CANNOT conclude (based on whether you have an observational study or experiment, what population the subjects come from, and if randomization was used properly).
      17. A company wants to know how much cold temperatures affect the elasticity of their rubber bands. 100 random bands from a box of 200 are placed in a freezer while the remainder are kept at room temperature. The amount of stretch before breakage is measured on each rubber band, and the average stretch distances are compared.

      19. A set of volunteers is posed the following thought experiment: if there were 5 people on a railroad track about to get hit by a train, but at the last second you could pull a lever and divert the train to another track where 1 person was standing, would you do it? They were then asked a second question: if there was a single track with 5 people about to get hit by a train, and you could push a man off the bridge overhead at the last second to bring the train to a halt, would you do that? The results of the two responses were later compared. 

      20. A pharmaceutical company wants to compare two varieties of a drug they have developed to alleviate allergies. The study director first described the two varieties and their differences to the pool of volunteers. Then each person decided which drug better fit their symptoms in order to find the most significant effect. Finally, the effectiveness of the two drugs (based on number of allergic reactions of the volunteers) was compared to decide which one to send to market. 

      21. Milgrim’s famous obedience experiment had a volunteer designated as the “teacher” and a lab helper as the “learner” (though the volunteer was told that the “learner” was just another volunteer in the study). The teacher would ask the learner a question, and if it was wrong, the experimenter ordered the “teacher” to give an increasingly more painful electric shock (or so they thought – since the “learner” was behind a wall, they were only faking the pain of the shock, but the “teacher” did not know this). The experimenter recorded how far people would go on the shock scale (from 30V to 450V). The results were shocking (pun slightly intended): 65% of the subjects administered the highest level shock of 450V despite cries of pain from behind the wall. 

      22. Birds eat fertilizer out of farmer’s fields, costing the farmer money and sometimes killing birds. A chemical company is designing a new coating for its fertilizer that will be less attractive to birds. They developed 4 varieties – red, purple, green with thin white stripes, and orange with thin purple stripes. A set of recently captured birds will first be held overnight with no food. Then, they will be randomly selected, one at a time, to go in a cage with a specific amount of food in one of the fertilizer colors. The observer will track how many granules of food are consumed by the bird. Finally, they will average the number of granules of each color eaten and compare to determine which is eaten the least.



      Free Response Prep
      • In order to test people's willingness to obey an authority figure against their conscience, the psychologist Stanley Milgram used scientists in white lab coats to ask subjects to provide an electric shock to a "student" who incorrectly answered questions on the other side of a curtain.  The subjects were told that the shock would help the student learn better and motivate correct answers.  In reality, the person behind the curtain was not actually being shocked, but was just an actor who was helping with the experiment.  The result of the study was that most people would apply the highest level shock, despite their reservations, because the scientist in the lab coat told them it was necessary.
      • Additional information of the scenario: YouTubeWikipedia
        • Question 1: Discuss the benefits of this research.  Justify why the experiment had to be conducted the way it was.
        • Question 2: Discuss the ethical problems with this kind of research.
        • Question 3: Explain what an IRB is and why they exist today for human-subject research.


      Practice solutions
      1. Gender cannot be randomly assigned -- the best you can do is an observational study of gender, but not a true experiment.
      2. It is not ethical to have people, even volunteers, be used to test a biological weapon.
      3. If low-income people were randomly assigned to be millionaires, that would cost a lot of money.  If you just looked for lottery winners, that would not be random assignment according to the setup of the study.  If you wanted to do a study of low-income people who buy lottery tickets (as opposed to all low-income people), then you might be able to consider the winners as randomly assigned and carry out an experiment.
      4. Because of scheduling conflicts, it is not truly random who ends up in each period.  Also, since students would know which method they are using, they may be biased and sabotage the experiment.  Data can still be gathered on the different methods, but it is not ideal experiment conditions.
      5. It is not ethical to socially isolate someone for a year to satisfy your curiosity.
      6. Gasoline
      • a)  cars
      • b) Type of gasoline
      • c) (1) Regular 87 octane gas, (2) premium 89 octane gas [the regular gas would be considered the control -- no other control group needed]
      • d) Gas mileage (quantitative)
      • e) Type of car, type of driver, road conditions, length of commute, others
      • f) Randomly assign half of the volunteers to use regular and the other half to use premium gas.  Don't tell the drivers which type of gas is being used in their vehicle so it doesn't affect how they drive (blinding).
      • g) No ethical problems -- the only thing to hide is what type of gas they get.
      • h) The only rules would be to not fill up with more gas or siphon gas out of the car until the measurement is complete.  This could be an issue if someone needs to get to work and is low on gas before the study is done.
      7. Appetizers
      • a) Customers at the restaurant
      • b) What the server says
      • c) (1) suggesting a specific appetizer, (2) saying "can I start you out with an appetizer?" [the second option would be considered the control -- no other control group needed]
      • d) Depends on how you interpret the question.  If you say "will make more customers buy appetizers", then it is categorical (the customer either buys or doesn't).  If you say "will increase the average number of appetizers per person", then it is quantitative. 
      • e) Which server, type of customer, time of day, family size, others
      • f) The manager could randomly assign tables to receive different treatments.  Since the customer wouldn't need to know they are part of this study, the customer is blinded.  It would be nice to blind the servers too, but since they need to be the ones who ask the question, that is impossible.
      • g) No ethical problems -- there is nothing wrong about running this experiment and not telling the customer.
      • h) Yes -- the subjects just have to order an appetizer or not.  The harder part would be to make sure the servers ask the right questions.
      8. Hip replacement
      • a) People who are receiving a hip replacement
      • b) What type of hip procedure you get
      • c) (1) new hip replacement procedure, (2) existing common procedure, (3) no procedure [the baseline control group]
      • d) How naturally the subjects walk (as rated 1 to 10 by physical therapists) 2 months after surgery -- quantitative
      • e) Exercise / therapy (the no procedure group would likely do no extra work if they knew they were a control group)
      • f) Study director would randomly assign patients to the treatments.  The patient should be blinded -- this means that they would need a fake surgery for the control group.  The doctor should also be blinded until they need to actually perform any procedure.
      • g) Yes -- you would just be up front about what the different possible treatments are, and as long as they understand that they might get the sham surgery with nothing done, it is very ethical.
      • h) Patients might not do their recommended exercise each day as part of the treatment process.
      9. Headaches
      • a) People who take the pill
      • b) If you take the pill or not
      • c) (1) taking the new pill, (2) taking a placebo [this group is added as a control for comparison to the new pill]
      • d) Number of headaches experienced over the next 3 months -- quantitative
      • e) If you are taking the real pill, and know it, you might assume that headaches will go away, causing them to go away.  If anything can be influenced by the placebo effect ("its all in your head!"), it would be literal pain in your head.
      • f) Half of the volunteers in the study would get randomly assigned to the real pill and the other half get the placebo.  For the placebo to be effective, the patient needs to be blinded.  The person talking to the patient should also be blinded (so, double blind) to reduce that as a confounding factor.
      • g) Yes -- it is a very standard clinical trial comparing a new drug vs. and placebo, as long as you are up front about what people could get.
      • h) Maybe -- some subjects might forget to take the pill for a day or two, messing up the results.
      10. Smoking
      • a) You can't use people.  If you did, you would need to randomly assign half of a group of non-smokers to start smoking for the rest of their lives without quitting in order to gather data for the study.  People would never want to do this, and it isn't too ethical given what we know about smoking already.  However, you could use rats and force half of them to start smoking (or breathing in something equivalent).  Though this is not ideal, it could at least establish causation within a living creature.  Many experiments that are tough to do on humans for cost or ethical reasons are done first on animals.
      • b) If the subject smokes
      • c) (1) smoke, (2) does not smoke [this is the control/comparison group]
      • d) If you get lung cancer or not during your life -- categorical
      • e) If not randomly assigned to treatments, people who smoke might be more likely to live a different lifestyle with other risky behaviors (drinking, driving a motorcycle, etc).
      • f) Half of the rats would be assigned to "smoke" and the other half would not and their lung cancer risk would be compared.
      • g) Unless you are not into experimentation on animals it should be okay.
      • h) n/a
      11. Trees
      • Let's be honest -- you didn't want to do #11 either.
      12. Blocking
      Since the study is comparing 2 treatments, students should be placed in blocks of 2.  The 2 highest scores, then the next 2 highest, and so forth should be in a block.  So:
      [Kirsten, T.J.], [James, Josh], [Suzy, Missy], [Mary, Bill]
      The reason you might want to do this is because it makes it less likely that one group starts with all the stronger students -- it minimizes lurking variables (confounding factors).

      13-16: see attached PDF file

      17. Rubber bands
      • a) Experiment -- there are 2 treatments and the experimental units (rubber bands) are randomly assigned to them
      • b) Control: one set of bands is kept at regular temperature but everything else is identical in the treatments
            Randomization: randomly half of the bands are in the freezer group and the other half are in the control
            Repetition: there are 200 bands, plenty to get good randomization
      • c) **What population do the individuals/experimental units/subjects come from?** (sorry about the typo): the company's rubber bands
      • d) Freezer and a standard room for some period of time -- there are 2 treatments
      • e) The amount the rubber band stretches before breaking -- quantitative
      • f) Since this is a valid experiment, you can claim causation (you can claim that the thing you studied was caused by your treatment).  Since the population under study was only one company's rubber bands, you are limited to claiming things only about this company's rubber bands and not all rubber bands.  Thus if the freezer bands broke statistically significantly shorter than the room temperature bands, you could claim that this company's rubber bands break sooner at cold temperatures.
      19. The moral train
      • a) Experiment...though not well run.  There are 2 treatments (questions) given to each subject in a matched pairs design.  There is no randomization with the order of the treatments.
      • b) Control: if you are not familiar with the experiment, either could be considered a control to the other.  Given the intent, the "pulling the lever" treatment is probably the control.
            Randomization: the order that subjects answer the questions should be randomized, but right now it is not
            Repetition: the size of the set of volunteers would determine this 
      • c) It does not specify where the volunteers come from.  It is typical to assume that many psychology studies use college students as subjects, so perhaps the volunteers do not represent all generations equally.
      • d) There are 2 treatments -- the 2 questions: "would you pull the lever?" and "would you push the man off the bridge?"
      • e) If the subject would do it (would say yes) -- this is categorical/binomial
      • f) You could conclude that more people would say yes to one question than the other.  Since it is an experiment, you can claim causation -- the difference between the lever question and the pushing question is what caused people to change their answer.
      20. Pharm. company:
      • a) Observational study -- there are two treatments, but patients chose which one they want, so the best you can do is to observe how they each do.  There are multiple variables now being confounded, making it impossible to determine causation
      • b) Control: either drug acts as a control / comparison to the other -- no clear control
            Randomization: none -- this is the problem!
            Repetition: only in the single setting with the unknown number of volunteers.
      • c) Unknown.
      • d) The 2 varieties of the drug
      • e) number of allergic reactions in a given time period
      • f) Since this is only an observational study, it will be harder to claim what is the cause.  Assuming the population the volunteers came from is all residents of the Rochester area, then the study conclusions would show that, when patients choose the best fitting medicine, the group using one of the medications improved more than the other.
      21-22 will be discussed in class.

      Notes
          AP does not use confounding / lurking terms, instead DESCRIBE the issue -- ASSIGNING treatment is the difference between experiment and observational study
          replication is not repeating one individual, it is using many different individuals from the population


          Ċ
          Andy Pethan,
          Nov 24, 2013, 10:00 PM
          Ċ
          Andy Pethan,
          Dec 3, 2014, 10:28 AM
          Comments