"I'm afraid I'm sunk. It's not working out at all the way I'd planned. So what's to be done?"

Troubles can crop up at any point in the process of doing theses and dissertations. However, the data collection stage seems particularly vulnerable to problems, so we've chosen this place in the book to identify a series of familiar difficulties and to propose ways of managing them. Throughout the chapter the presentation is cast as conversations between a succession of distressed graduate students and their academic advisors, with the students describing their troubles and the advisors suggesting potential solutions. It should be apparent that the suggestions offered by the advisors in these anecdotes are not ones that all advi sors would give. In effect, these examples illustrate no more than a few of the potential solutions that can be attempted to cope with things that go wrong.

The issues inspected in the five cases concern projects that (a) failed to achieve the authors' desired outcomes, (b) included uncooperative participants, (c) yielded statistically insignificant results, (d) failed to meet a professor's concept of an adequate number of subjects, and (e) the author feared would be less than perfect.

SHATTERED EXPECTATIONS

Student. "As I've collected more and more interviews, I realize my project's turning out all wrong. A lot of the kids aren't answering the way they should."

Advisor: "What do you mean by should?"

Student. "I asked them what consequence they would recommend for the main character in each of eight cases of wrongdoing. So I expected kids at the juvenile detention facility to give different answers than the ones in the church school. But while there's lots of variety among the consequences suggested within each of the groups, there doesn't seem to be any significant difference between the pattern of answers of the two groups as a whole. Something's wrong."

Advisor. "Obviously. But maybe what's wrong is that you've been hunting for evidence to support your preconception rather than accepting reality as revealed by your interviews. Why not follow your data instead of trying to force your hypothesis onto the data? The fact that your interview answers surprise you should be good news--you've learned something new; and the readers of your thesis will also learn something worthwhile if you interpret the outcome insightfully. If you accept the interview results as accurately showing diversity within each of the two groups rather than showing a significant difference between the groups as a whole, then you have valuable evidence about the danger of stereotyping people who are members of each group. Your interpretation task can now be one of investigating likely reasons behind such an outcome. Your project hasn't gone wrong. For me, your actual results are more interesting than the results you'd hoped for. Follow your data."

UNCOOPERATIVE PARTICIPANTS

Student. "I was trying to learn what changes in teachers' treatment of students result from teacher-parent conferences. A study like this had been conducted in a small California community and I wanted to see if similar changes would occur in schools in a South Sea Islands community as well. I wanted to compare the results from California with ones I collected in American Samoa. The Department of Education in American Samoa was fully behind the plan, so we had successful training sessions for teachers in how to conduct parent conferences. Then we had the teachers set schedules for their meetings with each child's parents. But at the end of the six weeks during which the conferences were to be completed, we discovered that hardly any teachers had followed the plan. They didn't object to the plan; they just didn't have any conferences. It was passive resistance, simple stonewalling. So I ended up with no results to report. I'm ruined."

Advisor. "Not necessarily. The interesting question here is: Why did the California teachers carry out the plan whereas the Samoan teachers did not? What you might do to save your project is to change the question you were trying to answer."

Student: "But in the middle of your project, you can't change your topic. I'm told that's not acceptable scientific procedure."

Advisor: "The purpose of research is to discover something that's a worthwhile contribution to knowledge. Why can't you change horses in the middle of the stream if it can save you from disaster and produce a desirable result? Finding that your plan succeeded in California and failed in Samoa is worth explaining and can provide guidance for other researchers who pursue studies in diverse social settings. Your new task can be that of identifying the factors in the two cultures--those of a California community and American Samoa villages-that account for the conflicting outcomes of teacher-parent conferencing in the two settings. This likely means conducting interviews with teachers and parents in each setting to learn their views of the proper roles of teachers and parents. Or, since you aren't Samoan, the interviews with the islanders might well elicit more candid responses if one of your Samoan colleagues did the interviewing. In any case, it seems to me that you can produce a first-rate contribution by setting aside the original question--'What changes in teachers' treatment of students result from teacher-parent conferences?'-- in favor of a new one--'What cultural factors influence the conduct of research in diverse social settings?' Your original plan and the way you implemented it are still important elements of your study. The aim, however, becomes that of explaining why the plan didn't work as you'd hoped."

NEGATIVE RESULTS

Student: "As I'd intended, I divided the philosophy class that I teach into two groups. I taught the unit on forms of logic to one group by my traditional lecture and discussion method. Then I had the other group study the same material individually on computers, using the program I wrote about forms of logic. Finally, I tested both groups, but I got negative results. It didn't work out."

Advisor: "What do you mean by negative results?"

Student. "There was no significant difference between the two groups' average test scores. The averages were nearly the same."

Advisor: "So that means your experiment failed?"

Student: "I figured one group would do better than the other, so I'd know which teaching method was better. That would be something worth reporting."

Advisor: "It seems to me you have plenty that's worth reporting. Let's imagine that a student misses your lectures about logic. So you can have her try the computer program on her own, being rather confident that she'll master the material as well as if she had attended the lectures. Discovering that on the average the two methods work equally well is good news. Your experiment wasn't a failure. You revealed something useful, and your readers can profit from that.

"Of course, what you don't know is which method might work better with some students than with others. It's the aptitude-treatment-interaction issue-the notion that some learners' characteristics equip them to succeed better with some instructional methods than with other methods. That's something you might want to investigate in the future by adopting a different experimental design than the one you used.

"The real danger in getting negative results comes from having chosen a noncontroversial topic for your experiment or survey in the first place. Let's say you chose to investigate the question 'Can individual differences among students in learning to play tennis be erased (so all succeed equally well) by their eating the same diet for three weeks?' This is a noncontroversial matter, because essentially no one beyond the years of childhood will imagine that diet is the sole determining factor differentiating one player's skills from another's. Thus, in order to produce a study of such a question that anyone takes seriously, you will have to get positive results. Your study will have to show that, within a heterogeneous group of tennis students, all of them demonstrated the same skills by the end of the three-week diet. Such a result would be dramatic, indeed, and warrant a lot of attention in both academia and the public press. But if you get negative results, showing that after three weeks there were still notable differences among learners in their tennis skills, people will simply think you weren't very bright for ever presuming otherwise. They'll think you were stupid for doing such a study.

"Thus, what you need is a true issue, one that reasonable people can disagree about. Then, whether you get positive or negative results from your survey or your experiment, the results are meaningful. Such has been the case with your study of lecturing versus a computer program. Before you conducted your experiment, the outcome would have been--in most people's minds-indeterminate; so doing the experiment would be worthwhile, no matter which way it turned out."

THE MEANING OF "ENOUGH SUBJECTS"

Student: "When I told Professor Green about my data, he told me I didn't have enough subjects. He said I'd have to get more. But there aren't that many autistic kids available, and it takes a lot of time--about four months--to apply my training program with each kid."

Advisor: "How many children are you using?"

Student: "Three --ages 8, 10, and 11."

Advisor: "Did Professor Green say how many he thought you needed?"

Student: "He said it would be best to have at least 30, so I could use largesample statistics. And if I don't have that many, I won't find out anything worth talking about."

Advisor: "So, what you need is a line of reasoning that persuades potential critics that the result you produce with only three autistic subjects will be of value?"

Student: "That's my hope."

Advisor: "Well, here's one possible approach. When you are deciding how many subjects you need for an experiment or survey, you answer two questions: What am I trying to test or reveal? How broadly do I wish to apply the interpretation of my study? Now, let's direct those questions at your autism project. What's your study supposed to test or reveal?"

Student: "A Professor Baron-Cohen at Cambridge University has a theory of autism which assumes that children's social behavior is crucially influenced by their ability to estimate what other people are thinking. According to BaronCohen, autistic children suffer from a genetic disorder which damages their ability to estimate other people's thought processes. He's reported some limited success in training children to make better estimates. But apparently nobody has tried to do that with pictures and videos. I've worked out a training system for using photos and videos of people that I think could help autistic kids improve their skill in judging other people's thoughts. I want to test out that training system on these three autistic kids to see if it works at all. And if it does work, is it more effective with one child than with another, and why? I think if I train them over a four-month period it should be long enough to see if my scheme is any good."

Advisor: "So, if you get positive results and the children do improve in their social perceptions, what research question does that answer?"

Student: "Well, it's this--'Can autistic children's understanding of other people's thought processes be improved by training the children with pictures and videos of people in social situations?'"

Advisor: "But what if you get negative results? What if the children's ability to estimate others' thoughts doesn't improve? Is such a result any good? Do you still have a viable thesis--something worth talking about?"

Student: "I think so. From Baron-Cohen's theory and the training he's done, I think it's reasonable to try my system. Trying it with three kids should give me some idea whether it's practical."

Advisor: "Even though 30 would be better, testing your system on three seems defensible. Now to the second question: How broadly do you wish to apply the results of your study? Do you plan to draw conclusions that can be applied to autistic children other than the three you train?"

Student: "Of course. That's why I'm doing it."

Advisor: "Then there's a two-part question to ask. It's the question I think Professor Green is concerned about: 'To which people, other than my three subjects, can my conclusions validly be applied; and how should I phrase my interpretation to make clear the conditions under which my results can legitimately be applied to other individuals who display symptoms of autism?'

"Now let's first speculate about how your interpretation might read if your system is at least modestly successful with your three subjects. Here's one way you might interpret such an outcome:

'The results of this study suggest that (a) at least some autistic children's ability to estimate other people's thought processes will improve under the study's training program and (b) some children's abilities will improve more than others'. The three subjects who participated in this research were all boys, all in the age range of 8 to 11, all from urban U.S. middle-class families, and all outpatients of a child guidance center. How well the treatment would succeed with girls, with other age levels, and with subjects from other types of homes is unknown. However, the outcomes of the study appear encouraging enough to warrant trying the treatment program with other kinds of autistic individuals to determine the conditions under which such treatment can profitably be employed.'"

"It seems to me that such an interpretation would be sufficiently specific and cautious to make the use of only three subjects defensible, since what you are attempting to test is the proposition that your training scheme will work at all."

Student: "But what if none of the kids improve? What can I say then?"

Advisor: "First, you can argue that it was reasonable to expect your program might work. You base that argument on Baron-Cohen's theory and the success he reported in training autists to recognize other people's thought processes. Next, you explain that your system did not improve your three subjects' socialobservation skills. You then speculate about possible causes for such a result; propose what sorts of studies might be conducted to investigate those possibilities; and offer an opinion about whether, in light of your scheme's apparent failure, such studies would be worth the cost and effort.

"So this brings us back to the beginning. The matter of how many subjects you need in an experiment or survey can depend on how you answer those two questions: 'What am I trying to test or reveal? How broadly do I wish to apply the results or interpretation of my study?' If you wanted your study to reveal how effective your scheme would be with all varieties of autists throughout the United States, you would need a large enough sample of subjects to fairly represent the range of influential characteristics within that entire population. In effect, you would need a large sample, drawn in a manner that convinced your readers that the participants indeed typified the nationwide autistic population's influential characteristics. Maybe Professor Green thought you planned to offer such a wide-ranging interpretation on the basis of your three-subject study. Why not organize your rationale so it fits what you are actually trying to do, then talk with him again?"

NOTHING SHORT OF PERFECT

Student: "I still can't decide which method to use for collecting information about people's use of credit cards. Every professor I talk with has a different idea."

Advisor: "Such as?"

Student: "One says I should do telephone interviews. Another says questionnaires is the way to go. And another recommends face-to-face interviews. Then one says I should do interviews and also get records from the interviewees' credit card company so I could compare what people say with how they really use their cards".

Advisor: "Well, each of those approaches obviously has its own advantages and disadvantages. Any one of them could yield useful results, depending on the exact questions you're trying to answer."

Student: "But I have to be sure which method's the right one. I don't want to get in the middle of it and find out I'm doing it wrong."

Advisor: "There isn't one perfect method. Each of those options offers a particular perspective toward credit card use."

Student: "But I want my thesis to be recognized as really important, so I want to do it right."

Advisor: "Maybe you're demanding too much of yourself--aiming at unquestionable perfection. Your study doesn't have to win a Nobel prize. Naturally you want your work to make a contribution, but you should recognize that doing a thesis is essentially a learning exercise and not your entire life's work. Doing the thesis gives you guided research experience. If you do a good job with any one of those data collection methods, you'll be making a contribution. Pick one method and go with it."

Student: "But I want to be sure it's going to work out right so I won't be criticized."