It should be quite apparent by now that the answer to this question is, "very differently." In this section, I shall try to provide a brief summary of several major types of research. For a more complete study of methodologies, please see Beach and Bridwell; Lauer and Asher; Phelps; Kantor, Kirby and Goetz; Flood, Jensen, Lapp, and Squire; and Hillocks; among others.

Quantitative Studies

Experimental designs are used most often when the researcher wants to compare one instructional methodology with another. The method depends upon severe control over the environment to insure that the "subjects" (i.e., student writers) are "randomly assigned to treatments" (i.e., experimental or controlled teaching conditions). The researcher begins with a "null hypothesis" (e.g., "There will be no significant differences between treatment A and treatment B on the criterion variable") and tests this hypothesis using statistical inference (e.g., "analysis of variance").

To translate this into "compositionese," researchers might decide to test the effects of using a particular computer software package to teach students to revise. They would want to compare results with this package to traditional methods. To insure that their statistical procedures would be valid, they would have to assign students randomly to separate groups that would get the different kinds of instruction. They might also want a "control" group that would have no instruction at all in revision. Next, they would have to determine what measure they would use to determine successful revision. A typical measure might be holistic ratings of differences in quality between first and second drafts of a paper composed by everyone in the experiment. They would then study the improvements in drafts and statistically test to see whether the variability in ratings could be systematically attributed to the treatments. They would decide this on the basis of a "probability" level that would be set in advance of the experiment. (See Lauer and Asher; Hillocks for extended explanations of experimental designs.)

The chief advantage of this kind of study is the clarity with which it is understood by the empirical research community and by consumers who like "yes/no" answers, or perhaps even "maybe" answers, so long as the degree of certainty or the percentages are reported. An argument can sometimes be made for a cause-effect relationship between treatments and outcomes, if the study is carefully conducted and if the theoretical justification behind the study is strong enough. These kinds of arguments using statistical evidence can be extremely powerful, particularly in efforts to obtain resources for costly instructional equipment or materials.

The obvious liabilities with experimental research are the artificial nature of the experiment and the lack of attention to contexts or differences within individuals. Statistical studies deal with patterns across large groups, and they depend upon controlled conditions if the assumptions upon which the statistical tests are based are to be satisfied. Classrooms and students don't often fit this model of research in the ways that rats or cornfields often do. Furthermore, good rat or cornfield researchers would never let themselves get caught with outcome variables as abstract and "ill defined" as some of the ones we have in writing (e.g., "good" writing, ability to compose). They use "speed through a maze" or "bushels per acre," variables that are easily counted or measured. Their studies do not often describe the lives of their subjects for many hours per week over several months. With such limitations, they really can "control" their experiments, unlike most composition researchers. In composition, we have difficulty grafting "soft" and subjective research questions onto an inappropriate research rhetoric from agronomy or behavioral science.

"Random assignment to treatment" illustrates one of the ways composition research often violates the assumptions behind experimental research. Students register for classes at a certain hour or in a certain location, and they may be unwilling to be assigned to other sections just to satisfy the researcher's needs. It may be very difficult to offer several different treatments within the same classroom without introducing "contamination" into the experiment. In many so-called experimental designs, the researchers assign the treatments to pre-existing classrooms, thus introducing potential bias (e.g., students in the 1:00 P.M. "control" class were sleepier than students in the 10:00 A.M. "experimental" section; somehow more honors students ended up in a particular section).

More subtle differences have to do with assuming that everything that is important is uncovered in an experimental design. The experimenter may deliberately stay out of the classroom to avoid the contamination that comes when subjects know they are being studied. Those in the classroom might be aware of all kinds of influences operating on the environment that were not a part of the formal research design (e.g., two weeks before the hypothetical study above, the students had been forced to try a poorly designed software package and had negative attitudes toward computer-assisted writing instruction). Of course, this kind of influence might be reported to the researcher who could then use it in his or her interpretations, but many times such influences are missed because of the barrier between the "researcher" and the subjects and their teachers, in the case of classroom research.

Another problem arises in assuming that the results of a particular study will apply in another context. One of the chief advantages of conducting experimental research, according to its proponents, is that we can safely "generalize" from one study to another. The irony is that so much control goes into experiments that we are unlikely ever to see the same conditions in a more natural setting. With our current interest in promoting diversity within composition, research designs predicated upon "averages" and "generalizations" may not be the method of choice. Nevertheless, many experimental studies produce results that ring true and are useful.

META-ANALYSES

Once a body of experimental (or "quasi-experimental," Lauer and Asher) research exists on a certain topic, the consumers of research are then faced with the problem of connecting results from one study to the next, particularly if the results are mixed. A technique called "meta-analysis" has become popular among social scientists because it allows investigators to blend findings from a number of studies, using special procedures for controlling certain types of statistical errors and design differences. Smith and Klein offer the following list of synthetic studies: literature review, research review, interpretive analysis, integrative review, research integration, meta-analysis, state of the art summarizing, evaluation synthesis, or best-evidence synthesis. Of these, only metaanalysis has its own rigorously defined rules for statistical analysis.

In one of the few such meta-analyses in our field, Hillocks constructed three broad categories--mode of instruction, focus of instruction, and duration of instruction--by combining results in individual investigations so that he might determine the factors that had the greatest influence on improvement in writing. A comprehensive and systematic meta-analysis such as Hillocks' allows composition researchers to synthesize findings from a number of sources. The weight of such evidence, scrutinized by rigorous statistical procedures, can be persuasive.

There are, however, a number of problems. Lauer and Asher outline some of the debates over the method, including problems with differences in research designs, variations in the quality of studies, biases depending on the source of the studies (e.g., refereed journals versus dissertations), the time period from which the studies are sampled, and so on. In addition, meta-analysis amplifies all the epistemological problems with positivistic quantitative research. We must subscribe to a view of the world that allows us to define and control variables, treat subjects, and measure effects in quantitative terms in order to place value on these kinds of inquiries.

QUANTITATIVE DESCRIPTIVE STUDIES

Lauer and Asher use the term "quantitative descriptive studies" to describe studies that examine variables with statistical measures (e.g., descriptive statistics such as mean, median, mode, standard deviation, and comparisons such as correlation). They are unlike experimental studies in that they do not compare experimental groups to each other or to control groups, and they do not involve a treatment. This approach was the one most widely used by the process researchers cited earlier. In my own study of revision, for example, I wanted to determine what kinds of revisions students could make in their writing after twelve years of instruction. I identified thousands of revisions in 200 first and second drafts of one assignment, classified them into categories, and used a regression analysis to determine whether certain kinds of revisions were associated with improvements between the first drafts and the second drafts. This kind of approach was typical in the late 1970s and early 1980s as researchers identified features of composing processes and described them quantitatively. (See Lauer and Asher for additional examples and methods.)