163 14.10 Data from Scholarly Sources
Scholars collect and report on data as part of their work as new knowledge creators. Physical scientists collect data from the innermost reaches of quantum space to the outermost reaches of the universe. Social scientists gather huge datasets about individuals, communities, governments and businesses as they try to understand how society works. Humanists gather data about languages or codes of writing, architectural developments, music, and other topics related to human beings’ creativity. Scholars also generate data when they report on their research work.
The scientific method is designed to generate publicly-verifiable results. Scholars are supposed to do their work under conditions that are reproducible by others. Scholars also try to minimize factors that affect the accuracy and interpretation of their results. Hence, they try to conduct their work under controlled circumstances as much as possible.
This is sometimes easier for physical scientists than for social or policy scientists, however. Physical scientists can do their work in the laboratory and/or with multiple observations of the phenomena they’re studying. Social scientists do much of their work in the social environment and are studying complex human behavior, thereby making control harder to achieve. Nonetheless, all reputable scholars strive to be precise in their methods and careful in their interpretation of research findings.
As much as scholars want to achieve this perfection of the scientific method, they are, in the end, human beings. The data they generate must be scrutinized and used with caution, just as with any other source of information. Critical thinking experts M. Neil Browne and Stuart Keeley suggest that you keep a number of factors in mind when you are considering using scholarly research data as a source of information. These include:
-
Research varies in quality, and because the research process is so complex and subject to so many external influences, even well-trained scholars sometimes make mistakes. Just because something was published in a peer-reviewed journal does not mean it is not flawed in other important ways. This is why it is important to try to interview several scholars about the findings of a particular study in order to understand the reputation of that work.
-
Research reports often contradict one another. Thus, single research studies presented out of the context of the body of work in that area often present misleading conclusions. You should look for studies that have been repeated by more than one researcher or group of researchers.
-
Research data do not “prove” conclusions. All someone can do with a set of data is say it does or does not “support” a particular conclusion. Scholars must always interpret the meaning of their findings, and all findings can be interpreted in more than one way. Thus, when you see the statement, “research findings show…,” you should translate that statement into “researc