Statistical data helps social scientists understand and describe social phenomena, but should never be interpreted as infallible or irrefutable proof.’ While good researchers take precautions to ensure the reliability and validity of their findings, no one is capable of designing a perfect’ study that will yield perfect’ data.
To exemplify the aspects of research design, implementation, and interpretation that impact the credibility of one’s findings, I provide a simplistic, fictitious, social-psychological study, from which I will make my points.
Imagine that I have decided that I want to find out if exposure to media violence reduces psychological well-being. I have determined that this question is best answered through quantitative research methods. After developing my research question and conducting a literature review, I move on to making operational definitions of the variables. At this point, I must decide how I am going to measure both media violence’ and psychological well-being.’ I must ask myself what I will include in my definition of each variable.
A couple questions I consider are: (1) “what forms of media will I inquire about?” and (2) “what items can I use in a scale to measure psychological well-being?” If I select an existing scale of psychological well-being that only measures depression and anxiety, I could be overlooking other indicators of psychological well-being such as self-esteem and self-efficacy. How I define my variables can dramatically affect my results. This is why we often see conflicting findings. Even though two researchers may claim to be measuring the same thing, they have measured it in different ways. As a result, the findings are incongruent.
From here, I formulate a hypothesis or a set of hypothesis, grounded in the findings of others. I then must decide what specific questions I will ask on a survey to collect the data needed to be able to reject either the hypothesis or null hypothesis. My choice of wording can dramatically influence my findings, as can the options I give for responses to each question.
These considerations are even more important if I am creating a scale of my own, rather than one that has been previously tested for reliability by another researcher. I must make sure that the questions that make up my scale are actually getting at what I want to measure. For this reason, new scales are often tried out on a preliminary smaller sample in order to run statistical tests such as inter-item reliability.
Back to my original point, though, your wording on survey questions should be easy to interpret and straightforward. The questions should be in language that the average person will be familiar with. Response options should be exhaustive, or at least offer the alternative of a response such as “don’t know” or “other.”
In this example study, I could develop a question as part of a scale like, “How often do you have thoughts of suicide?” The responses could range from “never” to “more than once a day.” The problem lies in questions like, “How often do you feel depressed?” This question is going to be interpreted in different ways by different people. Depression includes a wide variety of emotions, sensations, and behaviors. Another problematic question is “How often do you think about killing yourself or killing someone else?” What is the problem with this question? Well, what are you actually measuring by it, depression, psychosis, or something else? In other words, questions should be limited to one variable or else the result is not reliable data.
Once the survey is created, I then have to decide what population I want to study and how I will sample that particular population. Herein lays another reason why social scientific data can never be taken as proof’ of something. The fact that you are using probabilities to generalize the findings from a sample to the larger population it represents will always result in an inevitable amount of error. This is when we talk of the p-value or significance of a statistic; we consider how representative the findings are from the sample to the larger population.
Furthermore, any choices I make about who to include in my sample can bias my data, making it less representative of the population I am trying to study. Another related concern is low response rates. What if I send out 1000 surveys and only get back 360? Are those 360 persons who responded representative of the population or is there a distinguishing characteristic between those who responded and those who didn’t? Again, this reduces the strength of any claims you intend to make based on this data.
After this, I also must be proficient in data entry, getting all the cases entered in accurately. I then must be adept at using the statistical package to analyze the data. After I get all the numbers, I have to interpret what they mean. At any of these stages, there are opportunities for human error.
Skipping ahead somewhat, what happens when I report my findings and someone intentionally distorts my findings to advance their own agenda? What if there are others who misinterpret or only partially acknowledge my findings? A possible outcome is that the information transmitted to the general public may be quite different from my actual report.
Through this example, I have highlighted some of the aspects of the research process that are opportunities for error that can affect the accuracy of the findings of any given quantitative study. Does this mean that statistics are useless? Not by a long shot, but we need to be aware of the limitations of statistics. We also should consider the credibility and expertise of the source. Furthermore, users of statistics should keep in mind that many data sets are contingent upon the reporting activities of individuals, groups and organizations.
Crime statistics are an excellent example; less than half of all crimes are actually reported to the police, so the numbers reported to the FBI (which are compiled into the Uniform Crime Report annually) are not truly representative of the actual crime rate. This has been determined by victimization and criminal surveys. Also, not all law enforcement agencies turn in their data to the FBI. Therefore, when we see the crime rate on the UCR, it is only representing a small portion of crimes actually committed. However, this data can still be used to examine other social problems or forces.
The bottom line is that each person must use their best judgment when using social statistics. We must always keep an open mind for alternative explanations. Society is extremely complex and there may be another seemingly unrelated variable we have yet to consider that could latently be influencing the problem we are trying to explain. This is one of the purposes of social science research; with each new study we reduce the number of potential explanations and build upon the findings of those who came before us. In short, we get closer and closer to the truth.
