In non experimental social science, the subject cannot be studied under the holy grail of scientific experimentation: complete control over the conditions and completely random selection of test and control subjects. Very little of the quasi experimental technique and protocols can be used, either. The only approach left is to use the best substitutes for accuracy, control and comparison that are available.
As an example, if a major disaster destroys the economy of a city, as with the impact of Hurricane Katrina on New Orleans, and the surviving original residents have made a permanent diaspora from the area, then it is difficult to survey their evidence of the pre-existing economic conditions with those of the newcomers who have settled into the area. The original residents had generational businesses that were built and maintained on historical and cultural bases that existed for centuries. The newcomers arrived with better opportunity, but with no customer base and no track record of success that applied to the New Orleans community. As a result, many logical fallacies can apply.
The first is the “apples to oranges” pitfall, where faulty comparisons are more than likely to be made. If the population of newcomers is predominantly White, then comparing their economic conditions to those that were experienced by the predominantly Black community will lead to inaccurate results. If the newcomers have never lived in New Orleans, then comparing their economic conditions with long term residents will be flawed because of the completely different backgrounds and knowledge bases of the two groups.
When there is a complicated program that has many components and the participants answer survey questions that give first person detail about successes and failures of the program, there are enormous pitfalls that are based in logical fallacy!
The first logical fallacy challenge in the survey method of non experiment is in constructing the actual survey questions. The entire survey can be biased with questions that are limited in scope and which do not offer the opportunity to add comments or statements. The sample can be biased by being too small, by including staff and others who would benefit financially from the program, and by eliminating critical participants from the sample.
The second logical fallacy challenge lies in establishing quantitative values. Where would the numeric line be identified and drawn for absolute success or failure in a complex program that has several components, some of which might be highly successful and others which may have failed? Faulty quantification is the greatest challenge to the legitimacy of a social non experiment.
The third logical fallacy lies in the construction of hypothesis, cause and effect, and prediction that is based on anecdotal, unverifiable, and non quantifiable evidence. This is the pitfall of using subjective means to make evaluations, to categorize observations, and to identify phenomena.
Many logical fallacies are the result of either linguistic poverty or linguistic savvy. A challenge to an existing definition, accompanied by a new definition can become so convoluted and eventually inarticulate that the attempt to improve the language becomes a distraction and a failure. The new definition must be communicable, correct, well explained, and verifiable in its accuracy as a defining entity. In other areas, overuse of language can conceal just about anything, especially when we consider that complicated and obscure terminology can conceal more than it reveals. In other ways, poor organization and presentation can obscure some very solid material, as embellishment, rambling, digressions, and failure to write in clear, straight forward language distracts a reader to the point of abandoning attempts to understand the material.
In using excessive language, there also may be a pressure to commit logical fallacies in the form of focusing on the use of such language, rather than providing the best content. This constitutes a form of appeal to authority.
The final logical fallacy lies in the definition of terms, observations, categories and all other aspects of the experiment. When there is unclear understanding of the concepts and terminology, or when there is disagreement, flaw, or bias in any aspect of the experiment, then the legitimacy of the experiment can be challenged on a variety of levels and in a variety of ways, including using fallacy laden argument to attack the experiment and its claimed results.
The most powerful tools for the sociologist who is forced to use the non experimental method lie in a comprehensive understanding of the logical fallacies, or errors in cognitive thinking, and how they can be avoided to prevent flawed design, execution, interpretation and reporting of results.
A Good Resource For Understanding Logical Fallacies
