Sometimes we notice a statistical disparity between two groups. Shouldn’t these two groups be equal? Why aren’t they?
Since we are enterprising do-gooders, we make up plausible explanations to describe why there is a difference between the two groups. That way we can address the problem with a solution and make everything equal.
That is a misuse of statistics and a logical fallacy. Replace your plausible explanation with the more honest expression: Because of magic.
In a regression analysis, you take a dependent variables or group and test them with set of independent variables. You expect a certain results but find a difference in the actual results.
In statistics, a residual is the difference between an observed variable and the expected variable. This residual is unexplained by the study and could have been caused by many things. It’s an unexplained “gap.” It does not signal that the independent variables are related or demonstrate causal relationships. Residuals signal your ignorance.
Here’s the fallacy: You studied variables A, B, and C and found a residual. Afterwards, you claim variable X caused the difference.
You eliminated alternative explanations (ABC), and since variable X was omitted from the study, X must cause all remaining effects. You try to sound more clever than this, of course. You disguise the lack of substance with plausible explanations that appear substantial.
Paranormal beliefs rely on this logical fallacy to explain unexpected natural events. The residual result is a miracle caused by gods, space aliens, or spirits. Unfortunately, it’s even easier and less superficially absurd to make the same claims in social sciences.
When there is a residual, you cannot claim that an omitted independent variable caused the difference in results. You have no evidence for any claims. You cannot know what caused it.
Residuals may not be anymore meaningful than the residual scum in your bathtub. For instance, there could have been problems in your sample, like a biased or incomplete sample. This is the error term, which measures the difference between the representative sample and the unobserved total population. Another problem is observers having false expectations.
Say Group A makes 20% more than Group B. The sample only tested for income levels. What do the statistics say? It says Group A made 20% more than Group B. That’s all.
Remember, statistics do not lie to people. People lie to people.
The first thing researchers ask is, “why did group A make more than Group B?” They create plausible explanations to fill up that residual gap. These are fake explanations.
The first thing you should do is attempt to falsify your hypothesis. Why do you expect Group A and Group B to make the exact same amount of money? Why are you assuming their income levels are related? What evidence do you have to prove your explanation?
In other words, we must assume every variable is innocent until proven guilty.
We have a nasty cognitive bias though. We do not approach statistics as true objective observers. We only study certain things because we have prior assumptions about what will happen. Of course, we want to prove our assumptions and defend our hypotheses. That’s irrational.
Let’s replace our plausible explanations with the word magic. Why is there a gap? Because of magic. Does this change the substance of your argument?
Let’s study some magic!
Here’s a popular statistical residual – monetary gaps between groups. This includes differences in pay checks, bank loan rates, etc. So Group A gets more money than Group B. The differences between groups must be caused by omitted variable X.
-Because of Genetic Inferiority
-Because of Discrimination
-Because of Space Aliens
-Because of God
-Because of Magic
Pick one, anyone you want. Whatever makes you feel good! No need for evidence. It’s even better if evidence cannot falsify your hypothesis.
“Part of the wage gap results from differences in education, experience or time in the workforce. But a significant portion cannot be explained by any of those factors; it is attributable to discrimination. In other words, certain jobs pay less because they are held by women and people of color.”
“Because of discrimination” it is. The is a plausible explanation and you get to assert evil motives onto your opponents. Clearly, all males are a unitary rational actor and the hive mind decided to pay females exactly 19% less than males for the exact same work out of pure evil.
The evidence for discrimination? The lack of evidence to explain the residual pay gap. This is the logic behind “Because of discrimination”
Note they do not seek to explain why females earn 9% more than males at part-time jobs. This residual is left unexplained. Technically, this is a counterfactual which should reduce our estimated probability that the discrimination hypothesis is true. Instead, it is ignored.
Can we measure discrimination to see if this is a false hypothesis?
No, says the American Association of University Women:
“Discrimination cannot be measured directly. It is illegal, and for the most part, people do not believe that they discriminate against women or other groups. One way to discover discrimination is to eliminate other explanations for the pay gap. To uncover discrimination, regression analysis was conducted to control for the different choices women and men make.
Thus, while discrimination cannot be measured directly, it is reasonable to assume that this pay gap is the product of gender discrimination.”
Re-read that carefully. If you lack evidence, fill the gap with magic. It’s so magical, no one even knows they are using magic.
However, let’s use Bayes’ Theorem. Absence of evidence is evidence of absence.
What if we include more variables which were omitted in the first study? We have two aggregate groups – males and females. We want to test if there really is a gender wage gap at all.
So let’s include every single independent variable possible. Let’s control for education. Not all majors are equal. Engineering majors make far more money than English majors. And within the Engineering department, there are different sub-fields like Computer Science, Electrical, Mechanical, and so forth. What’s the highest degree attained? BS, MS, or PhD?
Let’s control for jobs – years of experience, specialty, government or private sector, region of employment, regional cost of living.
One interesting fact is that males are migrating. There are gender disparities across the US. Males are leaving Northeast cities like New York and Boston and moving to the south. This means that there are more young women in the northeast and more males in the south. Literally, there is a regional disparity between the two groups.
The National Science Foundation has a proper multivariate regression study of female and male engineers. In 1999, they cite various studies which show the aggregate female pay is 71-74% of males. When you compare only female engineers to male engineers in general, the gap closes to 87%. When you include the many variables listed above, such as specialty, employment sector, degree attained, regional position, years of experience, the final result is that female engineers earn $0.98 to every $1.00 a male makes.
There are many, many omitted variables that could account for the 2% residual difference. The authors of this study do not claim to know what caused the gap.
As we reduce the residual difference by factoring more independent variables, the probability of gender discrimination decreases. Gender discrimination is a null variable in engineering. It’s irrelevant in the aggregate.
Perhaps there is discrimination against individual females, but the aggregate results suggest that this is balanced by equal and opposite discrimination against individual males.
Here’s how we should handle this result. We should question our original expectation that Group A and Group B should have equal pay checks. Perhaps this group category are irrelevant to income. Males and Females may not matter to income anymore than comparing penguins and chimpanzees.
What about the explanation hypothesis about Omitted Variable X? (In this case, discrimination)
It should reduce our estimated probability of the initial hypothesis being true. Our first assumption was that discrimination explained a 20% wage residual difference. Now you see that other variables, like job specialty and years of experience, reduced the gap to 2%. There is an absence of evidence that points to an absence of significant discrimination. The probability of discrimination being a powerful variable is reduced incrementally as other variables are proven more significant.
But of course, humans don’t do that. We defend our intitial discrimination hypothesis and bizarrely claim it gets stronger as the probability is reduced. Afterall, once you eliminated the other important variables, then discrimination must fill the remaining residual gap. There’s always a residual so discrimination can never be falsified.
That is a magical explanation.
What causes wage gaps in society? I don’t know.
We are all tempted to fill in residuals with magical explanations to make ourselves feel good. This particular logical fallacy is widely used to justify political fantasies and ideologies.
Update: Just as a point of clarification. Discrimination can be directly measured, contrary to the bleifs of the AAUW and NOW. One example of evidence are laws that would forbid Group Y from getting paid the same as Group X, or forbid Group Z from voting. If discrimination is illegal, another ways would be to collect a pattern of legal violations that, in the aggregate, cause discernable effects. Both ways avoid statistical fallacies.