How Fact-Checking is Flawed
My inbox often includes a “PolitiFact Truth Rundown.” I have been critical of the fact-checking industry as unsound, but still look at their offerings. A recent one on Dan Crenshaw, a member of Congress from Texas, illustrates a major flaw in fact-checking that causes a great deal of distrust.
Crenshaw lost his eye serving as a Navy SEAL in Afghanistan. He received a great deal of attention for his sense of humor in an appearance on Saturday Night Live to receive an apology from Pete Davidson after a disrespectful joke. More recently he appeared on The View and made this statement about asylum seekers coming across the Mexican border:
“As it turns out, about 80 to 90 percent of those don’t have a valid asylum claim, once we actually get their documentation.” PolitiFact rated this statement to be False.
The interviewer immediately interjected, “That fact, what you are saying, actually has been debunked.” Crenshaw responded, “It has not been debunked. I’m sorry, no it has not. I deal with the Department of Homeland Security and these are the numbers that are coming out… Explain why those are false then. You’re just saying they’re false, you’re not providing evidence they are.”
Why does PolitiFact endorse The View and dispute Crenshaw? The PolitiFact report admits that he was citing the figures from Homeland Security accurately:
“The truth is, about 20 to 30 percent of asylum requests have been granted annually since 2009” and “asylum was granted in 16% of cases that originated from a credible fear claim.”
But here is the catch: “Experts said that does not mean that the remaining 70 to 80 percent of cases are invalid.”
According to PolitiFact’s experts—an analyst at the Migration Policy Institute and two law professors—the remaining claims were not necessarily invalid, but could have been dismissed for other reasons even though there was truth to them. So “we rate this claim False.”
In other words, Crenshaw’s experts were the administrators and judges who rejected the asylum requests. His definition of “valid” is what an immigration judge said passed the standard. PolitiFact's experts offer a different definition of “valid” and think many of the other cases might have merit as well.
Who is right? Whom do you trust? More importantly, what standard should PolitiFact apply?
One of the core flaws of fact-checking is that the answer depends entirely on which frame they choose. They could ask two distinct questions:
- Can this statement be considered to be true?
- Can this statement be considered to be false?
If they asked the first question about Crenshaw’s statement, it clearly could be considered to be true. He is relying on government data and the government’s definition of validity: a judge accepts the claim “once we actually get their documentation.”
If fact-checkers chose the second question, Crenshaw’s statement can be challenged by experts who prefer a different interpretation of validity and therefore question the government’s data.
Edited by Thoth101, 11 July 2020 - 03:16 AM.