So, yesterday was an interesting day. I had 4,000+ views of the original post of the Chart, Who Lies More – A Comparison (see image to the right). I also had the most comments about an individual post in the history of me blogging about data visualization.
A lot of you who posted comments are more in the political arena than the data visualization arena. I realize we are in a very divisive, polarizing election cycle for the next 100 days or so, but I was surprised by the anger and hate behind many of the comments I received. I don’t think I can physically do some of the things you told me to do.
The goal of posting the chart was to show an example of a survey conveyed as a chart. Regardless of your political leanings, this was PolitiFact’s interpretation of the data they used. However, with that said, I should have also included the methodology they used.
Below is their methodology.
For those of you who asked where PolitiFact got their data, hopefully, this will explain it better.
If there are similar data visualizations (charts, graphs, etc.) you think would benefit the dataviz community, please send them my way. I will ensure I credit you for them and would be glad to share them with my community.
Source: Bill Adair, Angie Drobnic Holan, The Principles of PolitiFact, PunditFact and the Truth-O-Meter, PolitiFact.com, Friday, November 1st, 2013, http://www.politifact.com/truth-o-meter/article/2013/nov/01/principles-politifact-punditfact-and-truth-o-meter/.
The Principles of PolitiFact, PunditFact and the Truth-O-Meter
PolitiFact staffers research statements and rate their accuracy on the Truth-O-Meter, from True to False. The most ridiculous falsehoods get the lowest rating, Pants on Fire.
PolitiFact checks claims by elected officials, candidates, leaders of political parties and political activists. We examine officials at all levels of government, from county commissioners to U.S. senators, from city council members to the president.
We also check claims by groups involved in the discourse — political parties, advocacy groups and political action committees — and examine claims in widely circulated chain emails.
PunditFact checks claims from pundits, columnists, bloggers, political analysts, the hosts and guests of talk shows, and other members of the media.
Choosing claims to check
Every day, PolitiFact and PunditFact staffers look for statements that can be checked. We comb through speeches, news stories, press releases, campaign brochures, TV ads, Facebook postings and transcripts of TV and radio interviews. Because we can’t possibly check all claims, we select the most newsworthy and significant ones.
In deciding which statements to check, we ask ourselves these questions:
- Is the statement rooted in a fact that is verifiable? We don’t check opinions, and we recognize that in the world of speechmaking and political rhetoric, there is license for hyperbole.
- Is the statement leaving a particular impression that may be misleading?
- Is the statement significant? We avoid minor “gotchas” on claims that obviously represent a slip of the tongue.
- Is the statement likely to be passed on and repeated by others?
- Would a typical person hear or read the statement and wonder: Is that true?
Transparency and on-the-record sources
PolitiFact and PunditFact rely on on-the-record interviews and publish a list of sources with every Truth-O-Meter item. When possible, the list includes links to sources that are freely available, although some sources rely on paid subscriptions. The goal is to help readers judge for themselves whether they agree with the ruling.
The goal of the Truth-O-Meter is to reflect the relative accuracy of a statement.
The meter has six ratings, in decreasing level of truthfulness:
TRUE – The statement is accurate and there’s nothing significant missing.
MOSTLY TRUE – The statement is accurate but needs clarification or additional information.
HALF TRUE – The statement is partially accurate but leaves out important details or takes things out of context.
MOSTLY FALSE – The statement contains an element of truth but ignores critical facts that would give a different impression.
FALSE – The statement is not accurate.
PANTS ON FIRE – The statement is not accurate and makes a ridiculous claim.
Principles in Truth-O-Meter rulings
Words matter – We pay close attention to the specific wording of a claim. Is it a precise statement? Does it contain mitigating words or phrases?
Context matters – We examine the claim in the full context, the comments made before and after it, the question that prompted it, and the point the person was trying to make.
Burden of proof – People who make factual claims are accountable for their words and should be able to provide evidence to back them up. We will try to verify their statements, but we believe the burden of proof is on the person making the statement.
Statements can be right and wrong – We sometimes rate compound statements that contain two or more factual assertions. In these cases, we rate the overall accuracy after looking at the individual pieces.
Timing – Our rulings are based on when a statement was made and on the information available at that time.
Process for Truth-O-Meter rulings
A writer researches the claim and writes the Truth-O-Meter article with a recommended ruling. After the article is edited, it is reviewed by a panel of at least three editors that determines the Truth-O-Meter ruling.
Corrections and review
We strive to make our work completely accurate. When we make a mistake, we correct it and note it on the original item. If the mistake is so significant that it requires us to change the ruling, we will do so.
Readers who see an error should contact the writer or editor. Their names are listed on the right side of every Truth-O-Meter item. Clicking on their names will take you to their bio pages, where you can find their email addresses.
When we find we’ve made a mistake, we correct the mistake.
- In the case of a factual error, an editor’s note will be added and labeled “CORRECTION” explaining how the article has been changed.
- In the case of clarifications or updates, an editor’s note will be added and labeled “UPDATE” explaining how the article has been changed.
- If the mistake is significant, we will reconvene the three-editor panel. If there is a new ruling, we will rewrite the item and put the correction at the top indicating how it’s been changed.
We respect that reasonable people can reach different conclusions about a claim. If you disagree with a ruling, we encourage you to email the writer or editor with your comments about our ruling. You can also post comments to our Facebook page or write a letter to the editor. We periodically publish these comments in our Mailbag feature.
PolitiFact has two other features:
- The Flip-O-Meter, which rates whether an elected official has been consistent on an issue.
- Promise meters, such as the Obameter and the GOP Pledge-O-Meter, that rate the status of elected officials’ campaign promises.