Readers:
So, yesterday was an interesting day. I had 4,000+ views of the original post of the Chart, Who Lies More – A Comparison (see image to the right). I also had the most comments about an individual post in the history of me blogging about data visualization.
A lot of you who posted comments are more in the political arena than the data visualization arena. I realize we are in a very divisive, polarizing election cycle for the next 100 days or so, but I was surprised by the anger and hate behind many of the comments I received. I don’t think I can physically do some of the things you told me to do.
The goal of posting the chart was to show an example of a survey conveyed as a chart. Regardless of your political leanings, this was PolitiFact’s interpretation of the data they used. However, with that said, I should have also included the methodology they used.
Below is their methodology.
For those of you who asked where PolitiFact got their data, hopefully, this will explain it better.
If there are similar data visualizations (charts, graphs, etc.) you think would benefit the dataviz community, please send them my way. I will ensure I credit you for them and would be glad to share them with my community.
Best Regards,
Michael
Source: Bill Adair, Angie Drobnic Holan, The Principles of PolitiFact, PunditFact and the Truth-O-Meter, PolitiFact.com, Friday, November 1st, 2013, http://www.politifact.com/truth-o-meter/article/2013/nov/01/principles-politifact-punditfact-and-truth-o-meter/.
The Principles of PolitiFact, PunditFact and the Truth-O-Meter
PolitiFact staffers research statements and rate their accuracy on the Truth-O-Meter, from True to False. The most ridiculous falsehoods get the lowest rating, Pants on Fire.
PolitiFact checks claims by elected officials, candidates, leaders of political parties and political activists. We examine officials at all levels of government, from county commissioners to U.S. senators, from city council members to the president.
We also check claims by groups involved in the discourse — political parties, advocacy groups and political action committees — and examine claims in widely circulated chain emails.
PunditFact checks claims from pundits, columnists, bloggers, political analysts, the hosts and guests of talk shows, and other members of the media.
Choosing claims to check
Every day, PolitiFact and PunditFact staffers look for statements that can be checked. We comb through speeches, news stories, press releases, campaign brochures, TV ads, Facebook postings and transcripts of TV and radio interviews. Because we can’t possibly check all claims, we select the most newsworthy and significant ones.
In deciding which statements to check, we ask ourselves these questions:
- Is the statement rooted in a fact that is verifiable? We don’t check opinions, and we recognize that in the world of speechmaking and political rhetoric, there is license for hyperbole.
- Is the statement leaving a particular impression that may be misleading?
- Is the statement significant? We avoid minor “gotchas” on claims that obviously represent a slip of the tongue.
- Is the statement likely to be passed on and repeated by others?
- Would a typical person hear or read the statement and wonder: Is that true?
Transparency and on-the-record sources
PolitiFact and PunditFact rely on on-the-record interviews and publish a list of sources with every Truth-O-Meter item. When possible, the list includes links to sources that are freely available, although some sources rely on paid subscriptions. The goal is to help readers judge for themselves whether they agree with the ruling.
Truth-O-Meter rulings
The goal of the Truth-O-Meter is to reflect the relative accuracy of a statement.
The meter has six ratings, in decreasing level of truthfulness:
TRUE – The statement is accurate and there’s nothing significant missing.
MOSTLY TRUE – The statement is accurate but needs clarification or additional information.
HALF TRUE – The statement is partially accurate but leaves out important details or takes things out of context.
MOSTLY FALSE – The statement contains an element of truth but ignores critical facts that would give a different impression.
FALSE – The statement is not accurate.
PANTS ON FIRE – The statement is not accurate and makes a ridiculous claim.
Principles in Truth-O-Meter rulings
Words matter – We pay close attention to the specific wording of a claim. Is it a precise statement? Does it contain mitigating words or phrases?
Context matters – We examine the claim in the full context, the comments made before and after it, the question that prompted it, and the point the person was trying to make.
Burden of proof – People who make factual claims are accountable for their words and should be able to provide evidence to back them up. We will try to verify their statements, but we believe the burden of proof is on the person making the statement.
Statements can be right and wrong – We sometimes rate compound statements that contain two or more factual assertions. In these cases, we rate the overall accuracy after looking at the individual pieces.
Timing – Our rulings are based on when a statement was made and on the information available at that time.
Process for Truth-O-Meter rulings
A writer researches the claim and writes the Truth-O-Meter article with a recommended ruling. After the article is edited, it is reviewed by a panel of at least three editors that determines the Truth-O-Meter ruling.
Corrections and review
We strive to make our work completely accurate. When we make a mistake, we correct it and note it on the original item. If the mistake is so significant that it requires us to change the ruling, we will do so.
Readers who see an error should contact the writer or editor. Their names are listed on the right side of every Truth-O-Meter item. Clicking on their names will take you to their bio pages, where you can find their email addresses.
When we find we’ve made a mistake, we correct the mistake.
- In the case of a factual error, an editor’s note will be added and labeled “CORRECTION” explaining how the article has been changed.
- In the case of clarifications or updates, an editor’s note will be added and labeled “UPDATE” explaining how the article has been changed.
- If the mistake is significant, we will reconvene the three-editor panel. If there is a new ruling, we will rewrite the item and put the correction at the top indicating how it’s been changed.
We respect that reasonable people can reach different conclusions about a claim. If you disagree with a ruling, we encourage you to email the writer or editor with your comments about our ruling. You can also post comments to our Facebook page or write a letter to the editor. We periodically publish these comments in our Mailbag feature.
PolitiFact has two other features:
- The Flip-O-Meter, which rates whether an elected official has been consistent on an issue.
- Promise meters, such as the Obameter and the GOP Pledge-O-Meter, that rate the status of elected officials’ campaign promises.
Michael Sandberg's Data Visualization Blog 
Great post Michael, shared with summer school students who are designing their own infographics, some based on politics. A few of the kids would have like a photo portrait of each candidate next to their name as, being young, they didn’t know who all of them were. We had one great discussion! ________________________________
Thanks, but I cannot take credit for the chart. It was created by Robert Mann, who has some relationship with Politifact. I am very please to hear you used it as a discussion topic in you class.
Yes, I like your idea of having a small headshot image next to the candidate’s name. We need to take into account that 1 out of 7 voters were not alive when Bill Clinton was elected president, so a visual image of that person may offer a deeper connection to turning the data into knowledge and memories for them.
Thanks again for visiting and I appreciate your feedback.
No data on Jill Stein or Gary Johnson?
Good point. They should have included the other parties too to be fair.
I think the chart is awesome, but when I show it to them they ask me how current the data is. On which date was the snapshot of the from politifact obtained? This would be a great piece of info to add to the bottom of the image.
Hi Frank: I agree with you. They should always give you some kind of indication in a chart or visualization of what timeframe the report was for. We only know it is relatively current in that they have Secretary Clinton and Mr. Trump in the survey. Thanks for your thoughts. Michael
Looks like Sanders should be last on the list with 0% Pants on Fire?
Thank you, how come Geirge Bush is not there?
A valid question that’s been raised is “How were the 50 statements chosen for each person?” The chart could easily be biased by selecting specifically rated quotes for some people and quotes rated better for others.
I dont think they picked 50 statements and used those one a curve. They used all the ratings for people with at least 50 items. This is something you can verify if you visit the site and compare the overall percentages available there.
Whenever Politifact themselves post comparisons, they are always careful to say that their methodology is not designed to look at the aggregate “honesty” of a person. Their goal is to test the validity of individual statements of importance to the public, not to judge everything a person says. Nor do they force themselves to be balanced. If Gary Johnson gets very little coverage and no one pays attention to what he’s saying, there’s not really a demand for fact checks. It doesn’t matter if someone’s lying if no one hears them. And sure enough, they’ve only checked him 10 times, only twice since the 2012 elections.
That means they aren’t running an aggregate of every factual statement coming from every politician, and a graph like is limited in utility. All it tells you is the division of the ratings for the statements they rated, and they would be the first to tell you that extrapolating to general trustworthiness is difficult. They aren’t an arbiter of trust, nor do they pretend to be. They check individual factual statements for validity, and that’s it. Whoever made this image (and it wasn’t Politifact; it’s not on their site) did a bad job with the captions, since it’s easily misunderstood, and claims to impart more information than it actually does. Obviously Politifact is sometimes wrong, but they are open about their limitations.
Hello. This is a fascinating chart, but I was suspicious of it when I first saw it. Why would it choose statements from the candidates since 2007? Clinton has now been a candidate twice (then Sec. of State in between). Then there’s Palin. Why include her? Basically, what is this? Why choose the people that appear here? They haven’t all had the same number of statements to sample from. Isn’t that problematic? Strangest of all is that it is attributed to Robert Mann. I searched last week for Robert Mann and politifact, and all I could come up with was an LSU political scientist who has written for POlitifact. I emailed him, asking him if he was the mysterious Robert Mann who had visualized this data. He responded that he was NOT the person. Where did this come from? Nothing like it appears on Politifact’s site. Ths would require us to go back through every time politifact has checked one of these people since 2007, and then try to sort through it, if we were to fact-check this elusive Robert Mann’s infograph. Have you heard anything about Mann? The place that I first saw this was on IMGUR.com, which is simply a site that posts the most viral memes of the day. That is, this could simply be political propaganda. Please let me know if you have any facts that will answer some of my questions. sincerely, Prof. Jayson Harsin, The American University of Paris
P.s. the closest thing I could find to an original for the Robert Mann politifact infographic is this following op-ed in the NY Times by Politifact’s Anglie Holan from last December. Some of the numbers correspond to the Mann infographic, It also says it begins to look at this selection of candidates since 2007: http://www.nytimes.com/2015/12/13/opinion/campaign-stops/all-politicians-lie-some-lie-more-than-others.html?_r=0
While your methodology ticks all the boxes making it seem like it is well done, it still is misleading. It would not pass muster in unbiased scientific journals and therefore would not be published.
I don’t have any inherent issues with the chart because I believe it is based on legitimate information and I am sure their methodology is as reliable as you can get on a subject such as this. I guess the only reason I find the entire concept irrelevant is that the lias themself carry no weight. If he said he had a tuna sandwich for lunch and they go fact check that he actually had a steak (they actually did fact check a food item once on Trump having something to do with a restaurant in his building) that is for the most part irrelevant…yet likely included in the statistics. On the other hand if someone were to tell a lie about activity that occurred while they were a public servant to a panel of congressman…while under oath…while running for another position as a civil servant,,,that would to me at least seem to carry quite a bit more weight. I don’t believe the topics and equally as important, the venue of the lies, were factored into the data. It would seem as though if you want Hillary to tell the truth, the whole truth and nothing but the truth…you better be carrying an FBI badge.
I don’t have any inherent issues with the chart because I believe it is based on legitimate information and I am sure their methodology is as reliable as you can get on a subject such as this. I guess the only reason I find the entire concept irrelevant is that the lias themself carry no weight. If he said he had a tuna sandwich for lunch and they go fact check that he actually had a steak (they actually did fact check a food item once on Trump having something to do with a restaurant in his building) that is for the most part irrelevant…yet likely included in the statistics. On the other hand if someone were to tell a lie about activity that occurred while they were a public servant to a panel of congressman…while under oath…while running for another position as a civil servant,,,that would to me at least seem to carry quite a bit more weight. I don’t believe the topics and equally as important, the venue of the lies, were factored into the data. It would seem as though if you want Hillary to tell the truth, the whole truth and nothing but the truth…you better be carrying an FBI badge.
(sorry if this posts my reply twice, the site feels like its locked up)