Correcting The Baltimore Sun’s Faulty Polling Analysis
This is my site Written by admin on August 28, 2011 – 7:59 pm

The Baltimore Sun conducted a poll surrounding the upcoming Mayoral primary election in Baltimore, and wrote about it on Saturday, August 27th.

While it is disappointing that the Sun produced this faulty and misleading polling and analysis, we will assume that everyone involved had the best intentions.

Regardless, these flaws are clear:

  1. A telephone poll of “likely voters” disproportionately reaches senior citizens. Senior citizens are the most likely to appear in “likely voter” databases, have a land-line phone number, and have a stable and identifiable voting record at a Baltimore city address. As their own article points out, the incumbent mayor’s strongest voting bloc is senior citizens, and Otis Rolley’s supporters are predominantly under 35. People under 35 are the least likely to have land lines and be reachable for a poll.
  2. There was no discussion at all about how turnout may affect the outcome – especially turnout among “new” or younger voters. The article seems to conclude that the people polled will be the only people doing the voting. There is no reason to think this will be the case. Both low and high turnout scenarios could materially affect the outcome, disproportionately emphasizing particular voting blocks in each case.
  3. The tone of the analysis is sweeping and distorted in favor of the incumbent. In fact, the tone is so forceful that it may actually distort the election by lowering turnout. Using phrases like “sweeping competition” and “more than all of her rivals combined” suggests an air of inevitability. However, the article itself goes on to say that a “challenger could still win it,” and states that 18% of those polled are undecided. More balanced analysis would have emphasized the number of undecided voters, and would have spent more time discussing scenarios where challengers could succeed.
  4. There was no attempt made to report how many new voters there might be. New voters could make up a significant percentage of the vote. Any valid attempt to prognosticate about the outcome of the election must factor in new voters.
  5. The reporter did not seek comment from the challenging campaigns. Subsequent comments made by both the Pugh and Rolley camps indicate that they are seeing wildly different measurements in their own field work. Both campaigns have spent money on polls, and have reported numbers as high as 50% undecided (Sun says 18%), and job approval ratings as much as 25% lower than what the Sun reported. The challengers should have had a chance to comment, and report their own poll findings. The Sun should have identified, and tried to explain, the disparities between their poll results and the numbers reported by the campaigns.
  6. There was no mention made about the cost of the poll, or the specific methodology employed. Primary elections are notoriously difficult to poll correctly. Fast, inexpensive polls are likely to be very noisy and inaccurate. The campaigns know this and have likely spent considerably more than the Sun did to collect their information. The Sun should have made a statement about the cost and quality of their poll, the historic track records of similar polls in comparable, small races done by the same firm, and how this poll compared to polling which may have been done by the campaigns.

It was possible for the Sun to write an article about this poll that was fair and balanced, but they chose not to do so. Why?

UPDATE: The Sun published a subsequent op-ed which addresses some, but not all, of our concerns.


6 Responses »

  1. “There was no mention made about … the specific methodology employed.”

    Except for, you know, this giant paragraph at the end of the article outlining the methodology:

    The Baltimore Sun commissioned a telephone survey of 742 likely Baltimore Democratic primary voters from Aug. 22-24. The Sun’s pollster, OpinionWorks of Annapolis, used the Baltimore City Board of Elections database to identify registered voters with a history of voting in municipal primary elections and gathered survey results from those who ranked their likelihood of voting in the September primary “50-50″ or higher. The Sun’s sample was designed to approximate the racial, gender, geographic, and age breakdown of the city’s Democratic primary voting population, based on turnout patterns averaged over the past three primary elections. The margin of error for questions that reflect the entire sample is 3.6 percentage points, which means that 95 times out of 100, the actual answer obtained by surveying every Baltimore City Democratic primary voter would be within 3.6 percentage points of the answer obtained by using the sample.

  2. There is some question about whether that ‘explainer’ was added after the article was originally published; we are inquiring about that. Regardless, it does not adequately address the specific concerns we raised.

  3. Valid points all, but they presume an infrastructure that The Sun simply does not have anymore. The staff has been cut tremendously. The skeleton crew that is left does not have the resources to be nearly as thorough as Baltileaks would like them to be.

    If a bias exists at The Sun, it is not one towards the incumbent. It’s a bias toward journalism produced as cheaply as possible. And that is not going away anytime soon.

    When I read Baltileaks, I get a pretty strong whiff of a bias toward Rolley. But that is of course hard to judge because the site is utterly opaque WRT who produces it. You can complain about process issues at The Sun, but at least the content creators at The Sun sign their names to their work.

  4. The point we are claiming here is that an incomplete story, presumably produced as cheaply as possible – as you suggest, became a biased one that might actually serve to dampen turnout, which should not be a goal of a newspaper. We raise these issues in order to correct that error.

    There is an established tradition of anonymous journalism and pamphleteering, and we offer our comments in that vein. The lack of a truly unfettered news and analysis organ in Baltimore is arguably one reason for the city’s many woes. We aim to correct that, and will speak freely when we are so moved. Those who take issue with our statements or publications are welcome to write counterarguments, independently or right here on our site, as you have done.

    Baltileaks is the work of multiple concerned citizens and we are not affiliated with any one political campaign. We are, however, strongly pro-truth, anti-corruption and anti-stupidity, and in most cases, that means we will be strongly anti-incumbent. We believe that the incumbent power structure is primarily to blame for the city’s problems, which are well-documented and legion. With this as our stated bias, you should be able to successfully interpret our work in its proper context. We encourage everyone to read and share information from multiple sources, as that is the only truly effective way to combat the biases inherent in all media.

  5. But I think that Dave’s point is, in part, that your website and tweets, which were originally marketed as seeking transparency in local government and promoting information of public value in Baltimore, have simply pushed a different agenda than the one you perceive in the mainstream media — namely, your support of Rolley and criticism of SRB. This website has simply become a group of anonomous Rolley supporters, who also chime in about the ridiculousness of the Conaways when the Conaways bring it upon themselves. There is been little else of public concern addressed.

    For example, note the complete dearth of information or discussion of any of the city council races or the examination of donors in those races where significant donations have a far greater impact.

    I think that there is plenty of value in criticizing the mayor, I do not think that she is innovative or the kind of person that we need to move the city forward, and also plenty of value in supporting Otis Rolley (though his crime and education “plans” were laughably bad and raise serious red flags) but it’s time to stop marking baltileaks as an altruistic endevor or change the tone…pushing a different agenda anonomously is hardly promoting transparency…it’s perpetuating the problem…just my two cents.

    Editor’s Note: This comment was published from an IP address associated with an anonymous proxy service “Hide My Ass” ( – which is fine, and in the spirit of this site. The errors in spelling are unedited and are the author’s.

  6. As mentioned before, we will publish anything we receive that would be of value to a Baltimore voter. We’ve disseminated information about Carl Stokes, Rikki Spector, Belinda Conaway, Jack Young, and many others, as well as copious information about the actions of the Board of Estimates. We’re sorry if you’re not paying attention. We have not conducted donor analysis in those races due to lack of resources to do so. Are you volunteering? Send us information; we will distribute it.

    Each of the candidates can be debated on their merits, but we have not advanced any agenda that favors one more than any other. And if you think so, you’re welcome to start your own website and build your own audience. The cure is always more speech, and not different or less speech.

    We stand by our mission of government transparency and will make every effort to publish any and all information that we receive in service of that goal, no matter who it targets or what agenda it may seem to promote, or not to promote.

Leave a Comment