A Closer Look at the Dan Jones and BYU Polls, or TL;DR: Why You Can’t Always Trust Pollsters

A Closer Look at the Dan Jones and BYU Polls, or TL;DR: Why You Can't Always Trust Pollsters

(Authors Note: While I am an unpaid intern on the Stormont Campaign, this article is entirely independent of that, and does not reflect the candidate or campaign’s views.)

It’s “Hey! Look at our Polling Data Week.” It seems like every whosit has a poll prognosticating how the election will turn out. I’m going to look at just two of them, both released on Monday October 27th:Utah Policy Daily (Dan Jones) and BYU’s Center for the Study of Elections and Democracy (BYU-CSED). Both have problems, but the Dan Jones poll is especially problematic, for several reasons.

Neither surprised much, except for the two races that save the election cycle from being a complete snoozer (and were the only ones we were watching anyway):

These two races are interesting either because of their story lines. Love lost oh so barely in 2012. In the AG race, the ethical issues of Reyes’ predecessors in the Attorney General’s office.

First,  UtahPolicy.com dropped the bomb that the Attorney General’s race was all but over. With 35 points between Reyes and Stormont he would glide to a quick and decisive victory.

Sure, it’s a tough race, I thought, but that bad? So I dug into the poll. Then, a little later Monday, BYU released a poll that put a difference of only 20 points between Reyes and Stormont. Again, though, Reyes was the winner. But why a 15 point swing. This same poll from BYU had put Doug Owens in the lead over Mia Love. The face of “New” republicans.

The more I dug, though, the more I found several irredeemable problems in the Dan Jones/UPD Poll (UPD).

First: Methodology

The UPD poll used “active voters” instead of “likely” or “registered” voters. If you’re confused about what an “active” voter is, so is the rest of the polling and political world. It’s not a standard measure.  Many voters who will and have voted do not closely follow the election. “Likely Voters” is a difficult thing to model, but how “closely” a voter is following the election should not be the sole criterion. A “likely voter,” however, is based on how often a voter shows up at the polls, not on a self-reporting measure of whether the voter is paying close attention to the race. And what voter who is answering a poll is going to say they are not paying attention to the race? They’re already answering questions–to say they don’t know what they are talking about would be embarrassing.

Second: Track Record

Dan Jones/UPD has a poor track record modeling the Utah electorate. Nate Silver’s FiveThirtyEight.com has given them a “C” rating, in part because of their issues in this regard.

Dan Jones and Associates poll rating from Nate Silver's fivethirtyeight.com
Dan Jones and Associates poll rating from Nate Silver’s fivethirtyeight.com

This includes practice of only calling people with land lines and not calling cell  phones. Because it is against federal law to auto dial cell phones respondents are more likely to be older, must have a land line (who has a land line anymore), and reside older areas of Utah. This will automatically result in a skew that is irrecoverable.

**Update** According to Utah Policy‘s managing editor Bryan Schott  they did in fact call Land Lines, Mobile Phones and used Online Surveys as part of this poll. However UPD continues to not release information regarding their polls including the Cross tabs so my original post went with the information I had access to. 

Third: Turnout

Turnout will not be evenly distributed among Utah’s four congressional districts. Because multiple polling sources indicate that the Fourth Congressional District race is within the margin of error, more resources are being devoted towards boosting turnout from all sides in that district. Moreover, the remaining three congressional races are not going to be close. So, Luz, Brian, and Donna: short of an election night upset and something going crazy, there is no path to victory through these numbers. However, John – your write in campaign might give you a chance since we don’t have a poll yet.

Democratic campaigns and aligned third parties will not dedicate resources to vote turn-out operations in the First, Second or Third districts nearly to the extent (if at all) that we will see them in the Fourth.

Fourth: Third Party Candidates

While the Libertarian candidate has received approximately 5% in prior races for statewide office in 2010 and 2012, other also-ran third party candidates get less than 1% each, not 2% each as Dan Jones polled. This just hasn’t been a watershed year for third party candidates and there’s nothing happening to indicate that it will be.

Combine an expectation for typical turn-out for third-party candidates, Dan Jones  poor track record ( at this point in 2012, he had Mia Love +6% over Matheson and the Tribune had Love at +12%), and the inaccurate distribution of voters from each congressional district, I have little confidence in the poll.

BYU-CSED’s Poll

That brings us to the BYU Center for the Study of Elections and Democracy (BYU-CSED) Poll. Who has run exit polling data for the 2004, 2006, 2008, 2010, and 2012 cycles by choosing representative precincts from around the state for various races including statewide. This exit polling has been extremely accurate in predicting the final results.

BYU-CSED’s latest poll reached out to participants in their exit polling over the last five cycles to complete an entirely online survey. 6,952 were contacted, and 812 of those participated and 733 completed the entire survey. To account for self-selection bias, BYU’s pollsters (who are two of the three partners in Mia Love’s internal polling firm along with Love’s campaign Manager Dave Hansen) “statistically adjusted the results using a technique called ‘rim weighting’ to correct for potential non-response and coverage error[.]”

**Update** I have been corrected in that while Quinn Monson and Kelly Patterson are not currently working for the BYU-CSED since their firm Y2 analytics is currently running the internal polling for Mia Love’s campaign, they are using the methodology developed by Monson and Patterson at BYU-CSED.

In short, their methodology has worked before and has been fairly accurate.

The Democratic improvements on the congressional race over the Dan Jones poll can be largely attributed to overly small sample sizes  rather than a skewed makeup of respondents. BYU’s distribution of these congressional races better approximates actual turnout for the state than the even 200 respondents per a congressional district as the Dan Jones poll does. In this case, the potential for a self-selection bias works in favor of the BYU-CSED poll.

Finally, it is salient to note that both polls find that  13%-18% of voters remain undecided with under a week to Election Day 2014.

So the bigger story is: Could Mia Love actually lose this again? In short, yes. While these two polls are the most public internal polling on other campaigns and groups are showing oddly similar results. Mia Love, who had this in the bag since Jim Matheson announced his intent to resign, could end up on the wrong side of history and end up waiting for 2016 to come around. If Republicans give her a third chance.

This just goes to show that polling in a state like Utah isn’t as cut and dry as it might seem and that election night might still have some shocking results.

Liked it? Take a second to support Utah.Politico.Hub on Patreon!

Related posts