Jump to Any Race
National Politics|By Charlie Cook, December 30, 2016

This story was originally published on nationaljournal.com on December 27, 2016

The Novem­ber elec­tions pit­ted Demo­crats against Re­pub­lic­ans, con­ser­vat­ives against lib­er­als, Trump-style pop­u­lists and tea parti­ers against the es­tab­lish­ment and con­ven­tion­al politi­cians. An­oth­er con­test, fol­lowed mainly by polit­ic­al afi­cion­ados, matched tra­di­tion­al poll­sters against newly fash­ion­able ana­lyt­ics wiz­ards, some of whom—pre­ten­tiously in my opin­ion—called them­selves “data sci­ent­ists.”

It was well known that tra­di­tion­al polling was hav­ing prob­lems. The numb­ing ef­fect of bil­lions of tele­market­ing calls and the ad­vent of caller ID and voice mail had re­duced re­sponse rates (the per­cent­age of com­pleted in­ter­views for every hun­dred at­tempts) from the 40s a couple of dec­ades ago to the high single di­gits. As they struggled to get truly rep­res­ent­at­ive samples, poll­sters “weighted” their data more than ever be­fore, mak­ing as­sump­tions of what the elect­or­ate would look like on elec­tion days that were weeks, months, or even a year or more away.

Prob­lems with tra­di­tion­al, live-tele­phone polling led to ex­per­i­ment­a­tion and more re­cently a grow­ing ac­cept­ance of new meth­ods like In­ter­act­ive Voice Re­sponse, pop­ularly known as robo-polls, and on-line polling. Each new meth­od brings both good and bad at­trib­utes. As a tra­di­tion­al­ist, I see the new tech­niques as bad ideas whose time is re­gret­tably com­ing.

The oth­er trend is “ana­lyt­ics,” which in­cor­por­ates in­form­a­tion from a vari­ety of sources—Census Bur­eau stud­ies, com­mer­cially avail­able mar­ket data com­bined with past elec­tion res­ults, and con­clu­sions gleaned from polling, voter can­vassing, and eco­nom­ic meas­ures such as the un­em­ploy­ment rate. This “big data” en­able cam­paigns to mod­el the an­ti­cip­ated elect­or­ate, identi­fy voters most likely to be sym­path­et­ic to their can­did­ates, and shape their mes­sages ac­cord­ingly.

The roots of cam­paign ana­lyt­ics go back to the 1970’s when Demo­crat­ic cam­paign con­sult­ant Matt Reese and Re­pub­lic­an con­sult­ant Ed­die Mahe pro­moted a new tech­no­logy branded Clar­itas, a geo-demo­graph­ic tar­get­ing sys­tem centered on life­styles and neigh­bor­hoods based on a mar­ket-seg­ment­a­tion plat­form de­veloped by com­puter sci­ent­ist Jonath­an Rob­bin (Clar­itas is now owned by Nielsen). It was an idea ahead of its time, too ex­pens­ive for most cam­paigns, and it even­tu­ally left the polit­ic­al theat­er al­to­geth­er.

In 2004 the Howard Dean, George W. Bush-Dick Cheney, and John Kerry-John Ed­wards pres­id­en­tial cam­paigns ad­vanced the uses of data to con­tact voters, but it was the 2008 cam­paign of Barack Obama that took ana­lyt­ics to a whole new level. The in­fatu­ation with ana­lyt­ics after Obama’s reelec­tion in 2012 promp­ted some of his op­er­at­ives to say they didn’t need tra­di­tion­al polling any­more.

When Hil­lary Clin­ton began put­ting to­geth­er her 2016 cam­paign, she brought on board many Obama vet­er­ans, go­ing all in for the new tech­no­logy. Don­ald Trump’s gen­er­al-elec­tion cam­paign also em­ployed ana­lyt­ics, though how soph­ist­ic­ated and im­port­ant it was in his vic­tory is a mat­ter of con­sid­er­able de­bate. House and Sen­ate cam­paign com­mit­tees and su­per-PACs also used ana­lyt­ics to vary­ing de­grees.

The re­li­ance, or per­haps over­re­li­ance on ana­lyt­ics, may be one of the factors con­trib­ut­ing to Clin­ton’s sur­prise de­feat. The Clin­ton team was so con­fid­ent in its ana­lyt­ic­al mod­els that it op­ted not to con­duct track­ing polls in a num­ber of states dur­ing the last month of the cam­paign. As a con­sequence, de­teri­or­at­ing sup­port in states such as Michigan and Wis­con­sin fell be­low the radar screen, slip­page that that tra­di­tion­al track­ing polls would have cer­tainly caught.

Ac­cord­ing to Kantar Me­dia/CMAG data, the Clin­ton cam­paign did not go on the air with tele­vi­sion ads in Wis­con­sin un­til the weeks of Oct. 25 and Nov. 1, spend­ing in the end just $2.6 mil­lion. Su­per PACs back­ing Clin­ton didn’t air ads in Wis­con­sin un­til the last week of the cam­paign. In Michigan, aside from a tiny $16,000 buy by the cam­paign and a party com­mit­tee the week of Oct. 25, the Clin­ton cam­paign and its al­lied groups didn’t con­duct a con­cer­ted ad­vert­ising ef­fort un­til a week be­fore the elec­tion.

In fact, the Clin­ton cam­paign spent more money on tele­vi­sion ad­vert­ising in Ari­zona, Geor­gia, and the Omaha, Neb­raska mar­kets than in Michigan and Wis­con­sin com­bined. It was Michigan and Wis­con­sin, along with Pennsylvania (the Clin­ton cam­paign and al­lied groups did spend $42 mil­lion on tele­vi­sion in the Key­stone State), that ef­fect­ively cost Demo­crats the pres­id­ency.

In the end, the na­tion­al polls fared bet­ter than com­monly thought. The Real­Clear­Polit­ics av­er­age of na­tion­al polls showed Clin­ton ahead by 3.2 per­cent­age points go­ing in­to Elec­tion Day, and the fi­nal ABC News/Wash­ing­ton Post, CBS News, NBC News/Wall Street Journ­al, and Fox News polls each had Clin­ton ahead by 4 points (the last CNN na­tion­al poll was taken two weeks be­fore the elec­tion and had Clin­ton ahead by 5 points). She ended up win­ning the na­tion­al pop­u­lar vote by 2.1 per­cent­age points, 48.2 to 46.1. Thus the RCP av­er­age was off by 1.1 per­cent­age points, the net­work polls were off by 1.9 per­cent­age points. They were off by far more in 2012, but nobody no­ticed be­cause the pop­u­lar vote and Elect­or­al Col­lege tally went the same dir­ec­tion. If one buys the ar­gu­ment that the race changed con­sid­er­ably in the last week, for whatever reas­on, then some of these polls may not have been off by much if at all.

Like so many oth­er as­pects of this elec­tion, a lot of small misses ad­ded up to one gi­ant er­ror on the out­come of the elec­tion. In 54 out of our 58 pres­id­en­tial elec­tions, the win­ner of the pop­u­lar vote also pre­vailed in the elect­or­al vote. A good rule of thumb is that if a can­did­ate wins the pop­u­lar vote by at least 2 per­cent­age points, he or she will al­most cer­tainly cap­ture the Elect­or­al Col­lege. So in an elec­tion when one can­did­ate is thought to have a com­fort­able lead of more than 2 per­cent­age points, there is a reas­on­able ex­pect­a­tion that the elect­or­al vote will go in the same dir­ec­tion. But if the fi­nal res­ult is hov­er­ing at the 2-point threshold, that’s a wrinkle that can cre­ate an un­ex­pec­ted out­come, as the Clin­ton team learned to its dis­may.

It was the in­di­vidu­al state polling that badly missed the mark. In Wis­con­sin, Clin­ton led in each of the 32 pub­lic polls from mid-Au­gust on. The fi­nal Mar­quette Uni­versity Law School, gen­er­ally con­sidered to be the most re­spec­ted in the state, had the Demo­crat up by 6 points. She lost by eight-tenths of a point.

In Pennsylvania, Clin­ton led in 37 out of 38 polls be­gin­ning in early Au­gust. CNN’s last poll had Clin­ton up by 4 points, the fi­nal Quin­nipi­ac poll had her up by 5 points, and the Real­Clear­Polit­ics av­er­age had her up by 1.9 per­cent­age points. She lost by eight-tenths of a point.

In Michigan, Clin­ton was ahead in 25 out of 26 polls taken from the be­gin­ning of Au­gust on. The De­troit Free Press’s last poll had her up by four points, and the Real­Clear­Polit­ics av­er­age had her up by 3.6 points. She lost by two-tenths of a point.

It’s worth not­ing that state polls con­duc­ted by news or­gan­iz­a­tions and uni­versit­ies vary enorm­ously in qual­ity and soph­ist­ic­a­tion. Few state-based news or­gan­iz­a­tions spend the kind of money on polling that many once did. Much of the state-level polling is of a dime-store qual­ity, con­duc­ted by polling firms that are even un­fa­mil­i­ar to polit­ic­al pros.

Ex­per­i­enced journ­al­ists might ar­gue that the over­re­li­ance by re­port­ers on both polls and ana­lyt­ics has led to a de­crease in shoe-leath­er, on-the-ground re­port­ing that might have picked up move­ments in the elect­or­ate that the polls missed. As the Michigan res­ults came in on elec­tion night, I vividly re­called that two con­gress­men from Michigan—one a Demo­crat, the oth­er a Re­pub­lic­an—had been warn­ing me for months that Michigan was more com­pet­it­ive than pub­licly thought. I wished I had listened.

The ana­lyt­ic­al mod­els for both sides poin­ted to a Clin­ton vic­tory, al­beit not a run­away. The Clin­ton cam­paign and su­per PACs had sev­er­al of the most highly re­garded polling firms in the Demo­crat­ic Party, yet in the places that ended up mat­ter­ing, very little if any polling was done. So while 2016 wasn’t a vic­tory for tra­di­tion­al polling, it cer­tainly took a lot of the luster from ana­lyt­ics. In the end, big data mattered very little.

CLA­RI­FIC­A­TION: Ac­cord­ing to Kantar Me­dia/CMAG, a firm that mon­it­ors polit­ic­al ad­vert­ising, the Clin­ton cam­paign’s ad­vert­ising star­ted the week of Nov. 1 in Michigan and Oct. 25 in Wis­con­sin. The cam­paign also made a $70 mil­lion na­tion­al ad buy, $59 mil­lion of which would have been pri­or to Oct. 25, and some of that would have gone in­to Michigan and Wis­con­sin. The cam­paign also had field or­gan­iz­a­tions in both states.