Leading Poll Questions

One of the points made in the text is how the wording of poll questions can influence the outcome of the poll. Here's a particularly egregious example, from an article entitled The Good, the Bad, and the Ugly of Public Opinion Polls, which can be read in its entirety here.

In 2002 the professional golf tour witnessed a political fight which ultimately yielded a hired gun poll that almost exactly resembles a push poll. Chairman and CEO Hootie Johnson of the Augusta Golf Club chose an aggressive counter-campaign to Martha Burk of the National Council of Women's Organizations, who sought to oblige the Masters' Golf Tournament's host club to open its doors to women for the first time. He hired The Polling Company and WomanTrend, a Washington D.C. polling firm chaired by a prominent Republican woman named Kellyanne Fitzpatrick Conway (The Polling Company TM + Biography).

The result was satisfying for CEO Johnson and unsatisfying for Burk. Conway accompanied Johnson at a November 13, 2002 press conference to announce the poll result, which had an 800-person-based sampling error of +/- 3.5%. As portrayed in the official PGA website (Poll shows support for Augusta's right to choose membership - PGATOUR.COM): "When asked whether -- like single-sex colleges, the Junior League, sororities, fraternities and other similar same-sex organizations -- "Augusta National Golf Club has the right to have members of one gender only," 74 percent of respondents agreed. Asked whether Augusta National was "correct in its decision not to give into Martha Burk's demand," 72 percent of the respondents agreed.'" That would appear to wrap the matter up.

But a look at the poll questions is instructive. They are clearly aimed at a push throughout the survey. We get this language in Question 21 (Augusta National poll Part III - PGATOUR.COM):

21. As you may or may not know, Augusta National Golf Club is a private golf club in Augusta, Georgia that does not receive any type of government funding. Each year, the Masters Tournament is held at Augusta National Golf Club. Currently, only men are members.

Martha Burk, the President of the National Council of Women’s Organizations, wrote a letter to the Augusta National Golf Club, saying that the Masters Golf Tournament should not be held at a club that does not have women members. She demanded that the Golf Club review its policy and change it immediately, in time for the tournament scheduled for April 2003.

Do you recall hearing a lot, some, only a little, or nothing about this?

The result is that 51 percent of the sample heard nothing about this. Normally that's a warning to pollsters not to proceed further with questions except under very high cautions. But here, Question 22 proceeds immediately with this stem:

22. And, as you may or may not know, the Chairman of Augusta National Golf Club, William Johnson, responded to Martha Burk by saying that membership to the club is something that is determined by members only, and they would not change their policies just because of Burk’s demand.

And, do you support or oppose the decision by Augusta National Golf Club to keep their membership policy as it is?

The result was net support by 62 percent, opposition by 30 percent, and a volunteered "do not know" from the remainder. Then Question 23:

23. Although currently, there are no women members of the Augusta National Golf Club, the Golf Club does allow women to play on their golf course, and visit the course for the Masters Tournament. In other words, women are welcome to visit the Club and often play golf there as guests.

Knowing this, would you say that you support or oppose the Augusta National Golf Club’s decision to keep their membership policy as it is?

This time we get 60 net support for the status quo and 33 percent opposed. Questions 24 and 25 then sally forth in this fashion:

Please tell me if you agree or disagree with the following statements:

24. “Martha Burk had the right to send a letter to Augusta National Golf Club about their membership policies, but if she really wanted to make some progress on behalf of women, she would have focused her time and resources on something else.” [and]

25. “Martha Burk did not really care if the Augusta National Golf Club began allowing women member, she was more concerned with attracting media attention for herself and her organization.”

The replies to this being satisfactory, the key item 26 comes in:

26. “The Augusta National Golf Club was correct in its decision not to give into Martha Burk’s demand. They should review and change their policies on their own time, and in their own way.”

That got 72 percent to agree to not bending to this awful woman's unreasonable demands against a selfless and public-minded private club that welcomes women golfers with open arms. A little later, Question 29 kept up the drumbeat:

And please tell me if you agree or disagree with the following statement:

29. “Just like single-sex colleges, the Junior League, Boys and Girls Scouts, Texas Women’s Shooting Club, Sororities and Fraternities, and women business organizations, Augusta National Golf Club has the right to have members of one gender only.”

Lo, this produced a full 74 percent sample agreement with some form of defense for the existence of single-gender organizations in America. That was the highest proportion of any of these leading items, and thus was the single one seized by Mr. Johnson for highlighting in his press conference accompanied by this hired-gun poll's principal.

But a rebuke to the "Hootie Poll" soon come from within the golf community itself. The November 14, 2002 issue of PGA Tour's Golf Web carried a piece entitled Is the Augusta National poll misleading? - PGATOUR.COM from GolfWeb Wire Services. Its verdict: "The "Hootie Poll" is a mishmash of loaded statements and biased, leading questions that are unworthy of Johnson or Augusta. It is a poll that is slanted to get the answers they wanted, and in that it succeeded."