6.3: Measuring Public Opinion

6.3.1: Constructing Public Opinion Surveys

Learning Objective

Discuss how public opinions surveys are designed and executed

Key Points

• There are several ways of administering a survey, such as telephone, mail, online surveys, personal in-home surveys, personal mail or street intercept survey, and hybrids of the above.
• A survey typically consists of a number of questions the respondent answers in a set format. There are open-ended and closed-ended questions. An open-ended question asks the respondent to form their own answer, while closed-ended questions have the respondent choose an answer from a given option.
• The most important aspects of a survey include identifying and selecting potential sample members, contacting individuals and collecting data from those who are hard to reach, evaluating and testing questions, and selecting the mode for posing questions and collecting responses.

Key Terms

Self-Selection Bias

Although individuals chosen to participate in surveys are often randomly sampled, errors due to non-response may exist. Some prospective respondents may simply be less likely to respond to polls generally, they may not be interested in the subject, or they may be unreachable for a variety of reasons.

Opinion Poll

An opinion poll, sometimes simply referred to as a “poll,” is a survey of public opinion from a particular sample. Opinion polls are usually designed to represent the opinions of a population by conducting a series of questions and then extrapolating generalities in ratio or within confidence intervals.

An opinion poll, sometimes simply referred to as a “poll,” is a survey of public opinion from a particular sample . Opinion polls are usually designed to represent the opinions of a population by conducting a series of questions and then extrapolating generalities in ratio or within confidence intervals.

Modes of Data Collection

There are several ways of administering a survey. The choice between administration modes is influenced by: 1) cost, 2) coverage of target population, 3) flexibility of asking questions, 4) respondents’ willingness to participate, and 5) response accuracy. Different methods create mode effects that change how respondents answer. The most common modes of administration are:

• Telephone
• Mail
• Online surveys
• Personal in-home surveys
• Personal mall or street intercept surveys
• Hybrids of the above.

Response Formats

Usually, a survey consists of a number of questions the respondent answers in a set format. A distinction is made between open-ended and closed-ended questions. An open-ended question asks the respondent to formulate his or her own answer, while closed-ended questions have the respondent choose an answer from a given number of options. The response options for a closed-ended question should be exhaustive and mutually exclusive. The four types of response scales for closed-ended questions are:

• Dichotomous: The respondent has two options.
• Nominal-polytomous: The respondent has more than two unordered options.
• Ordinal-polytomous: The respondent has more than two ordered options.
• (Bounded) Continuous: The respondent is presented with a continuous scale.

A respondent’s answer to an open-ended question can be coded into a response scale afterwards or analyzed using more qualitative methods.

Survey Methodology

The most important aspects of a survey include:

• Identifying and selecting potential sample members.
• Contacting individuals and collecting data from those who are hard to reach (or reluctant to respond).
• Evaluating and testing questions.
• Selecting the mode for posing questions and collecting responses.
• Training and supervising interviewers.
• Checking data files for accuracy and internal consistency.
• Adjusting survey estimates to correct for identified errors.

Advantages

• They are relatively easy to administer.
• Can be developed in less time compared with other data-collection methods.
• Can be cost-effective.
• Few “experts” are required to develop a survey, which may well increase the reliability of the survey data.
• If conducted remotely, can reduce or obviate geographical dependence.
• Useful in describing the characteristics of a large population assuming the sampling is valid.
• Can be administered remotely via the Web, mail, e-mail, telephone, etc.
• Efficient at collecting information from a large number of respondents.
• Statistical techniques can be applied to the survey data to determine validity, reliability, and statistical significance, even when analyzing multiple variables.
• Many questions can be asked about a given topic, giving considerable flexibility to the analysis.
• A wide range of information can be collected (e.g., attitudes, values, beliefs, and behavior).
• Because they are standardized, they are relatively free from several types of errors.

Disadvantages

The reliability of survey data may depend on the following:

• Respondents’ motivation, honesty, memory, and ability to respond:
• Structured surveys, particularly those with closed-ended questions, may have low validity when researching effective variables.
• Self-selection bias: Although the individuals chosen to participate in surveys are often randomly sampled, errors due to non-responses may exist. Some prospective respondents may simply be less likely to respond to polls generally, not interested in the subject, or may be unreachable for many reasons. For example, polls or surveys that are conducted by calling a random sample of publicly available telephone numbers will not include the responses of people with unlisted telephone numbers, cell phone numbers, who are unable to answer the phone, and who do not answer calls from unknown/unfamiliar telephone numbers.
• Question design: Survey question answer-choices could lead to vague data sets because, at times, they are relative only to a personal abstract notion concerning “strength of choice”. For instance, the choice “moderately agree” may mean different things to different subjects and anyone interpreting the data for correlation. Even “yes” or “no” answers are problematic because subjects may for instance put “no” if the choice “only once” is not available.

Non-response Reduction

The following ways have been recommended for reducing non-response in telephone and face-to-face surveys:

• Advance letter: A short letter sent in advance to inform the sampled respondents about the upcoming survey.
• Training: The interviewers are thoroughly trained in how to ask respondents questions, work with computers and make schedules for callbacks to respondents who were not reached.
• Short introduction: The interviewer gives the basic information on him/herself and the survey.
• Respondent-friendly survey questionnaire: Questions must be clear, non-offensive, and easy to respond to.

6.3.2: Early Public Opinion Research and Polling

Learning Objective

Identify the historical origins of public opinion research in the United States

Key Points

• As The Harrisburg Pennsylvanian correctly predicted a Jackson win, such straw votes gradually became more popular, but they remained local, usually city-wide phenomena, until the early 1900s.
• The method The Literary Digest used to correctly predict the victories of Warren Harding in 1920, Calvin Coolidge in 1924, Herbert Hoover in 1929, and Franklin Roosevelt in 1932 was mailing out millions of postcards and simply counting the returns.
• Gallup was able to correctly predict the Roosevelt-Landon race by using a sample that was small, however, representative of the general population, while the Digest’s mistake was using a large but skewed sample.

Key Terms

Literary Digest

The Literary Digest was an influential general interest weekly magazine published by Funk & Wagnalls. When the Digest conducted their 1936 election using an inaccurate sample causing them to predict the wrong winner, they lost all credibility and the Digest itself soon went out of business.

The Harrisburg Pennsylvanian

The Harrisburg Pennsylvanian was the publication that conducted the first example of an opinion poll during the Jackson-Adams presidential race.

George Gallup

George Gallup conducted a small but scientific survey that correctly predicted a landslide victory for Roosevelt in the Roosevelt-Landon race, thus establishing the Gallup Poll.

The first known example of an opinion poll was a local straw poll conducted by The Harrisburg Pennsylvanian in 1824, showing Andrew Jackson leading John Quincy Adams by 335 votes to 169 in the contest for the United States Presidency. Since Jackson won the popular vote in the full election, such straw votes gradually became more popular, but they remained local, usually city-wide, phenomena.

In 1916, the Literary Digest embarked on a national survey, partly as a circulation-raising exercise, and correctly predicted Woodrow Wilson’s election as president . Mailing out millions of postcards and simply counting the returns, the Digest correctly predicted the victories of Warren Harding in 1920, Calvin Coolidge in 1924, Herbert Hoover in 1929, and Franklin Roosevelt in 1932.

In 1936, its 2.3 million “voters” constituted a huge sample; however, they were generally more affluent Americans who tended to have Republican sympathies. The Literary Digest was ignorant of this new bias. The week before Election Day, it reported that Alf Landon was far more popular than Roosevelt . At the same time, George Gallup conducted a far smaller, but more scientifically based survey, in which he polled a demographically representative sample. Gallup correctly predicted Roosevelt’s landslide victory. The Literary Digest soon went out of business, while polling started to take off.

Elmo Roper was another American pioneer in political forecasting using scientific polls. He predicted the reelection of President Franklin D. Roosevelt three times, in 1936, 1940, and 1944. Louis Harris had been in the field of public opinion since 1947 when he joined the Elmo Roper firm, then later became partner.

Gallup launched a subsidiary in the United Kingdom, where it almost alone correctly predicted Labour’s victory in the 1945 general election. This contrasted with virtually all other commentators, who expected a victory for the Conservative Party led by Winston Churchill.

By the 1950s, various types of polling had spread to most democracies.

6.3.3: The Gallup Organization

Learning Objective

Locate the historical origins and significance of the Gallup Organization for public opinion research in the United States

Key Points

• To ensure his independence and objectivity, Dr. Gallup resolved that he would undertake no polling that was paid for or sponsored in any way by special interest groups such as the Republican and Democratic parties, a commitment that Gallup upholds to this day.
• Gallup Polls are often referenced in the mass media as a reliable and objective measurement of public opinion. Historically, the Gallup Poll has measured the public’s attitudes concerning virtually every political, social, and economic issue of the day.
• Gallup Polls are best known for their accuracy in predicting the outcome of United States presidential elections.

Key Terms

The Gallup Organization

Gallup Inc. was founded in 1958, when George Gallup grouped all of his polling operations into one organization. Today, Gallup, Inc. is primarily a research-based, performance-management consulting company.

Gallup Poll

The Gallup Poll is the division of Gallup that regularly conducts public opinion polls in more than 140 countries around the world. Gallup Polls are often referenced in the mass media as a reliable and objective measurement of public opinion.

The Gallup Organization

Founded by George Gallup , Gallup, Inc. is primarily a research-based, performance-management consulting company. Some of Gallup’s key practice areas are – employee engagement, customer engagement, and well-being. Gallup has more than 40 offices in 27 countries. World headquarters are in Washington, D.C. Operational headquarters are in Omaha, Nebraska.

Gallup currently has four divisions: Gallup Poll, Gallup Consulting, Gallup University, and Gallup Press.

History

Gallup Inc. was founded in 1958, when George Gallup grouped all of his polling operations into one organization. After Gallup’s death in 1984, The Gallup Organization was sold to Selection Research, Incorporated (SRI) of Lincoln, Nebraska in 1988. George Gallup founded the American Institute of Public Opinion, the precursor of The Gallup Organization, in Princeton, New Jersey, in 1935. He wished to objectively determine the opinions held by the general public. To ensure his independence and objectivity, Dr. Gallup resolved that he would undertake no polling that was paid for or sponsored in any way by special interest groups such as the Republican and Democratic parties, a commitment that Gallup upholds to this day.

In 1936 Gallup successfully predicted that Franklin Roosevelt would defeat Alfred Landon for the U.S. presidency; this event quickly popularized the company. In 1938 Dr. Gallup and Gallup Vice President David Ogilvy began conducting market research for advertising companies and the film industry. In 1958 the modern Gallup Organization was formed from a merger of several polling organizations. Since then, Gallup has seen huge expansion into several other areas.

Gallup died on November 21, 2011.

Gallup Poll

The Gallup Poll is the division of Gallup that regularly conducts public opinion polls in more than 140 countries around the world. Gallup Polls are often referenced in the mass media as a reliable and objective measurement of public opinion. Gallup Poll results, analyses, and videos are published daily on Gallup.com in the form of data-driven news.

Historically, the Gallup Poll has measured and tracked the public’s attitudes concerning virtually every political, social, and economic issue of the day, including highly sensitive or controversial subjects. In 2005, Gallup began its World Poll, which continually surveys citizens in more than 140 countries, representing 95% of the world’s adult population. General and regional-specific questions, developed in collaboration with the world’s leading behavioral economists, are organized into powerful indexes and topic areas that correlate with real-world outcomes.

Gallup Polls are best known for their accuracy in predicting the outcome of United States presidential elections. Notable exceptions include the 1948 Thomas Dewey-Harry S. Truman election, where nearly all pollsters predicted a Dewey victory. The Gallup Poll also inaccurately projected a slim victory by Gerald Ford in 1976, where he lost to Jimmy Carter by a small margin. For the 2008 U.S. presidential election, Gallup was rated 17^th out of 23 polling organizations in terms of the precision of its pre-election polls relative to the final results.

In 2008, Gallup interviewed no fewer than 1,000 U.S. adults each day, providing the most watched daily tracking poll of the race between John McCain and Barack Obama.Gallup also conducts 1,000 interviews per day, 350 days out of the year, among both landline and cell phones across the U.S. for its health and well-being survey.

6.3.4: The National Election Studies

Learning Objective

Identify the purpose of national election studies

Key Points

• The consistency of the studies, as in asking the same questions repeatedly over time, makes it very useful for academic research. As a result, it is frequently cited in works of political science.
• Though the ANES was formally established by a National Science Foundation grant in 1977, the data are a continuation of studies going back to 1948.
• The American Voter‘ concluded that most voters cast their ballots primarily on the basis of partisan identification and that independent voters are actually the least involved in and attentive to politics.

Key Terms

The American Voter

The American Voter (1960) a seminal study of voting behavior in the United States, authored by Angus Campbell, Philip Converse, Warren Miller, and Donald Stokes, colleagues at the University of Michigan.

National Election Studies

The American National Election Studies is the leading academically-run national survey of voters in the United States, conducted before and after every presidential election.

The American National Election Studies (ANES) is the leading academically-run national survey of voters in the United States, conducted before and after every presidential election. Though the ANES was formally established by a National Science Foundation grant in 1977, the data are a continuation of studies going back to 1948. The study has been based at the University of Michigan since its origin and, since 2005, has been run in partnership with Stanford University. Its principal investigators for the first four years of the partnership were Arthur Lupia and Jon Krosnick. Its current principal investigators are Vincent Hutchings, Gary Segura, and Simon Jackman.

The consistency of the studies, which includes asking the same questions repeatedly over time, makes it very useful for academic research. As a result it is frequently cited in works of political science. Early ANES data were the basis for The American Voter a seminal study of voting behavior in the United States, by Angus Campbell, Philip Converse, Warren Miller, and Donald Stokes, colleagues at the University of Michigan. Based on one of the first comprehensive studies of election survey data (what eventually became the National Election Studies), came the conclusion that most voters cast their ballots primarily on the basis of partisan identification (which is often simply inherited from their parents), and that independent voters are actually the least involved in and attentive to politics.

Today ANES data are used by numerous scholars, students, and journalists. It is widely considered the “gold standard” of election studies.

The ANES also has a long history of innovation. In 2006, it opened the ANES Online Commons, becoming the first large-scale academic survey to allow interested scholars and survey professionals to propose questions for future ANES surveys.

6.3.5: Types of Polls

Learning Objective

Compare and contrast the different types of polls utilized to measure public opinion

Key Points

• Benchmark polls, brushfire polls and tracking polls are used by political campaigns to gauge interest in a candidate’s office bid, the success of his/her messaging, and his/her weekly standing. Tracking polls may also be used by news organizations to inform their own reports of the campaign.
• A push poll is an interactive marketing technique, most commonly employed during political campaigning, in which an individual or organization attempts to influence or alter the view of respondents under the guise of conducting a legitimate poll.
• A straw poll, or straw vote, is a poll with nonbinding results. In meetings subject to rules of order, impromptu straw polls often are taken to see if there is enough support for an idea to devote more meeting time to it, and for the attendees to see who is on which side of a question.

Key Terms

Entrance Poll

An entrance poll is taken directly before voters cast their votes; the voter is unlikely change his/her mind after the poll, typically making the margin of error lower than that of an opinion poll.

Tracking Poll

A tracking poll is a poll repeated at intervals generally averaged over a trailing window.

Exit Poll

An exit poll is taken immediately after the voters have exited the polling stations. Pollsters conduct exit polls to gain an early indication as to how an election has turned out.

Types of Polls

There are nine main types of polls :

• An opinion poll is a survey of public opinion from a particular sample. Opinion polls are usually designed to represent the opinions of a population by asking a series of questions and then extrapolating generalities from responses in ratio or within confidence intervals.
• A benchmark poll is generally the first poll taken in a campaign. It is often taken before a candidate announces his or her bid for office, but sometimes it occurs immediately following the announcement, allowing some opportunity to raise funds. This poll is generally a short and simple survey of likely voters.
• Brushfire polls are polls taken during the period between the benchmark and tracking polls. The number of brushfire polls taken by a campaign is determined by how competitive the race is and how much money the campaign has to spend. These polls usually focus on likely voters and the length of the survey varies on the number of messages being tested.
• A tracking poll is a poll repeated at intervals generally averaged over a trailing window. A weekly tracking poll uses the data from the past week and discards older data.
• An entrance poll is a poll that is taken before voters cast their votes. It is akin to an opinion poll in the sense that it asks who the voter plans to vote for and other similar questions. The possibility that the prospective voter might change his or her mind after the poll is very small compared to that of an opinion poll; therefore, the margin of error of an entrance poll is expected to be lower than that of an opinion poll.
• An exit poll is taken immediately after the voters have exited the polling stations. Pollsters —usually private companies working for newspapers or broadcasters—conduct exit polls to gain an early indication as to how an election has turned out, since in many elections the actual result may take hours or even days to count. Exit polls have historically and throughout the world been used as a check against and rough indicator of the degree of election fraud. Like all opinion polls, exit polls by nature do include a margin of error. A famous example of exit poll error occurred in the 1992 UK General Election, when two exit polls predicted a hung parliament. Widespread criticism of exit polling has occurred in cases, especially in the United States, where exit-poll results have appeared and/or have provided a basis for projecting winners before all real polls have closed, thereby possibly influencing election results
• The deliberative opinion poll is a form of opinion poll that incorporates the principles of deliberative democracy. In the deliberative opinion poll, a statistically representative sample of a community is gathered to discuss an issue in conditions that further deliberation. The group is then polled, and the results of the poll and the actual deliberation can be used both as a recommending force and, in certain circumstances, to replace a vote. Rather than simply determining existing public opinion, a deliberative poll aims to understand what public opinion would be if the public were well-informed and had carefully discussed a particular issue. Citizens are invited by modern random sampling techniques to participate; a large enough sampling group will provide a relatively accurate representation of public opinion.
• A push poll is an interactive marketing technique, most commonly employed during political campaigning, in which an individual or organization attempts to influence or alter the view of respondents under the guise of conducting a poll. In a push poll, large numbers of respondents are contacted and little or no effort is made to collect and analyze response data. Instead, the push poll is a form of telemarketing-based propaganda masquerading as a poll and is generally viewed as a form of negative campaigning. This tactic is commonly considered to undermine the democratic process since false or misleading information is often provided about candidates. Push polling has been condemned by the American Association of Political Consultants and the American Association for Public Opinion Research. The term is also used in a broader sense to refer to legitimate polls that aim to test political messages, some of which may be negative. In all such polls, the pollster asks leading or suggestive questions that “push” the interviewee towards adopting an unfavorable response towards the political candidate.
• A straw poll or straw vote is a poll with nonbinding results. Straw polls provide dialogue among movements within large groups. In meetings subject to rules of order, impromptu straw polls often are taken to see if there is enough support for an idea to devote more meeting time to it, and (when not a secret ballot) for the attendees to see who is on which side of a question.

6.3.6: Conducting Polls

Learning Objective

Describe the various methods taken by pollsters to conduct surveys

Key Points

• A questionnaire is a series of questions asked to individuals to obtain statistically useful information about a given topic. When properly constructed and responsibly administered, questionnaires become a vital instrument for polling a population.
• Inappropriate questions, incorrect ordering of questions, incorrect scaling, or bad questionnaire format can make the survey valueless, as it may not accurately reflect the views and opinions of the participants.
• According to the three stage theory, or the sandwich theory, initial questions should be screening and rapport questions. The second stage should concern the product specific questions. In the last stage demographic questions are asked.

Key Terms

Stratified Sampling

Stratified sampling is a method of probability sampling such that sub-populations within an overall population are identified and included in the sample selected in a balanced way.

Open-Ended Question

An open-ended question asks the respondent to formulate his/her own answer.

Closed-Ended Question

A closed-ended question asks the respondent to pick an answer from a given number of options.

Conducting Polls

Generally, in order to conduct a poll, the survey methodologist must do the following :

• Identify and select potential sample members
• Contact sampled individuals and collect data from those who are difficult to reach
• Evaluate and test questions
• Select the mode for posing questions and collecting responses
• Train and supervise interviewers
• Check data files for accuracy and internal consistency
• Adjust survey estimates to correct for identified errors

Survey samples can be broadly divided into two types: probability samples and non-probability samples. Stratified sampling is a method of probability sampling such that sub-populations within an overall population are identified and included in the sample.

Usually, a poll consists of a number of questions that the respondent answers in a set format. A distinction is made between open-ended and closed-ended questions. An open-ended question asks the respondent to formulate his or her own answer; a closed-ended question asks the respondent to pick an answer from a given number of options. The response options for a closed-ended question should be exhaustive and mutually exclusive. Four types of response scales for closed-ended questions are as follows:

• Dichotomous: the respondent has two options
• Nominal-polytomous: the respondent has more than two unordered options
• Ordinal-polytomous: the respondent has more than two ordered options
• (bounded) Continuous: the respondent is presented with a continuous scale

A respondent’s answer to an open-ended question can be coded into a response scale or analyzed using more qualitative methods.

A questionnaire is a series of questions asked to individuals to obtain statistically useful information about a given topic. When properly constructed and responsibly administered, questionnaires become a vital instrument for polling a population.

Adequate questionnaire construction is critical to the success of a poll. Inappropriate questions, incorrect ordering of questions, incorrect scaling, or bad questionnaire format can make the survey valueless, as it may not accurately reflect the views and opinions of the participants. Pretesting among a smaller subset of target respondents is useful method of checking a questionnaire and making sure it accurately captures the intended information.

Questionnaire construction issues

The topics should fit the respondents’ frame of reference. Their background may affect their interpretation of the questions. Respondents should have enough information or expertise to answer the questions truthfully.

The type of scale, index, or typology to be used is determined. The level of measurement used determines what can be concluded from the data. If the response option is yes/no then you will only know how many, or what percent, of your sample answered yes/no. You cannot, however, conclude what the average respondent answered.

The types of questions (closed, multiple-choice, open) should fit the statistical data analysis techniques available and the goals of the poll. Questions and prepared responses should be unbiased and neutral as to intended outcome. The order or “natural” grouping of questions is often relevant. Prior previous questions may bias later questions. Also, the wording should be kept simple: no technical or specialized vocabulary. The meaning should be clear. Ambiguous words, equivocal sentence structures and negatives may cause misunderstanding, possibly invalidating questionnaire results. Care should be taken to ask one question at a time. The list of possible responses should be collectively exhaustive. Respondents should not find themselves without category that fits them. Additionally, possible responses should be mutually exclusive; categories should not overlap. Writing style should be conversational, concise, accurate and appropriate to the target audience. “Loaded” questions evoke emotional responses and may skew results.

Many respondents will not answer personal or intimate questions. For this reason, questions about age, income, marital status, etc., are generally placed at the end of the survey. Thus, if the respondent refuses to answer these questions, the research questions will have already been answered.

Presentation of the questions on the page (or computer screen) and the use of graphics may affect a respondent’s interest or distract from the questions.

Finally, questionnaires can be administered by research staff, by volunteers or self-administered by the respondents. Clear, detailed instructions are needed in either case, matching the needs of each audience.

Question sequence

Some further considerations about questionnaires are the following.

Questions should flow logically from one to the next, from the more general to the more specific, from the least sensitive to the most sensitive, from factual and behavioral questions to attitudinal and opinion questions, from unaided to aided questions.

Finally, according to the three stage theory, or the sandwich theory, initial questions should be screening and rapport questions. The second stage should concern the product specific questions. In the last stage demographic questions are asked.

6.3.7: Analyzing Data

Learning Objective

Discuss how data is broken down and subject to analysis after conducting surveys

Key Points

• The margin of error is usually defined as the “radius” of a confidence interval for a particular statistic from a survey.
• The confidence level, the sample design for a survey, and in particular its sample size, determine the magnitude of the margin of error. A larger sample size produces a smaller margin of error, all else remaining equal.
• The margin of error for a particular sampling method is essentially the same regardless of whether the population of interest is the size of a school, city, state, or country, as long as the sampling fraction is less than 5%.

Key Terms

margin of error

expression of the lack of precision in the results obtained from a sample

Finite Population Correction

The “finite population correction” (FPC) is used to adjust the margin of error to account for the added precision gained by sampling a larger percentage of the population.

margin of error

The margin of error is a statistic that expresses the amount of random sampling error in a survey’s results.

The margin of error is a statistic used to analyze data. It expresses the amount of random sampling error in a survey’s results. The larger the margin of error, the less faith one should have that the poll’s reported results are close to the “true” figures—the figures for the whole population .

Explanation

The margin of error is usually defined as the “radius” of a confidence interval for a particular statistic from a survey. When a single, global margin of error is reported for a survey, it refers to the maximum margin of error for all reported percentages using the full sample from the survey. If the statistic is a percentage, this maximum margin of error can be calculated as the radius of the confidence interval for a reported percentage of 50%.

The margin of error can be described as an “absolute” quantity. For example, if the true value is 50 percentage points, and the statistic has a confidence interval radius of 5 percentage points, then we say the margin of error is 5 percentage points.

However, the margin of error can also be expressed as a “relative” quantity. For example, suppose the true value is 50 people and the statistic has a confidence interval radius of 5 people. If we use the “relative” definition, then we express this absolute margin of error as a percent of the true value. Therefore, the absolute margin of error is 5 people, but the “percent relative” margin of error is 10% (because 5 people are ten percent of 50 people).

The margin of error can be defined for any desired confidence level, but usually a level of 90%, 95%, or 99% is chosen. This level is the probability that a margin of error around the reported percentage would include the “true” percentage. The confidence level, the sample design for a survey, and in particular its sample size, determines the magnitude of the margin of error. A larger sample size produces a smaller margin of error, all else remaining equal.

If the exact confidence intervals are used the margin of error takes into account both sampling error and non-sampling error. If an approximate confidence interval is used then the margin of error may only take random sampling error into account. It does not represent other potential sources of error or bias such as a non-representative sample-design, poorly phrased questions, people lying or refusing to respond, the exclusion of people who could not be contacted, or miscounts and miscalculations.

Basic Concept

Polls typically involve taking a sample from a certain population. In the case of the Newsweek 2004 Presidential Election poll, the population of interest was the population of people who would vote. Sampling theory provides methods for calculating the probability that the poll results differ from reality by more than a certain amount simply due to chance. For example, if the poll reports 47% for Kerry, his support could actually be as high as 50% or as low as 44%. The more people that are sampled, the more confident pollsters can be that the “true” percentage is close to the observed percentage. The margin of error is a measure of how close the results are likely to be.

Effect of Population Size

The margin of error for a particular sampling method is essentially the same regardless of whether the population of interest is the size of a school, city, state, or country, as long as the sampling fraction is less than 5%.

In cases where the sampling fraction exceeds 5%, analysts can adjust the margin of error using a “finite population correction” (FPC) to account for the added precision gained by sampling a larger percentage of the population.

The FPC, factored into the calculation of the margin of error, has the effect of narrowing the margin of error. It holds that the FPC approaches zero as the sample size approaches the population size, which has the effect of eliminating the margin of error entirely.

Comparing Percentages

The terms “statistical tie” and “statistical dead heat” are sometimes used to describe reported percentages that differ by less than a margin of error, but these terms can be misleading. For one thing, the margin of error as generally calculated is applicable to an individual percentage and not the difference between percentages. The difference between two percentage estimates may not be statistically significant even when they differ by more than the reported margin of error. The survey results also usually provide strong information even when there is not a statistically significant difference.

6.3.8: Sampling Techniques

Learning Objective

Compare and contrast the different sampling techniques used for opinion polls

Key Points

• Probability-proportional-to-size (PPS) is sampling in which the selection probability for each element is set to be proportional to its size measure, up to 1. This approach can improve accuracy by concentrating a sample on large elements that have the greatest impact on population estimates.
• Maintaining the randomness in a sample is very important to each sampling technique to ensure that the findings are representative of the population in general.
• Panel sampling is the method of selecting a group of participants through a random sampling method and then asking that group for the same information again several times over a period of time. This longitudinal sampling-method allows for estimates of changes in the population.

Key Terms

Systematic Sampling

Systematic sampling relies on arranging the target population according to some ordering scheme, a random start, and then selecting elements at regular intervals through that ordered list.

Simple Random Sampling

A simple random sampling (SRS) is a sample of a given size in which all such subsets of the frame are given an equal probability to be chosen.

Stratified Sampling

Stratified sampling is a method of probability sampling such that sub-populations within an overall population are identified and included in the sample selected in a balanced way.

Sampling Techniques

In statistics and survey methodology, sampling is concerned with the selection of a subset of individuals from within a statistical population to estimate characteristics of the whole population . The three main advantages of sampling are that the cost is lower, data collection is faster, and the accuracy and quality of the data can be easily improved.

Simple Random Sampling

In a simple random sample (SRS) of a given size, all such subsets of the frame are given an equal probability. Each element has an equal probability of selection. Furthermore, any given pair of elements has the same chance of selection as any other pair. This minimizes bias and simplifies analysis of results. In particular, the variance between individual results within the sample is a good indicator of variance in the overall population, which makes it relatively easy to estimate the accuracy of results.

However, SRS can be vulnerable to sampling error because the randomness of the selection may result in a sample that doesn’t reflect the makeup of the population.

Systematic Sampling

Systematic sampling relies on arranging the target population according to some ordering scheme, a random start, and then selecting elements at regular intervals through that ordered list. As long as the starting point is randomized, systematic sampling is a type of probability sampling. It is easy to implement and the stratification can make it efficient, if the variable by which the list is ordered is correlated with the variable of interest.

However, if periodicity is present and the period is a multiple or factor of the interval used, the sample is especially likely to be unrepresentative of the overall population, decreasing its accuracy. Another drawback of systematic sampling is that even in scenarios where it is more accurate than SRS, its theoretical properties make it difficult to quantify that accuracy. As described above, systematic sampling is an EPS method, because all elements have the same probability of selection.

Stratified Sampling

Where the population embraces many distinct categories, the frame can be organized by these categories into separate “strata. ” Each stratum is then sampled as an independent sub-population, out of which individual elements can be randomly selected. In this way, researchers can draw inferences about specific subgroups that may be lost in a more generalized random sample. Additionally, since each stratum is treated as an independent population, different sampling approaches can be applied to different strata, potentially enabling researchers to use the approach best suited for each identified subgroup. Stratified sampling can increase the cost and complicate the research design.

Probability-Proportional-to-Size Sampling

Probability-proportional-to-size (PPS) is sampling in which the selection probability for each element is set to be proportional to its size measure, up to a maximum of 1.The PPS approach can improve accuracy for a given sample size by concentrating the sample on large elements that have the greatest impact on population estimates. PPS sampling is commonly used for surveys of businesses, where element size varies greatly and auxiliary information is often available.

Cluster Sampling

Sometimes it is more cost-effective to select respondents in groups (“clusters”). Sampling is often clustered by geography or by time periods. Clustering can reduce travel and administrative costs. It also means that one does not need a sampling frame listing all elements in the target population. Instead, clusters can be chosen from a cluster-level frame, with an element-level frame created only for the selected clusters.

Cluster sampling generally increases the variability of sample estimates above that of simple random sampling, depending on how the clusters differ between themselves, as compared with the within-cluster variation.

Quota Sampling

In quota sampling, the population is first segmented into mutually exclusive subgroups, just as in stratified sampling. Then judgment is used to select the subjects or units from each segment based on a specified proportion. For example, an interviewer may be told to sample 200 females and 300 males between the age of 45 and 60. In quota sampling the selection of the sample is non-random. The problem is that these samples may be biased because not everyone gets a chance of selection.

Accidental Sampling

Accidental sampling (or grab, convenience, or opportunity sampling) is a type of non-probability sampling which involves the sample being drawn from that part of the population which is close to hand. The researcher cannot scientifically make generalizations about the total population from this sample because it would not be representative enough.

Panel Sampling

Panel sampling is the method of first selecting a group of participants through a random sampling method and then asking that group for the same information again several times over a period of time. This longitudinal sampling-method allows estimates of changes in the population.

6.3.9: The Importance of Accuracy

Learning Objective

Discuss the importance of maintaining accuracy when conducting measuring public opinion

Key Points

• Relevance of the survey information, quality of the data, and overcoming personal bias are integral to polling accuracy.
• The quality of the accurate and timely results must be assessed prior to release. If errors in the results occur they should be directly corrected and the public should be informed as soon as possible.
• When social scientists speak of good research the focus is on how the research is done rather than on whether the results of the research are consistent with personal biases or preconceptions.

Key Term

Literary Digest

The Literary Digest was an influential general interest weekly magazine published by Funk & Wagnalls. When the Digest conducted their 1936 election using an inaccurate sample causing them to predict the wrong winner, they lost all credibility and the Digest itself soon went out of business.

The importance of accuracy may be illustrated through the example of the Literary Digest Roosevelt-Landon presidential election poll. After correctly predicting the victories of Warren Harding in 1920, Calvin Coolidge in 1924, Herbert Hoover in 1929, and Franklin Roosevelt in 1932, the Literary Digest had established itself as a well-known and well-respected publication. One reason for their previous successes was the use of a very large sample population.

In 1936, the Digest conducted their presidential poll with 2.3 million voters, a huge sample size. However, the sample turned out to be an inaccurate representation of the general population as those polled were generally more affluent Americans who tended to have Republican sympathies. The Literary Digest was ignorant of this new bias. The week before Election Day, it reported that Alf Landon was far more popular than Roosevelt. At the same time, George Gallup conducted a far smaller, but more scientifically based survey, in which he polled a more demographically representative sample. Gallup correctly predicted Roosevelt’s landslide victory. The Literary Digest lost its reputation for accuracy and the trust of the readers and soon went out of business.

Maintaining Polling Accuracy

Relevance of the survey information, quality of the data, and overcoming personal bias are integral to polling accuracy.

When releasing information, data and official statistics should be relevant to the needs of users as well as both public and private sector decision makers. The quality of results must be assessed prior to release. If errors in the results occur before or after the data revision, they should be corrected and users should be informed as quickly as possible. Finally, when social scientists speak of “good research,” the focus is on how the research is done–whether the research is methodologically sound–rather than on whether the results of the research are consistent with personal biases or preconceptions.

Glenn Firebaugh summarizes the principles for good research in his book Seven Rules for Social Research. He states that “there should be the possibility of surprise in social research. ” In other words, it is imperative that the researchers look past their preconceived notions or desires to conduct a study that reflects whatever the reality may be. Additionally, good research will “look for differences that make a difference” and “build in reality checks. ” Researchers are also advised to replicate their polls, that is, “to see if identical analyses yield similar results for different samples of people. ” The next two rules urge researchers to “compare like with like” and to “study change;” these two rules are especially important when researchers want to estimate the effect of one variable on another. The final rule, “let method be the servant, not the master,” reminds researchers that methods are the means, not the end, of social research; it is critical from the outset to fit the research design to the research issue, rather than the other way around.

6.3.10: The Problems with Polls

Learning Objective

Identify some of the common problems with conducting opinion polls

Key Points

• It is well established that the wording of the questions, the order in which they are asked, and the number and form of alternative answers offered can influence results of polls.
• Coverage bias is another source of error involving the use of samples that are not representative of the population due to the polling methodology.
• Self-selection bias arises in any situation in which individuals select themselves into a group, causing a biased sample with non-probability sampling.

Key Term

Literary Digest

The Literary Digest was an influential general interest weekly magazine published by Funk & Wagnalls. When the Digest conducted their 1936 election using an inaccurate sample causing them to predict the wrong winner, they lost all credibility and the Digest itself soon went out of business.

Potential for Inaccuracy

In practice, pollsters need to balance the cost of a large sample with the reduction in sampling error. A sample size of around 500 – 1,000 is a typical compromise for political polls . Another way to reduce the margin of error is to rely on poll averages. This method is based on the assumption that the procedure and sample size is similar enough between many different polls to justify creating a polling average.

Another source of error stems from faulty demographic models by pollsters who weigh their samples by particular variables such as party identification in an election. For example, one could assume that the breakdown of the US population by party identification has not changed since the previous presidential election. As a result, one would underestimate a victory or a defeat of a particular party candidate that saw a surge or decline in its party registration relative to the previous presidential election cycle.

Theories on Erroneous Polling Results

A number of theories and mechanisms have been offered to explain erroneous polling results. Some of these reflect errors on the part of the pollsters; many of them are statistical in nature. Others blame the respondents for not giving candid answers (the controversial Bradley effect & Shy Tory Factor).

Non-response Bias

Since some people do not answer calls from strangers or refuse to answer the poll, poll samples may not be representative samples from a population due to a non-response bias. Because of this selection bias, the characteristics of those who agree to be interviewed may be markedly different from those who decline. That is, the actual sample is a biased version of the universe the pollster wants to analyze. In these cases, bias introduces new errors, in addition to errors caused by sample size. Error due to bias does not become smaller with larger sample sizes–taking a larger sample size simply repeats the same mistake on a larger scale.

Response Bias

Surveys may be affected by response bias, where the answers given by respondents do not reflect their true beliefs. This may be deliberately engineered by unscrupulous pollsters in order to generate a certain result or please their clients, but more often is a result of the detailed wording or ordering of questions. Respondents may deliberately try to manipulate the outcome of a poll by advocating a more extreme position than they actually hold in order to boost their side of the argument or give rapid and ill-considered answers in order to hasten the end of their questioning. Respondents may also feel under social pressure not to give an unpopular answer. In American political parlance, this phenomenon is often referred to as the Bradley effect. If the results of surveys are widely publicized this effect may be magnified in a phenomenon commonly referred to as the spiral of silence.

Wording of Questions

It is well established that the wording of the questions, the order in which they are asked, and the number and form of alternative answers offered can influence results of polls. For instance, the public is more likely to indicate support for a person who is described by the operator as one of the “leading candidates. “

A common technique to control for this bias is to rotate the order in which questions are asked. Many pollsters also split-sample in that one of two different versions of a question are presented to half the respondents.

Coverage Bias

Another source of error is the use of samples that are not representative of the population as a consequence of the polling methodology. For example, telephone sampling has a built-in error because in many times and places, those with telephones have generally been richer than those without.

Selection Bias

Selection bias occurs when some units have a differing probability of selection that is unaccounted for by the researcher. For example, some households have multiple phone numbers making them more likely to be selected in a telephone survey than households with only one phone number.

In statistics, self-selection bias arises in any situation in which individuals select themselves into a group, causing a biased sample with non-probability sampling. It is commonly used to describe situations where the characteristics of the people which cause them to select themselves in the group create abnormal or undesirable conditions in the group.

There may be a purposeful intent on the part of respondents leading to self-selection bias whereas other types of selection bias may arise more inadvertently, possibly as the result of mistakes by those designing any given study.

6.3.11: Telephone and Internet Polling

Learning Objective

Identify the advantages and disadvantages of telephone and internet polling

Key Points

• Probability samples of internet polls are highly affected by problems of non-coverage. Not all members of the general population have Internet. In addition, online survey invitations are distributed using e-mail, but there are no e-mail directories of the general population.
• Online survey may be affected by non-response as response rates are generally low and vary extremely. Some may refuse participation, terminate surveys during the process, or not answer certain questions.
• Telephone interviewers encourage sample persons to respond, leading to higher response rates and interviewers may increase comprehension of questions by answering respondents’ questions.
• There are some disadvantages to telephone polling such as interviewer bias, the fact it cannot be used for non-audio information, and it is unreliable for consumer surveys in rural areas where telephone density is low.

Key Terms

Telephone Polling

Telephone polling is also fairly cost efficient, depending on local call charge structure, which makes it good for large national (or international) sampling frames.

Internet Polls

Internet polls are becoming an essential research tool for a variety of research fields, including marketing and official statistics research. Web polls are faster, simpler, and cheaper than many other polling methods.

Internet polls

Online polls are becoming an essential research tool for a variety of research fields, including marketing and official statistics research.

Advantages of Internet Surveys

Web polls are faster, simpler, and cheaper than many other polling methods. However, lower costs are not so straightforward in practice, as they are strongly interconnected to errors. Because response rate comparisons to other survey modes are usually not favorable for online surveys, efforts to achieve a higher response rate may substantially increase costs. Additionally, the entire data collection period is significantly shortened, as all data can be collected and processed in typically little more than a month.

Interaction between the respondent and the questionnaire is also more dynamic compared to e-mail or paper surveys. Online surveys are also less intrusive, and they suffer less from social desirability effects. Questions with long lists of answer choices can be used to provide immediate coding of answers to certain questions that are usually asked in an open-ended fashion in paper questionnaires. Finally, online surveys can be tailored to the situation (the questionnaire may be preloaded with already available information).

_{Methodological Issues of Online Surveys}

Sampling

The difference between probability samples (where the inclusion probabilities for all units of the target population is known in advance) and non-probability samples (which often require less time and effort but generally do not support statistical inference) is crucial. Probability samples are highly affected by problems of non-coverage (not all members of the general population have Internet access) and frame problems (online survey invitations are most conveniently distributed using e-mail, but there are no e-mail directories of the general population that might be used as a sampling frame). Because coverage and frame problems can significantly impact data quality, they should be adequately reported when disseminating the research results.

Invitations to Online Surveys

Due to the lack of sampling frames, many online survey invitations are published in the form of an URL link on web sites or in other media, which leads to sample selection bias that is out of research control and to non-probability samples. Traditional solicitation modes, such as telephone or mail invitations to web surveys, can help overcoming probability sampling issues in online surveys. However, such approaches are faced with problems of dramatically higher costs and questionable effectiveness.

Non-response

Online survey response rates are generally low and also vary extremely. In addition to refusing participation, terminating surveying during the process, or not answering certain questions, several other non-response patterns can be observed in online surveys, such as lurking respondents and a combination of partial and item non-response. Response rates can be increased by offering monetary or some other type of incentive to the respondents, by contacting respondents several times, and by keeping the questionnaire difficulty as low as possible.

Questionnaire Design

The use of design features should be limited to the extent necessary for respondents to understand questions or to stimulate the response. The features should not affect their response as that would mean lower validity and reliability of data.

It is important that uncontrolled variations in how a questionnaire appears are minimized. Web-based survey methods make the construction and delivery of questionnaire instruments relatively easy, but what is difficult to ensure is that everyone sees the questionnaire as its designer intended it to be. This problem can arise due to the variability of software and hardware used by respondents.

Telephone Polling

An important aspect of telephone polling is the use of interviewers. Interviewers encourage sample persons to respond, leading to higher response rates and interviewers may increase comprehension of questions by answering respondents’ questions.

Telephone polling is also fairly cost efficient, depending on local call charge structure, which makes it good for large national (or international) sampling frames .

However, there are some disadvantages to telephone polling. For instance, there is some potential for interviewer bias (e.g. some people may be more willing to discuss a sensitive issue with a female interviewer than with a male one), telephone polling cannot be used for non-audio information (graphics, demonstrations, taste/smell samples), and it is unreliable for consumer surveys in rural areas where telephone density is low .

There are three main types of telephone polling: traditional telephone interviews, computer assisted telephone dialing, and computer assisted telephone interviewing ( CATI ).