August 31, 2007

Perspective - Pooh Poohing the Polls

By Ken Draper

CityWatch published the results of a survey a week ago that claimed that 47.5% of the City’s population has heard of neighborhood councils. As the whine continues that neighborhood council outreach has fallen short and that nobody outside of a ‘few of my closest friends’ has ever heard of neighborhood councils, we found the report rather remarkable. Not that surprising, I must admit, but remarkable none-the-less.

CityWatch had no sooner hit the internet ‘doorstep’ than a reader was in our ear about the authenticity of the results. Here’s his unedited case:

“This is sooooooooo bogus. you have to be kidding me. 650 people is not sufficient for a sample size for a city this large. they would need to have at least 2000 respondents to have been taken seriously.

i mean c'mon. there are 40,000 people in the west l.a. neighborhood council district. the number of people who are aware of the council is
not even 200. do the math. thats less than 1%.

why do you post such bogus news??”

I will tell you why, and so will Dr. Robinson. You can choose the answer you like, but, I will tell you up front, they both have the same ending.

First … and easiest … this work was done by the good professional folks at the Social Science Research Center at Cal State Fullerton. The word science is not in there by accident. This kind of research requires people with expertise. It is not a seat-of-the-pants racquet.

This kind of polling success is not the result of numbers alone. The process is layered, complicated and scientific. It requires knowledge, experience and expertise. Dr. Robinson has been doing this work for a goodly number of years.

Polling is a part of our daily lives. It tells us how popular … or unpopular … the Mayor is. It forecasts election results. It projects the popular tastes of television viewers. And polling science is the spine of all of that work. Point being: whether a sample of 640 … with a margin of error at 3.95% … is appropriate for the conclusions presented is being decided by persons with experience and expertise and not the result of a gut reaction or the inability to envision the possibility that the earth is not flat.

Lastly, I’m not clear on why 47.5% of the adult population being aware of neighborhood councils comes as a surprise. Councils have been around now for seven years. A couple of hundred thousand Angelinos voted for neighborhood councils as far back as 1999. Thousands of stakeholders have moved into and through the NC process over that time. Thousands of current board members have families, friends and co-workers … who must hear the words, neighborhood council, occasionally. Two and a half million viewers watched the CityWatch/CBS Mayoral Debates two and a half years ago. City Council and Council committees include neighborhood councils in the city process on a televised and broadcast daily basis. The words are no longer foreign to LA’s media. Positive, front page, opinion section of the LA Times two Sunday’s ago. Perhaps, in the awareness sense, NCs have the benefit of more outreach than most of us perceive. So, why should we wonder when told that almost half of the population has heard of councils?

The research that Cal State Fullerton has offered up is not bogus. Neither is the work of the USC and the Loyola Marymount researchers. And, it certainly outweighs the guestimates of amateurs who feel they have a sense of what numbers are appropriate for credibility. And, used properly, we think that the information is helpful in letting us know who we are, how we got here and what we need to do yet to fulfill the neighborhood council vision. Make what you want of the results. But, don’t blame the research because you don’t like the answers.

Dr. Gregory Robinson offers a more scientific response.

This is a good and a fair question. Generally, a confidence interval of plus or minus five percent is the target for policy-relevant survey data. This refers to the level of precision required of estimates of an entire population or study area that are based upon sample data. When a confidence interval is specified, we can be 95% confident that the true population parameter lies within an interval extending that percentage above and below any proportion derived from survey data. A population parameter is the result one would obtain if every member of the population (with regard to the NCRC survey, the head of household or her or his spouse or domestic partner in every household with a telephone in the City of Los Angeles) were interviewed.

Sampling error, as indexed by the confidence interval around reported proportions, varies in relation to sample size and to the variability of survey responses (among other factors). In general, as the sample grows larger, the confidence interval grows narrower. That is, inferences about population parameters based upon larger samples are more precise (are associated with narrower confidence intervals) than inferences based upon smaller samples.

Also, as the proportion of some attribute in the sample (e.g. awareness of Neighborhood Councils) approaches a fifty/ fifty split, (say, 50% of LA residents are aware of NCs and 50% aren’t) sampling error increases, resulting in a wider confidence interval. Conversely, sampling error decreases as the proportion of a given attribute approaches a five/ ninety-five split, (5% are aware and 95% aren’t) resulting in a more precise estimate and a narrower confidence interval. This is what has made predicting recent national elections such a bear…when the electorate is split at close to 50/50, the population parameter (remember Gore vs. Bush?) can easily fall within the confidence interval.

For the NCRC survey we calculated the confidence interval conservatively, that is, based upon the presumption of a fifty/ fifty proportion in the sample data. Although it may seem incredible, provided that it is randomly selected, a sample size of 400 in very large populations (more than 20,000) produces the requisite confidence interval of plus or minus five percent (whether the population is 200,000 2,000,000 or more). Consider that the voting behavior of the nation is inferred from samples of 2,000. Counter-intuitively, when the population is small, the sample size must be proportionately larger. For example, if the population size is just 1,000 the necessary sample size to produce a confidence interval of plus or minus five percent is 286.

The NCRC survey sample of 640 resulted in a confidence interval of plus or minus 3.95%, more precise than the convention for policy-relevant research of plus or minus 5%. Again, the key here is that a random sample is selected. When this has been done successfully, the sample tends to emulate the characteristics of the population from which it is selected. Our report shows that the NCRC sample is pretty close on several key demographic attributes to the LA population.

--Gregory Robinson, Ph.D.

Director, Social Science Research Center