RE: Criteria for policy effectiveness/success

[Mia Zmud <mia@nustats.com>]

Larry's question as to whether "using some of the feedback techniques used for years to measure customer satisfaction (such as surveys, focus groups, staff interviews) are readily useful for measuring the public's satisfaction with involvement in a rulemaking or decision" is a good one.  These would work, and would be "reliable" and "unbiased" if the results were carefully and appropriately reported.  For instance, since you don't have a "representative, random sample" you wouldn't be able to generalize survey results to the population or even participants on a whole--the data would only be representative of those who participated in the PIP or event that was being evaluated. Here are two other ideas (I recommend using a third party skilled in research for conducting comprehensive evaluations):  (1) If a national PIP, conduct a random sample of all known participants of the entire public involvement process/event(s) and (2) If the PIP was conducted on a community or regional level--conduct a random sample of all residents--whether they were participants or not.  For #2, your evaluation could explore many issues such as participation motivation, effectiveness of PR/advertisement, perceptions, behaviors, etc.

After having said all this...before EPA starts thinking about methods for evaluating (survey, focus group, in depth interviews--all good methods!), you have to focus on what is the purpose of evaluating Public Involvement, define what is "successful" Public Involvement or get at the core of why or what you are measuring to begin with.   What do you want to learn through the evaluation? Who is your audience for the results? What will be done with the results?  Answering these and other questions helps zero in on what it is you want and need to measure.

I believe that many of the comments given already today are helping to do this.  Here's my two cents.  As someone who conducts public involvement using a "Virtual Forum" approach (incorporating elements of market research, communications and traditional and non-traditional public involvement and bridging it together with technology), I believe successful public involvement: (1) Captures an inclusive community or stakeholder voice (this ensures equal influence on decision making and assuring that the louder voices (often gatekeepers) are not given more weight)  (2) Involves the "silent majority"--those who tend to be the traditionally under-served or those who can't participate because of situational, cultural, or financial barriers,(3) Provides multiple public participation channels to give all stakeholders flexibility to participate and the freedom to choose the participation mode they find most comfortable, and (4) Offers two-way communications with feedback on how input is being or was incorporated.  There are countless ways to collect both quantitative and qualitative feedback on these criteria for successful public involvement.