Discussion Summary: July 20, 2001

Dialogue Day 10 Agenda: Evaluation

Dialogue moderator Bob Carlitz and host Larry Teller from EPA's Office of Communications and Government Relations welcomed participants to the final day of this Dialogue, and introduced today's topic: Evaluation. Participants were invited to focus on four areas:

• Increasing the transparency of decision-making
• Criteria for policy effectiveness
• Criteria for public involvement success
• Lessons for EPA from this Dialogue.

The Environmental Protection Agency would like to remind you that your messages during the National Dialogue on Public Involvement in EPA Decisions were not a formal part of the comment process on the draft Public Involvement Policy. The Agency would like to receive your comments on the draft policy before the comment period closes on July 31st. Information on how to submit your comments on the draft Policy to EPA can be found in the Formal Comment section of this Web site.

For the evaluation of this National Dialogue, Resources for the Future has constructed an Evaluation Survey that all participants are invited to complete. Survey results will be posted on this Web site and the Resources for the Future (RFF) website http://www.rff.org.

Participants' comments clustered in the following three areas.

Increase Transparency of Decision-Making

• Several participants, in different ways, called on EPA to clearly describe its decision-making process, and state specifically the points at which public input could make a difference. In all actual decisions, EPA should note early in the document how the Agency had reacted to public input. EPA should do this even when the public input and the ultimate decision don't agree. Host Larry Teller supported this, noting that such a policy is most likely to prevail if EPA middle and senior managers support it. The use of neutral facilitators as record-keepers was again suggested.

• One participant called for making the EPA web page easier to use, with hot links to information instead of the present system of print links to the Federal Register (often experienced as unavailable and unreadable).

Criteria for Policy Effectiveness and Public Involvement Success

• A posting from John V. Stone described two possible models for evaluation: "outcome" evaluation, in which success is defined in terms of goals clearly articulated at the outset; and "formative" evaluation, where feedback is continuous and goals are reshaped appropriately. (A URL with details is given.) Many participants agreed on the need for continuous feedback and continuous attention to feedback. There seemed to be implicit agreement that EPA needs to pay attention to feedback early in any decision- making process, and at least fine-tune goals accordingly.

• One participant suggested that public involvement success be measured by the level of public comment, and suggested a variety of ways to gauge the level. Other participants objected to this approach, saying that members of the public who were completely satisfied with EPA's participation process would simply feel no need to weigh in (a detailed example from New York State was offered). A bell-curve model offered by David James suggested that people in complete disagreement with EPA's objectives would be equally unlikely to weigh in.

• Several participants discussed the potential of market- research tools like customer-satisfaction surveys and focus groups. EPA was cautioned not to use these tools without informed guidance; it is easy to make mistakes by generalizing survey results inappropriately. However, the participant who offered this caution had a number of specific suggestions for using surveys to capture the input of those who cannot participate otherwise because of situational, cultural, or financial barriers.

• Panelist Eric Marsh outlined features of the draft Public Involvement Policy and noted that EPA does plan to develop customer-satisfaction survey forms.

• Analyzing press coverage of EPA actions, and a conference for face-to-face discussion of specific instances of public discussion, were both suggested.

• One participant suggested using an April 1998 EPA report titled "Final Guidance for Incorporating Environmental Justice Concerns in EPA's NEPA Compliance Analysis" as a tool for evaluation. This report is said to offer useful guidance on how to conduct environmental assessments.

Lessons from the Dialogue for EPA

• Panelist Eric Marsh urged participants to fill out the survey evaluating the Dialogue (see URL above). Panelist Caron Chess cautioned against lumping Agency participants' comments in with everyone else's, since Agency personnel will probably have a different perspective from that of the non-Agency public.

• One participant would like to have the final day's discussion be about linkages between all of the Dialogue agenda topics. He worries that since the topics have been treated as discrete, EPA will fail to look for fruitful connections. Another participant said that each topic should have been given more than one day.

• Several participants (and one panelist) warned against making the evaluation process too much of an end in itself. One participant cautioned against "group-think," where a group values its internal harmony over the achievement of an outcome.

• Showing that EPA is indeed listening, Lisa Kahn reported that she and Pat Bonner will be analyzing the Dialogue, and have been tabulating all of the "worst practices" messages. She will be posting her list (130 pp. at time of writing) to the Dialogue website.

• There was a lively exchange about the value of tolerance, with some participants concerned that too much tolerance will let polluters off the hook. Others answered that only some degree of empathetic listening to a polluter would lead to effective measures. While civility ultimately won in this exchange, EPA was called upon to champion public health and environmental protection, no matter how controversial its necessary actions.

CORRECTION:Fred Stoss's database of subnational agencies and health sites is compiled by and must be accessed directly from him, not from the National Council for Science and the Environment, as stated in the Summary of Day 9. (See his message on Day 9.)

Each day's summary is intended to capture the essence of the conversation. While this summary contains the highlights of participants' contributions relating to today's topics, more comprehensive information may be found in the individual postings.

Katherine Carlitz, Reporter