Discussion #1 – 2/1/11

1. Welcome and Introductions (15 min)

2. Environmental Evaluators Network: (10 min)

a.    History of the EEN and the Forum
b.    EEN in 2011

3. 2011 Forum Content and Format (20 min)

a.    Theme

i.    What are your thoughts about “Complexity” as the theme for the 2011 EEN Forum?
ii.   In the context of complexity, what are some specific topics or issues that EEN participants should discuss?

b.    Improving capacity for evaluation in the environmental sector (20)

i.    Considering a theme of ‘Complexity’, what skills, knowledge, tools, research, experience, etc. are (or will be) most critical for   improving the capacity of EEN participants to advance the practice, theory and policy of measurement and evaluation?
ii.   How should these be discussed or presented at the Forum?

c.     Effective networking (20 min)

i.    What do you want to achieve through networking at the EEN Forum?
ii.   How would you like to network with EEN participants?

4. Future role of the Advisory Group  (5 min)

a.    Continued feedback
b.    Call for proposals…for presentation and discussion
c.    Promoting EEN in our networks

(15 min) (Led by Matt Keene)

Matt Keene of the EPA’s welcomed all participants and explained the purpose, scope, and structure of the conference call.  He clarified that the Advisory Group would help brainstorm the content, structure, and delivery of the 2011 EEN Forum, which the EEN agency sponsors and EEN contractor would use to develop the 2011 Forum.  There was no intent to achieve consensus on any of these issues by the end of the call as Matt would be coordinating future calls with AG to further define these ideas.  He stressed the importance of leveraging the diversity of the AG and reinforcing the notion of the EEN as a Community of Practice (a thought he repeated later in the meeting).

Participants included representatives from both domestic and international organizations, from the U.S. public sector, private sector, and nonprofit sector.  Many individuals participated in the call, introduced themselves and informed the group about their professional background and interests.

Click here to view a complete list of the Group Members.

(10 min)

a.    Background—Matt Keene provided the Creation Story for EEN and explained how the network and Fora have evolved over time.  Christina Kakoyannis’ predecessor at NFWF, Matt Birnbaum and the EPA’s Katherine Dawes met at AEA annual conference 6 years ago and formed a group interested in environmental evaluations and the first forum was held in 2006.  In 2010—90 different organizations participated in the forum.

b.    EEN in 2011—The 2011 forum will be held on June 23rd & June 24th.  Matt Keene announced that he would share the survey results from the 2010 EEN Forum with the AG since most of the AG members had volunteered to review and analyze those results to help shape the 2011 Forum.

a.    Theme (20 min)

i.    What are your thoughts about “Complexity” as the theme for the 2011 EEN Forum?

ii.    In the context of complexity, what are some specific topics or issues that EEN participants should discuss?

iii.    Jonny – People use complexity in different ways – intuitive sense and a formal sense.  People often believe that if its hard to understand, must be complex but there is a formal sense of complexity like Christina talked about.  Must distinguish between the two.  In science we must distinguish.  We are much stronger on explanation of things than the prediction of things.  We should deal with this.  Complexity leads one to think of butterflies in the amazon causing hurricanes but complex systems can be stable.  What does change mean in a complex system?  We need to not always think in terms of complexity.  Sometimes models can be trusted and sometimes they can’t and so we must consider why and when.  We should make sure we think of evaluation methods and how they deal with unexpected change.

Four more thoughts: Love the idea of “navigating complexity” as a theme; A lot of interest in garden variety evaluation issues like Performance Measurement and we shouldn’t lose site of this; Knowledge use and behavior should be addressed – what’s it mean to use Knowledge, The event must provide opportunities for rich networking

iv.    Mary – really critical to talk about interdisciplinary methods that must be considered in complexity.  We should encourage interdisciplinary case studies that connect with evaluation.  Must come back to where rubber meets road in political life of country so we must talk about social and environmental systems.

v.    Nick – Complexity is something at OMB that we hear a lot about from the programs we work with.  There are a lot of different directions we could go with it – to what level do we want to talk about modeling? Programs often say there was no way to develop a model because dependent variable always changing…is water quality improving…where do you get the baseline to use.  I hesitate to focus on ‘modeling’ but that’s what’s on my mind right now.  How do nonlinear models and their results get communicated to decision makers?

vi.    Richard – A theme of “complexity” could be daunting if we don’t handle it as a theme that is empowering vs overwhelming to EEN participants.  I’m more used to working with smaller groups using less sophisticated tools than what I’m hearing on the line today.  We should discuss aspects of bringing evaluation tools to bear throughout the project lifecycle in order to get the perspective of what the evaluation professional can bring to the table throughout the project lifecycle vs only summative (at the end of the project).  How is complexity thinking related to program cycle adaptive management? EEN should consider evaluating for program co-benefits in the context of climate adaptation and biodiversity …the benefits that we are trying to optimize for.  The theme should consider tools and techniques for addressing complexity vs. complexity as standalone issue.

vii.    Steve – I’m new to EEN and I work in climate change and view the world from that perspective.  10 years ago I was working with indicators at EPA, and now issues of climate change present fundamental challenges to environmental law (CWA, CAA etc). The majority of laws suggest the need to restore the environmental to some previous condition but now climate change is beginning to impact current conditions so the conceptual basis is different for existing programs. Co-benefits is a crucial issue to deal with in considering existing environmental policy.

viii.    Ana – I’ve only attended one EENF.  EEN will need a workshop on Complexity 101 so that people are not overwhelmed and to cover folks that are new to the subject.  To help with this we need case studies and examples of where complexity was considered and where it was not.

ix.    Hans – I think ‘complexity’ can be a good theme.  If we start at assumption of complexity then what kind of new knowledge do evaluators need in order to deal with it?  What kind of new challenge does complexity pose to the policymaker?  We are talking and thinking of biodiversity and climate change but we must consider interactions between social and natural systems and more complex social systems and their rules, norms or governing principles and the actors that have an impact on environmental logic.  As a social scientist I want to link societal complexity to environmental complexity. Social scientists don’t model as often as in the sense we’ve discussed so far [biologist] but they still grapple with complexity.  Christina talked about uncertainty and in the context of uncertainty, evaluators must consider what to do and how to phrase recommendations and conclusions.

x.    Richard – Seems to be lots of convergence in this conversation.  I suggest the title “navigating complexity” for the meeting – its more active.  We I love about EEN is that it has done a good job of framing issues each year and giving participants the latest and greatest of whatever is pertinent to that issue so that participants can use the tools, ideas, etc that are presented that they can take them back home and use them.  Most people are at EEN events to learn and do and not just talk theory.  We could divide the conference to conceptualizing complexity, complexity in design, managing complexity and then the whole M&E side of complexity.  Suggest, splitting the Forum into four categories 1. Conceptualization 2. Design 3. Management 4. Measurement and Evaluation

xi.    Sacheen – The theme is well timed for us given what folks are talking about in climate change and adaptation, coastal marine and spatial planning, which are really complex issues that lend themselves to that theme.  We’ve been struggling to measure those issues.

xii.    Christina – I think it’s a great topic and not just because it was my dissertation topic.  There are two types of complexity to discuss: Detail complexity as with engineering problem where there are many variables concerning a well defined problem…but our discussion must go beyond this, to dynamic complexity, where there are still many variables that interact but also a less well understood problem involving multiple parties with differing values and interpretations of the issue/problem with high uncertainty and incomplete information.  High uncertainty must be addressed and how would you deal with it and how do you address non linear change and how do you look at feedback loops in environmental systems and how do you define a system at all.  We need to make sure we get something useful at the end of the event…must be practical. For instance, you can have fire suppression that causes fires in future with long time delays but how do we deal with these feedback loops and long delays in evaluating our programs.  We must apply our discussion to specific programs and show how to deal with complexity in a variety of contexts.

xiii.    Lavanna – Strategic management should be a focus.  We are seeing programs that are not performing but we are not looking at the evaluation strategy as a whole.  AEA recently distributed information on evaluation strategy. EEN should host a workshop that looks at evaluation and strategic management as a whole.

xiv.    Scott Bowles – co-benefits is important issue…we need to hear more from Richard G. about this. How can we make what we learn as participants in the EEN implementable and build the network from that practical learning?  When talk about history of this event, what would we like to see happen as result of this event?  Spin off groups?  Would there be something rolling from this year into next year that we could track?…but do we have the capacity to do this?

xv.    Katherine – EEN can track itself better now than before with social media to understand the network and where it’s going.

xvi.    Kim Damm – EEN is a place incubating ideas in communities of practice.  Would like to consider evaluation of renewable energy and use of evaluation in the legislative process…including evaluation in the appropriations process.

1.    EEN Program and communication materials…As in the performing arts we should include photo and bio in the program, etc.

2.    EEN subgroup. Can we start a student’s, young professionals, network group with EEN? I would be happy to kick this off.  Great to talk to other students…I feel more comfortable asking some questions.

a.    Improving capacity for evaluation in the environmental sector (20 min)

i.    Considering a theme of ‘Complexity’, what skills, knowledge, tools, research, experience, etc. are (or will be) most critical for improving the capacity of EEN participants to advance the practice, theory and policy of measurement and evaluation?

ii.    How should these be discussed or presented at the Forum?

iii.    Jonny – Knowledge use and knowledge transfer is a field with a lot to offer to the EEN.  Try for a panel or workshop on research on these topics

iv.    Nick – Keep the categories open like Richard M. said and put out the 4 categories and let them fill through the call for proposals.  I tend to think about the more rigorous evaluations so distinguishing between complex and not would be helpful.  Some Performance measurement would not be complex but others may be very with modeling and such.

v.    Richard G. – Learning about process and distributional equity questions.  Also advancing practice in performance management.  Build organizational capacity to manage measures for organizational efficiency and effectiveness.

vi.    Mary – I like the knowledge transfer issue for capacity building discussions and sessions.  Presentations in that arena should talk about dissemination and diffusion and uptake of information and how to effect program management and policy development

vii.    Ana – Workshop on performance metrics in the context of complexity would be very interesting in terms of capacity building for the event

viii.    Mya – Performance measurement is a crucial topic to cover.  Lack of data is often the issue for us.

ix.    Hans – interested in workshop on non-linear evaluation methods that can be used under high uncertainty conditions

x.    Richard M. – I like the idea of picking a theme and adding on capacity building sessions that are not too engineered so that it works based on what people have to bring to the EEN.  Continue to alllow  EEN to self organize via the call for proposals.  Consider my 4 sections for the EEN (conceptualize, Design, Management, M&E)

xi.    Christina – We need to work with programs on performance measurement…this is the biggest issue for us at NFWF.  Without this we are not going to be able to learn what we’ve done in the past and everyone will continue to think that their project is unique without common metrics in other programs to learn from.  We need more common standards and metrics because without this the environmental sector is hampered.  Not just performance measurement for environmental changes but also for capacity building, enforcement, prevention, and more broadly how to develop good Performance metrics.  Great need in the field.  Should discuss how have people done this well in the past.

xii.    LaVanna – We should invite key leaders and/or evaluation staff from OMB, various Federal Inspector General offices, GAO and even Congressional committee staff that oversee the U.S. environmental agencies to the 2011 Forum.  We would love to have them sit side by side and participate in a panel discussion about how the U.S. Federal government should be linking these internal and external evaluations/audits to determine whether programs are performing as intended and how we can reduce duplication of efforts in both evaluation and program delivery.  Apparently some of these organizations have participated in AEA conferences in the past and this has been very useful for Federal evaluators in attendance. (We need better direction and coordination and it would be wonderful if our funders and program examiners understood the challenges we face). There also seems to be very little transparency about how OMB is managing its evaluation policy implementation, including disbursing extra funds to Federal agencies that apply for them to conduct “rigorous program evaluations.”

(20 min)

i.    What do you want to achieve through networking at the EEN Forum?

ii.    How would you like to network with EEN participants?

iii.    Mary – Through EEN there has been some informal communities of practice around environmental justice issues but we need more networking effort with communities of practice in this context.

iv.    Richard G. – I like the speed dating kind of networking where we are moving around and choosing who to interact with – its been fun and engaging so continue this kind of networking and interaction.  Try to use the 4 phases that Richard laid out for mixing and interacting.  Agree with Hans about the need to focus on enthusiasm and creativity.  I have sought but haven’t gained at EEN the emergence of relevant communities of practice….need to incubate communities of practice in EEN and to explore their capacities and approaches around nascent topics.  Even if topic isn’t ‘workshop level’ or enormous interest but it needs to be discussed…examples are perhaps climate adaptation and equity etc.

v.    Ana – Evaluators Café was very good from last year’s event.  Through networking, I want resources in terms of evaluation expertise (evaluation professionals) that we can bring to our projects and programs to do the work we need done.  All of our programs need expertise and we are new to this so we need access to the expertise.

vi.    Hans – I like networking opportunities that stimulate creativity and enthusiasm. I like formats that surprise me and stimulate me.

vii.    Richard – open bar at 10am per Matt’s suggestion

viii.    Christina – last few years have been really good at networking in different ways, Evaluators Café is good.  Want to meet new people and I want to have time to ask folks specific questions that I’ve been wanting to address in my work…I need some time for this type of small group interaction and discussion over particular problems.

(5 min)

a.    Continued feedback– Next call will be in about one month.  Matt will send out invitation via Doodle to determine best time & date.

b.    Call for proposals, speakers (Call proposals will not be limited to traditional 15-20 minute PowerPoint presentation and Q &A; EEN is seeking thought leaders and creative minds)

c.    Promotion—AG members are “hubs” within their organizations and related networks and should leverage these relationships and associations to attract speakers and participants, if not also other ideas for structuring the Forum

More on Complexity from Christina Kakoyannis

Two different types of complexity—detail complexity and dynamic complexity—can be found in natural resource problems (Senge 1990). Detail complexity is associated with the existence of many variables, such as in engineering problems. Problems of detail complexity are characterized by a clear, agreed-upon problem, low uncertainty, and stakeholders whose interests are aligned. For these types of well-defined problems, scientific or technical solutions are often effectively used.

In contrast, problems of dynamic complexity—the focus of this study—are more difficult to manage. Different terms have been used to describe complex problems of this type:  dynamic complexity (Senge 1990), wicked problems (Rittel and Webber 1973, Allen and Gould 1986), and organized complexity (Weaver 1948). Similar to problems of detail complexity, problems of dynamic complexity involve many variables; however, they also contain other attributes which make them particularly challenging (Table 1).

A defining characteristic of dynamically complex problems is that the problem itself is ambiguous and open to question (Weick 1985, Salwasser 2002); therefore, problem framing becomes extremely difficult, but especially critical. How one formulates or frames a problem influences the search for, and the range of, solutions one has to address it (Rittel and Webber 1973). In part because problem framing is so ambiguous, dynamically complex problems are those in which no correct answer exists, though solutions differ in their utility for managing the problem (Allen and Gould 1986, Rittel and Webber 1973). In reality, the term “solutions” is inappropriate for use in discussing issues of dynamic complexity. “We do not really ‘solve’ wicked problems, rather, we ‘design’ more or less effective solutions based on how we define the problem” (Pacanowsky 1995: 37).

Another feature of dynamically complex problems is that many interested parties with differing values compete for scarce resources (Lang 1990, Weick 1985). The parties’ differing value orientations lead them to hold multiple, often conflicting, interpretations of both the problem and scientific data (Weick 1985). Thus, scientific data have less utility for addressing problems of dynamic complexity than for addressing problems of detail complexity. The fact that differing parties do not agree on interpretations of scientific evidence, coupled with the fact that organizations have incomplete or inconsistent scientific information, creates high uncertainty (Lang 1990). Furthermore, complex problems contain unclear cause and effect relationships because they have multiple, dynamic, and non-linear interactions and feedback loops (Cilliers 1998, Senge 1990). Because one action can have different spatial and temporal effects, developing appropriate management strategies can be difficult (Senge 1990).

Many natural resource issues, such as habitat fragmentation or reductions in water quality, are dynamically complex. Particularly due to its public context, natural resource management and policy issues typically include multiple parties and multiple issues (Faure and Rubin 1993). Furthermore, these multiple stakeholders hold differing values and risk tolerances (Lang 1990, Daniels and Walker 2001). With continually increasing demands placed on natural resources and the rapidity of changes occurring in the environment, the complexity of these resource conflicts and policies is likely to grow. Our ability to satisfactorily manage natural resources will depend upon using approaches that take into consideration the type of complexity associated with resource issues.