North American Test of Criteria and Indicators of Sustainable Forestry

Executive Summary

This report presents an independent review of various sets of criteria and indicators of sustainable forestry. The review was conducted under the auspices of The Center for International Forestry Research (CIFOR) by the USDA Forest Service Inventorying and Monitoring Institute. This review constitutes CIFOR’s North American test of sustainable forest management, which is the seventh worldwide CIFOR test. The test, hosted by the Boise National Forest, was conducted in southwest, Idaho, USA.

The test is part of a larger CIFOR project to develop sets of locally appropriate criteria and indicators (C&I) at the forest management unit (FMU) level. C&I are tools that can be used to conceptualize, evaluate, and implement sustainable forest management. The principal aim of C&I field-testing is to identify C&I that are objective, cost-effective, and relevant to the sustainable management of forests. The focus of the testing procedure is to identify the smallest number of C&I needed to reliably assess forest management in a cost-effective manner. The CIFOR tests are unique in testing the application of C&I to the field level, where key decisions are made.

The Boise Study Area represents a forest management unit (FMU) a sophisticated level of forest management. Most landholders maintain comprehensive resource management plans generally aimed towards long-term productivity and ecological health of the forest. The area also has a comprehensive database and a high level of stakeholder involvement. The forest represents a valued resource for a wide range of users, supplying local peoples with revenue from timber products, outdoor recreational opportunities, fuelwood, and other forest products. The area also serves as a refuge for many animals and plants, and protects ecosystems and natural processes, which may be declining, on adjacent lands.

The lead agency for the evaluation was the United States Department of Agriculture’s Forest Service (USDA – Forest Service). The Boise National Forest was at the heart of the study area. Other key cooperating land management agencies were the Boise Cascade Corporation and the Idaho Department of Lands. The Project team, selected from a wide range of disciplines from throughout the United States, Canada, and Mexico, consisted of three ecologists, one social scientist, one economist, three forest managers, and one forest geneticist. Additional specialists included a carbon biochemist, an anthropologist, a systems ecologist, and a forest ecologist.

 Which Sets of Indicators Were Tested

The sets of C&I selected for evaluation during the North American test included: 1) those that emerged from the CIFOR Phase I synthesis; 2) CIFOR’s basic assessment guide for human well-being; 3) Canadian Council of Forest Ministers (CCFM) Criteria and Indicators of Sustainable Forest management in Canada (which are similar, but not the same as, the Montreal Process - see following paragraph); 4) local/regional indicators including the Idaho Forest Practices Act; and, 5) the Greater Fundy Ecosystem Guidelines developed for the Fundy Model Forest (see Appendix B for a detailed listing of all indicator sets tested).

This project considered testing C&I that had come directly from the Montreal Process. These C&I were developed following the 1992 United Nations Conference on Environment and Development (UNCED), where there was international agreement to formulate guidelines or criteria to ensure sustainable forest management. One of the groups to take up this challenge was the Working Group on Criteria and Indicators for the Conservation and Sustainable Management of Temperate and Boreal Forests ("or the Montreal Process" Working Group) which was formed in Geneva in June 1994. It developed from the work of an International Seminar of Experts on Sustainable Development of Boreal and Temperate Forests, held in Montreal, Canada, in September 1993. The Montreal Process Working Group membership now includes 12 countries covering over 90 per cent of the world's temperate and boreal forests, including Australia, Canada, Chile, China, Japan, the Republic of Korea, Mexico, New Zealand, the Russian Federation, the United States of America and Uruguay.

In Canada, The Canadian Council of Forest Ministers had an ongoing project to develop C&I (1992) which was presented at Rio. In 1993, following UNCED, the Montreal process began with the Canadian Government initiative. The CCFM project predates the Montreal Process and the indicator sets are not identical. However they are substantially similar, were further developed and thus more testable. For this test, therefore, we used the CCFM indicator set. The CCFM has six criteria, 22 elements and 83 indicators while the Montreal Process has seven criteria and 67 indicators. Any Montreal Process indicators not covered by the CCFM set, were generally covered by the significant overlap with the CIFOR Phase 1 synthesis set.


Methods generally followed those prescribed by CIFOR (1996) with modifications to fit the site and North American context. Field testing of criteria and indicators (C&I) involved three phases, conceived as four separate filters. The filters used evaluation parameters to examine each C&I. At each stage a particular C&I could be rejected, merged, or passed to more detailed evaluation in the next filter. These filters were conceived to be more than a simple, staged sifting process. The three stages explicitly allowed creative inputs and modifications to C&I, which were also subjected to the evaluation process.

Stage 1, essentially a desk exercise, was a preliminary evaluation of all C&I in the initial sets selected. First each team member was asked to read a comprehensive set of information on the study site, as well as local planning and evaluation documents. Then each criterion and indicator was numerically ranked against a set of parameters. Each parameter was given a score of 1-5 and general comments were given. The results from stage 1 were tabulated and averaged in a spreadsheet (Form 1, Appendix E) and made available to the team at the start of the fieldwork stage.

Stage 2, the Initial Fieldwork stage, consisted of a detailed three-day orientation workshop on local social, economic, and ecological conditions, as well as a review of a summary of data available for the test. Following the orientation workshop, the team worked in sub-groups and discussed the tabulated results from Form 1. After debate, individual criterion and indicators were rejected, merged or allowed to go to the next, more detailed evaluation.

Stage 3 was a detailed, field evaluation of the each of the remaining criterion and indicators. We worked with reference material, other experts, and in discussion groups, to critique or refine the theoretical basis for each criterion or indicator. For each indicator, we also attempted to use data from the Boise Study Area to assess its practicality. Wherever possible, we talked to local resource people to get their views on the value of the indicator. For each indicator tested, team members filled out assessment forms (see Form 2, Appendix F). The detailed test results are presented in Volume II of this report.

Stage 4, the post-fieldwork stage, consisted of a two-day workshop where 60 new participants were drawn from different institutional and disciplinary backgrounds to critique the results of stages 1-3. This final workshop was held in Boise, Idaho, in the fall of 1998.


In total, the group tested 207 indicators in detail and scanned another 200. We accepted, or accepted with revision, 71 of the original 207 C&I tested. In most of these cases we suggested changes. We rejected 65 of the 207 because they were conceptually weak, impossible to use operationally, or irrelevant to the North American context. We also proposed 5 new indicators.

Breaking down the results by indicator set, the Canadian Council of Forest Ministers (CCFM) set of criteria and indicators had 62 of 104 accepted. The remainder was rejected because they were conceptually weak, not well defined, or not relevant at the forest management unit level. Some of these latter indicators had to do with carbon issues and contributions to global warming.

In general, the team felt that the use of criteria and indicators is still in the development phase: there remain considerable difficulties to their use. Some of the indicators are very well developed, while others remain weak. There were also difficulties integrating indicators across disciplines. Despite our best efforts to work in an interdisciplinary manner, the team still tended to work on a sectoral basis. There is still a lack of a basic theory of sustainability that integrates across sectoral lines.

 Ecological C&I have probably received the most attention in the development of indicator sets. Some good ecological indicators are being used throughout North America. The main hindrances to further development of ecological indicators appear to be: i) limitations in theoretical understanding of how ecosystems function, and ii) how to practically measure complex ecological variables in the long term. On the practical side, measuring ecological parameters can be costly and time consuming. Even tracking the changes in population for one species, if done with reasonable confidence limits, is extremely costly. Very few agencies have the staff, expertise, and financial resources to do this for the long term. Most long-term monitoring programs fail because programs are cut during periods of budget constriction. Of the C&I sets reviewed, the CIFOR ecological indicators tended to be geared to tropical forests. In practical terms, many of the indicators and even criteria were not applicable to temperate North American forests. An example would be to "minimize gaps in the forest". Such an indicator might be applicable to tropical systems and even mesic or wet forests in North America. In the dry pine forests of western North America, however, the objective is often to open the forest canopy.

A major problem in testing the indicator sets is that they often had no supporting or explanatory material to support the concept. The CCFM material was generally more clear and applicable than the CIFOR material. However the theoretical rationale for indicator selection was often very brief or absent. For example, under "Extant Biomass" in the CCFM indicator set, the only indicator provided was mean annual increment. This might be a good measure of the growth rate of selected trees, but is a very limited measure of biomass as it does not include other biota. Often it seemed that the measurement side of both CCFM and CIFOR relied too heavily on available data, even if the date were only vaguely relevant to the criterion or indicator. While the team understands the needs for practicality in data use, it seems to undermine the indicator concept if available data are stretched to fit.

One subset of ecological C&I was related to carbon and global climate change. These C&I were difficult to assess at the field management unit because they were generally written to apply at the national level. There is a link between all forests, no matter how small, and the global levels of CO2 that should be recognized; however, the C&I tested do not reflect the critical aspect of forest management that will mitigate the rise in CO2 levels.

Economic C&I - The CIFOR and CIFOR-BAG C&I included no overt economic indicators other than some equity considerations in the CIFOR-BAG group. A CIFOR working paper by Ruitenbeek and Cartier (1998) does address C&I from an economic perspective and was used as a source document for the test, although their suggested C&I were not actually part of the test set.

The Ruitenbeek and Cartier discussion offers some strong proposals for C&I, especially from the economic perspective, although it is weighted towards tropical forestry situations. One particularly useful theme is that it may be easier to find the negation of an indicator rather than the affirmation (e.g., inequality rather than equality; unfairness rather than fairness). The discussion also provides a useful stratification of issues regarding core themes of efficiency, equity and sustainability. It also offers insights into the importance of policy related to sustainability and possible policy intervention points. Unfortunately, from the perspective of the test, the paper actually provides an alternate set of C&I, reconfigured according to a pervasive economic perspective. This made it difficult in some ways to extricate primarily economic C&I that could be added to the CIFOR set for evaluation. Further, these C&I were largely organized using general sustainability principles rather than those largely focusing on forestry.

The CCFM C&I related to economics were generally rather limited in scale. Nearly all C&I related to efficiency were focused on national economic parameters (e.g., productivity, capacity, non-market benefits, and contributions to GNP) and as a result were difficult to apply at the FMU level. Further, several of the other economic indicators, while measurable and at an appropriate scale, have non-obvious links to sustainability (e.g., "index of the diversity of the industrial base"). That is, the linkages may be based on second-order hypotheses (e.g., economic diversity promotes economic sustainability) which are not stated and may be untested.

The GFE/Idaho C&I do not include any economic indicators.

The team felt that economic C&I still required further development. The set tested was quite limited in that it was primarily diagnostic and focused on economic structure with few dynamic aspects. The relationships to sustainability are mostly second-order. Further, there was the fundamental difficulty of trying to incorporate the sustainability of economic/social systems into the realm of forest sustainability. Is sustainable forestry a sufficient condition for economic and social sustainability? Human systems are highly dynamic and adaptable. While forest ecosystems are fundamental to meeting human needs, are they the vehicles for assuring economic and social sustainability? Finally, the economic indicators should encompass more of the dynamics of economic processes (changes in system variables over time), since it seems that sustainability is inherently a dynamic concept.

It appears that the principles of sustainability that were applied in the test (i.e. Ecological integrity is maintained, Yield and quality of forest goods and services are sustainable, and Society accepts responsibility for sustainability) might better have been organized or simply evaluated by using principles from the general literature on sustainability. For example, it might have been more useful to have categorized the C&I according to the following principles:

Management C&I sets that were tested focused on areas of the ecosystem that are actively managed for forestry. This becomes both a strength and weakness. Due to the long history of forest management in the US and other countries, management and concepts are generally based on analysis. Forest resource data are generally readily available to support measurement or assessment of management indicators. Forest inventory data have been collected and organised for decades, dating back to 1920's for portions of the Boise Study Area. A wealth of forest inventory data exists for the last five decades. The richness of information on timber resources often leads to its use as a surrogate measure for other resource values. The management indicators tested are reasonably good measures of "good timber management" or management aimed at the most judicious use of the available resource, providing the best allocation for co-ordinated resource management. This notion is a remnant of the multiple-use philosophy that dominate natural resource management in the 1960's, 70's and 80's.

The downfall of the existing management indicators as a measure of overall sustainability is the fact that the indicators are only aimed at acres of areas included in the forest management program, and tend to be focused only on the balanced management of trees. These measures are relevant to overall sustainability of ecosystem conditions only to the extent that other resource values can be correlated with measures of the forest resource. In the North American test, the concepts of forest management theory were applied to approximately 25% of the total area because only 25% of the test area is available for forest harvest. This is problematic for a set of indicators intended to provide insight about the whole area.

Suggestion for improvement of the management indicator set would be to develop measures or indicators that can be appropriately applied to the whole study area or an entire ecosystem. This would undoubtedly require a loosening of the definition of management to include activities such as no management, terrestrial and aquatic restoration, non-traditional forest removals, planning for ecological functions, and others.

Additional suggestion for improvement of the management C&I were developed at the Boise workshop. These include the following:

1. More than one-half of all forests in the world are owned by Non-industrial private forest owners (NIPF). These lands make a significant contribution to all aspects of sustainable forestry. The Boise test was not able to incorporate evaluation of management of NIPF lands. In the future, efforts need to be made to account for the contributions of these lands to sustainability.

2. C&I need to be developed to measure the sustainable management of non-traditional forest products.

3. The existence of a management plan does not guarantee sustainability. The plan has to be implemented and be effective in achieving desired future conditions. Following the Boise workshop, efforts were made to develop and align existing indicators to meet this need. However, more work is needed to make this C&I set more robust.

The Social C&I tested came from two sources, CIFOR-BAG and CCFM. CIFOR-BAG indicators have been designed to cover a range of social issues, but the developed nation/developing nation contrast between the North American test and previous tests resulted in significant incompatibility in applying the indicators. Specifically, the CIFOR-BAG indicators were written from a context where the social systems were more firmly imbedded within the forest. This might be best described as the difference between forest-dwelling or forest dependent people and people who live in a forested area. The connections in the North American context are generally (although not always in Mexico or in some Aboriginal communities) less tightly related, which means that some indicators must be interpreted in a different way.

In the North American context, particularly in Canada and the U.S., there is an extensive legal and constitutional structure, that protects many of the property and treaty rights that are more variable in developing countries. The result is that the indicators are relatively easy to monitor and assess, but not useful to measure people's satisfaction with these legal/ constitutional structures.

An extensive set of CIFOR-BAG methods was developed for testing in other locations. These methods were undoubtedly particularly useful where no data sources exist on the social systems. These methods are almost uniformly anthropological in origin and are not designed to take advantage of the existing data sources in North America. These anthropological techniques may be most useful in relatively small test areas or as initial means of scoping or refining methods. We found these techniques to be less useful in an area that was relatively large and where we wanted to be able to have generalizable results. In a context where data validity and reliability are hotly contested, techniques that acquired perspectives from a wide range of sources in a replicable fashion were more desirable. Part of this is the adaptation of the techniques to the North American context and part is undoubtedly the difference associated with the background of the social scientists involved in the North American test.

The CCFM indicators were written from the North American context, and consequently were already adapted to address some of the more pertinent specific topics. Specific comments regarding this set are largely focused on the extent to which this set of C&I were developed. In general, CCFM indicators were poorly detailed. Data sources were suggested, but only occasionally were methods discussed. Finally, no means of scaling these indicators were discussed. The CCFM indicators were designed for a national level scale. As a result, some indicators were not relevant at the study area level, or data sources and methods of data collection were not specific enough to be useful.


Clearly, it is important to better understand the status of our relationship with the ecosystems that sustain us. For all the primary resource industries -- agriculture, fishing and forestry -- the idea of living sustainably with the land seems obvious. It would seem to be a simple task to assess if we are living sustainably or not. In practice, however, assessing "sustainability" is confounded by a host of difficulties. Some of these are scientific, pointing to flaws in our basic understanding of systems. However, scientific issues seem minor compared to the problems that arise from different perspectives on the nature of sustainability.

We ask a great deal from the definition of sustainability. We want "healthy ecosystems" rich in native biodiversity, "equitable" social systems, and a continual flow of goods and services for humans. When we look for examples of sustainable human systems we can find none that meet all these criteria. Certainly, history shows us examples of human societies living in the same way on the same place for hundreds or even thousands of years. But the team knows of no examples where this was done without inequity or major ecological impacts.

We continue to have difficulty with the concept of sustainability. Some view it as a way to limit development, others as a hoop to jump through to ensure development can occur. Both views operate from a perspective of minimum. In one view it is minimum development, from the other it is minimum level of interference with development. The team felt that sustainability will only be helpful when we look for optimum arrangements of the ecological, economic, and social values. As long as we stay in the argument of the minimums, we have not changed the nature of the debate, only changed the words we use to argue. If we fail to look for the best allocation or set of conditions through time, we are likely not to find it.

In addition to the basic conceptual problems with defining sustainability, the sets of criteria and indicators we tested to assess sustainability of managing forested ecosystems all have major problems. The problems are listed below:

Despite these criticisms, the team felt that criteria and indicators can fill a critical role in assessing sustainability. There are many excellent ideas in the sets of C&I we tested. We accepted the majority of indicators as providing valuable understanding on the sustainability of actions in the forest. However there is a long way to go to get these ideas operating and accepted at the field management unit. It is time to move the debate over C&I from national policy forums to the field management unit. At their heart, C&I are practical applications of knowledge. We must remember to focus on their practicality. Otherwise we will ignore many pressing and real problems while we "get the science right".