USDA FOREST SERVICE : Strategic Planning and Resource Assessment

Management's Discussion and Analysis

Validation, Verification, and Limitations of Data Sources

In the previous section on Performance Highlights, the variable ‘Data Source’ is referenced in Exhibit 3: Data Sources with Actual and Estimated Performance.

The following discussion gives readers a description of the quantitative data sources listed, and follows with a brief discussion on a few qualitative measures of performance. Data sources that are not complete or reliable are for management of invasive species and accessibility in Forest Service programs and facilities.

Management of invasive species is significantly limited by the lack of comprehensive and accurate inventories of infestations and treatment efficacy. This impedes the agency’s ability to track this work accurately. Most invasive species data that has been collected is for invasive plants, yet even that data is incomplete and variable. For fiscal year-end, the field is required to report treatments for invasive plants (noxious weeds) directly to the Washington Office, but does not have any means by which to transfer this information electronically.

In fact, overall data management, including geospatial capabilities, for the inventory, treatment, and monitoring of invasive species are not fully operational, and in some cases, do not exist at all.

The Forest Service expects to have implemented a corporate database to manage invasive species information late in FY 2004.

Information concerning the accessibility of programs and facilities is not currently integrated into any corporate database, impeding the ability to report results when needed.

Quantitative Measures of Performance

Management Attainment Reporting

Performance numbers shown with a data source indicator of ‘MAR’ are collected in the Management Attainment Reporting database. MAR data is compiled at the district and forest levels and then reviewed by regional and national offices for accuracy.

The NFS roads performance data is a national summary of what each region accomplishes at the forest level. At the forest level, data is collected by road program managers and verified by budget personnel. The forest data is then reviewed at the regional and Washington Office levels for accuracy.

Individual forests and grasslands record boundary management accomplishments in their respective Corner Status Atlas, in conformance with direction provided in the Surveying Manual (FSM 7150). These accomplishments are physically marked on hard copy maps and then reported in the MAR system by each region for national reporting. Boundary management accomplishments will soon be electronically tracked in the Automated Lands Program (ALP) database.

Forest products activities and their outputs are presumed to be within sustainable limits because the levels of most outputs today are significantly less than the historical levels. If the Forest Service is to achieve “products and services…for subsistence, commercial, and noncommercial uses within sustainable limits,” the agency must establish how sustainability will be defined and measured. Processes designed to assess sustainability are under development, but in the meantime, periodic assessments of inventory and monitoring data must serve as indicators of sustainability.

Environmental Compliance and Protection/Abandoned Mine Lands

The Environmental Compliance and Protection (ECAP) program provides for the cleanup of hazardous substances on national forests and grasslands to improve and protect watershed conditions and human and ecological health. Regional program managers itemize ECAP and abandoned mine lands (AML) work plans, progress, accomplishments, and financial data on a project-by-project basis.

Sales Tracking and Reporting System and Timber Sale Accounting

Performance information for timber sale volumes offered for sale is entered by field personnel into the Sales Tracking and Reporting System (STARS), from which accomplishment reports are run. Performance information for timber sale ‘sold and harvest’ information for each sale is recorded on form 2400-17, and regularly entered into the Timber Sale Accounting (TSA) system. These processes are managed in conformance with the direction provided in the Timber Management Information System Handbook (FSH 2409.14), Chapter 30, Timber Sale Information, and Chapter 40, Timber Harvest Information, as well as the Timber Sale Accounting Handbook (FSH 6509.17).

Timber Information Manager

The Timber Information Manager (TIM) system is a family of integrated applications that help to support the daily business needs of the NFS lands. TIM currently uses STARS (a legacy system) to complete the necessary functionality.

TIM is used to automate business functions such as—

  • Sale of Special Forest Product Permits
  • Timber Sale Contract Preparation and Administration
  • Trust Fund Management
  • Activity Tracking


The Forest Service uses INFRA as an integrated data management tool to manage and report accurate information and associated financial data on the inventory of constructed features, such as buildings, dams, bridges, water systems, roads, trails, developed recreation sites, range improvements, administrative sites, heritage sites, general forest areas, and wilderness.

Forest Service tracks NEPA analysis and recent decisions using INFRA’s Range Module. This Range Module, which is used on all national forests with a livestock grazing program, contains current information for nearly all grazing allotments on NFS lands.

Database queries are used to report the number of allotment acres to standard. The Forest Service’s Wilderness Program continued its national upward reporting exercise using INFRA-WILD, first conducted in FY 2002. INFRA-WILD data is used for program management and public information dissemination and forms the basis for the State of the Wilderness Report, currently under development.

The majority of the roads, trails, and facilities data is obtained through the Forest Service INFRA corporate database system. INFRA data entry is at the field level; therefore, there are limitations or inconsistencies in the data’s accuracy. Currently, the only active process for data verification and validation is in condition surveys throughout the year. The facilities program began reviewing existing data for accuracy and will edit the data as needed. These surveys provide a look at the progress of the performance measures.

Although trail data is currently incomplete, the Forest Service expects the INFRA trails module, complemented by cost information and assessment and condition survey from the trails Assessment and Condition Survey to provide complete trail information by local, regional, and national levels, as well as by State and political divisions.

The facilities program uses the facilities condition index in INFRA as a measure to assess the condition of every building.

Special Uses Database System

The INFRA Special Uses Database System (SUDS) is a corporate database that integrates several systems—land use records, accounting records, Geographic Information System (GIS), resource data, and administrative information. As a repository for information on Lands Special Use Authorizations (SUA), SUDS data is entered at the field level, including documentation of completed inspections, land use fees billed, and status of the term of the authorization. The accuracy of SUA data is dependent, in part, on whether inspections are documented in SUDS. In FY 2004, SUDS will be modified to more consistently collect data for administration to standard, automatically capturing the specific data used to measure completed authorizations that are administered to standards. The variation between 2003 Planned and Projected 2003 Performance columns in the 2003 Planned and Actual Performance table (in the previous section) is primarily due to confusion in interpreting the definition of to standard, as well as inconsistently applying the standards across all field units.

Land ownership case information is entered on a Proposed Exchange form (FS-5400-10) or proposed Purchase Sheets (FS-5400-9) at the field level in conformance with direction provided in the Land Acquisition Handbook (FSH 5409.13). Acquired acreage that is reported on the digest sheets is then entered into the MAR system by each unit for national reporting.

Title management information is reported in several formats. Small Tract Act case information is reported through Form 5500-3, Small Tract Act Parcels Report; land status information is reported through the Land Areas Report and also in the ALP system; and title claims are reported through the litigation process or through administrative procedures. These reporting requirements have been in place for several years and provide an accurate and reliable measurement of the annual accomplishments and the agency’s progress in resolving access issues.

Performance Measures Accountability System

The Performance Measures Accountability System (PMAS) is the S&PF Deputy Area performance management system. It ties measurements to the corresponding corporate performance resources committed to.

This is used to meet reporting requirements of GPRA, for year-end accomplishment reporting of the Forest Service, to meet legal requirements (i.e., Title VI), and in accordance with Forest Service Handbooks (FSHs) and Forest Service Manual. Data is captured from the States, forests, and other cooperators to generate these reports.

National Fire Plan Operations and Reporting

The National Fire Plan Operations and Reporting System (NFPORS) is an interagency system mandated by Congress. While the Forest Service is the primary owner of this system, it is a partner in this project with the Department of the Interior’s Bureau of Indian Affairs, Bureau of Land Management, U.S. Fish and Wildlife Service, and the National Park Service.

Economic Action Programs–Program Management Tool

The Economic Action Programs—Program Management Tool (EAP-PMT) is used to report data relative to the NFP and other regulations. EAP work includes forest products conservation and recycling. A separate database is used to maintain wood in transportation data.

Accomplishments are entered based on projects that have officially closed for the fiscal year.

Wildlife, Fish, and Rare Plant Management System

The Wildlife, Fish, and Rare Plant Management System (WFRP) gives users the opportunity to enter projects and opportunities and to view what has been accomplished on a project. (Project in this case is defined as a management activity that is designed to meet specific resource objectives.) The system tracks accomplishments, funds, and work used to complete the project.

Qualitative Measures of Performance

The complex and unstructured processes found in R&D are not easily quantified. In the physical sciences, measurement such as length, temperature, and mass may be measured using single standard units—the adequacy of each measurement depends on the qualities of the instrument, but the standards are well defined and widely accepted.

In contrast, the creative aspects of R&D make direct measurement impossible. The dilemma is balancing objectivity with the subjective selection and interpretation of measurement indicators, recognizing the cognitive and social structure of science. This is not only true for the Forest Service, but for scientists in all Federal agencies.

Three dimensions of R&D—concept generation, product development, and leadership–are distinct phenomena with unique characteristics within the innovative process of research. These dimensions are not amenable to forced correlations and patterns, which can result in comparing “apples and oranges.”

Alternatively, indicators may be used for certain aspects. The degree to which such indicators “measure” R&D performance depends on their accuracy, their quantity, and whether any one indicator may be aggregated with others for indexing. Empirically, this means one measure will be inherently insufficient to capture all the information required.

The current single measure of R&D performance—number of products, technologies, and tools produced —has a reasonably high bias for accuracy, precision, and repeatability, but has variable tolerance and sensitivity. A more plausible approach would be to use a set of performance measures that can be linked to outputs. A systematic design and understanding of the process by which R&D impacts agency performance, and to which the agency remains committed to working with users and the scientific community, will allow the Forest Service to identify and define meaningful performance measures for the future.