Home | Projects | Mapping | Data & AA's | Forest AA Data or Zone AA Map | AA Interpretation

Appendix A. How to Interpret the Accuracy Assessment Tables

Essential Background Information

First, it is important to understand that the accuracy assessment numbers are based on the assignment of fuzzy ratings. Fuzzy ratings are values that rank the level of correctness that all possible map labels have relative to the data collected at each reference or 'ground truth' site. Comparing labels assigned to the reference sites with the geographically related map labels is the basis of the accuracy assessment process. Accuracy assessment sites are initially assigned covertype, vegetation type, tree size, and tree crown closure labels based on the raw data collected in the field and the classification keys for each of those attributes. This ensures an objective assignment of labels to the reference sites using the same logic that was used for developing the map, allowing for an 'apples to apples' comparison.

In addition to a label based on the plot data, a specific reference site will also have fuzzy ratings assigned to all possible cover type, CalVeg type, tree size, and crown closure labels. The logic that determines the assignment of fuzzy ratings for each reference label/map label combination is based on deviation from the class parameters defined in the classification keys. Cover type and CalVeg type fuzzy ratings are determined by the percent composition of life forms and species on a reference site. The example below illustrates the required percentage of species composition for hardwood CalVeg types. Similar rules apply to conifer and shrub types. Fuzzy ratings assigned to the structure attributes of crown closure and over-story tree size are based on the deviation from a defined class measured as a percentage of class width. The following table depicts the allowable deviation from a class expressed in terms of percent.

Fuzzy ratings are ultimately used to determine what percentage of each map class is acceptable and the magnitude of the errors within each map class. The fuzzy rating scale used for Region 5 accuracy assessments is as follows:

• 5: absolutely right. If this were the map label it would be a perfect match.
• 4: good. Would be happy to find this label on the map.
• 3: acceptable. Maybe not the best possible map label but it is acceptable.
• 2: understandable but wrong. Not an acceptable map label. There is something about the site that makes the label understandable, but there is clearly a better one.
• 1: absolutely wrong. The label is absolutely unacceptable.

The following example illustrates the logic for assigning fuzzy ratings to hardwood CalVeg types based on the percent cover of species present on a reference site.

Note: Species and pseudo species - QUCH2 Quercus chrysolepsis; QUDO Lithocarpus douglasii; QUKE Quercus kelloggii; QUWI Quercus wislizenii.

Accuracy Assessment Tables

The results of the Region 5 accuracy assessment process are represented in a series of tables. These tables depict overall error, confusion between classes, and the magnitude of error within classes for each of the assessed map attributes - lifeform, CalVeg conifer types, CalVeg hardwood types, CalVeg shrub types, confer tree size, hardwood tree size, total tree crown closure (10% and S,P,N,G). The following sections illustrate and discuss the important characteristics of the four tables produced by this process in the general order of their significance to the map user. Lifeform tables generated for the 1997 Tahoe National Forest vegetation layer are used as interpretation examples.

Overall and Class Accuracy

This is the most general accuracy table generated, providing information about the overall accuracy for the map attribute as well as accuracies for each map class. Two accuracy parameters are reported in this table - MAX and RIGHT. The MAX operator accuracy depicts the strictest measure of accuracy in that no fuzziness is considered when calculating the accuracy of map labels at each reference site. Put another way, a map label matches the reference data completely or its counted as an error. No error tolerances are allowed. The RIGHT operator, on the other hand, relies on the fuzzy ratings to indicate an acceptable level of error when considering the correctness of the map label. Map labels with a fuzzy rating of 3 or better are considered correct under the RIGHT operator. Generally, an increase in accuracy under the RIGHT operator results from the higher error tolerance. The user must determine whether his or her analysis objectives are unduly compromised with higher error tolerances. In general, class maps characterizing real life continuums such a vegetation, can have some error while still having utility.

Significant characteristics of the overall and class accuracy table are highlighted in the following life form example and are subsequently described.

Appendix A. Lifeform - Tahoe National Forest.
Output of the MAX and RIGHT operators.

Map Label

Assessed map classes.

# of Sites

Number of sites used in the accuracy assessment. Important for indicating the rarity of each map class and the validity of class accuracy. A low number of sites (< 10) indicates a less common condition and less reliable map class accuracy.

MAX

Overall and class accuracies that are not based on fuzzy ratings. Most conservative measure of map accuracy.

RIGHT

Overall and class accuracies that are based on fuzzy ratings. Sites where the corresponding map label receives a fuzzy rating of 3 or better are considered correct.

Confusion Table

A standard confusion matrix is generated to provide information about the type of error that occurs within and between classes. The confusion table lists all possible map classes and the number of reference sites within each class. It also summarizes the number of reference sites that were considered correct (3 or higher) when the map label was wrong (2 or less). This table is useful for determining whether a class is over-mapped, under-mapped, or randomly erroneous. By looking at the intersection of two classes in the matrix, the nature and significance of map class error can be evaluated. As an example, highlighted in blue italics in the following table, the intersection of conifer map labels and shrub reference sites shows a relatively low number of conifer map labels (5 out of 88) with sites that were better labeled as a shrub. While the total conifer class error is low, 83% of the conifer label mismatches are related to shrub and indicate a clear trend in conifer/shrub confusion where error does exist. Overall error trends for each map class are determined by comparing the total number of mismatches for each map class (right column) to the total number of sites better labeled as that map class (bottom row). The mismatches in the right total column are known as errors of commission while those listed in the bottom total row are referred to as errors of omission. In the shrub example below, only 1 shrub polygon is mislabeled while 10 reference sites occur in polygons better labeled as shrub, clearly depicting shrub as under-mapped.

Difference Table

The difference table arranges the error within each map class based on its magnitude. Error magnitude is calculated by comparing the fuzzy rating of each assessment site to the highest rank assigned to all other possible map labels. The difference in ratings provides a simple, relative measure of error severity. When the rating assigned to the map label is higher than the highest rating for all other labels, the resulting difference value is positive. Conversely, when the map label rating is lower than the highest rating, a negative value is tallied. For example, a conifer map label receiving a fuzzy rating of 3 on a reference site whose highest possible rating is 4, would show in the difference table as -1 under conifer map labels. Difference values of -1 through 4 generally correspond to correct map labels. Values of - 2 through - 4 generally correspond to map errors. Higher positive values indicate that pure conditions are well mapped while lower negative values show pure conditions to be poorly mapped. Mixed or transitional vegetation conditions, where a greater number of map labels are likely to be considered acceptable, will fall somewhere in the middle.

Ambiguity Table

The ambiguity table tallies map classes that characterize a reference site as well as the actual map label. The table is useful in identifying more subtle confusion between map classes. Trends that show greater ambiguity between two map classes may be useful for identifying additional map classes to be considered for analysis. In the following example 15 of the 88 reference sites associated with conifer map labels would have been equally well labeled as shrub. If shrub were a target class for a specific analysis, the analyst may need to consider some portion of conifer in the map (i.e. low crown closure classes) to ensure complete inclusion of possible shrub conditions.

Updated page content 04/09/03
Updated html code 02/24/04

USDA Forest Service · Pacific Southwest Region