6109.16,06.2,Ex.01,Part02 Page 1 of 15 FSH 6109.16 - DEMONSTRATION PROJECT HANDBOOK WO AMENDMENT 6109.16-92-1 EFFECTIVE 10/21/92 06.2 - Exhibit 01 (Project Plan)--Continued IV. Training A coordinated and extensive training effort, for managers and personnelists at both experimental and comparison sites, will be conducted prior to project implementation. The purpose of the training is to ensure that the experimental interventions are implemented as originally conceived, and that the intent of all interventions, policies, and procedures developed in connection with the project are clearly communicated to all managers, supervisors, employees, and others affected by project implementation. Training will be conducted prior to implementation to ensure that managers and personnelists understand not only the internal and standard operating procedures established in connection with the project but also the expanded role of the personnelist in providing management advisory services. Training and orientation will include the following major elements: 1. An in-depth description of each of the project interventions, including philosophy and expected effects; 2. A detailed outline of the policies and procedures established in support of each intervention; and, 3. An overview of the project evaluation effort, including the methods through which data collection will be accomplished. More specifically focused and ad hoc training modules will be developed as additional training needs are identified throughout the life of the project. To the extent that the continuing success of the project is a function of the adequacy of the training effort, periodic refresher training sessions are planned for the purpose of ensuring that experimental interventions continue to be implemented as originally conceived. As the system was designed to increase managerial and supervisory accountability for the integrity as well as the success of the recruitment and hiring program, a training effort particularly directed at managers and supervisors is critical to implementation. A discussion of the Department's plan to monitor managerial accountability follows in Section V., below. 06.2 - Exhibit 01 (Project Plan)--Continued V. Managerial Accountability Managerial accountability will be monitored as follows: 1. The extent to which merit system principles are upheld will be examined during regularly scheduled reviews of agency recruitment and hiring activity conducted as part of the Department's ongoing personnel management evaluation program. The review will include examining case file data which summarizes recruitment sources, whether or not incentives were offered to individual candidates, selections made, and other factors. 2. Agency personnel specialists will be responsible for the technically accurate implementation of project provisions on a day-to-day basis. Managerial reviews will focus on such areas as the modification of X-118 qualification standards, the placement of candidates in quality groups, and the application of veterans preference. 3. OPM will conduct a periodic review of particular authorities delegated under the demonstration project, including the determination of shortage categories and the approval to pass over a preference eligible, as part of each OPM region's ongoing personnel management evaluation program. 4. The Department's overall performance management program requires that senior executives be evaluated on the extent to which they meet the performance standard established for management and organizational effectiveness, as well as that which addresses equal opportunity/civil rights. The former stipulates, among other action items, that senior managers establish and maintain effective management control systems to monitor activities, identify problem areas, and initiate timely corrective action; the latter requires, in part, that Departmental and agency equal employment plan objectives and concepts are part of the total management process. VI. Cost/Benefit Analysis The experimental modifications are expected to be budget neutral. Current decentralized policies and procedures regarding the expenditure of agency funds will be retained. No additional funding will be requested specifically for thisproject; all costs will be charged to available funds through existing appropriations, including those incurred in the areas of project development, training, and project evaluation. No additional costs are expected to accrue to the operation of the alternative recruitment and hiring system, the extended probationary period, or the recruitment incentive system. In fact, the alternative recruitment and hiring system is expected to reduce the number of staff hours expended testing, examining, and rating applicants, as well as to alleviate some of the administrative burden generated in support of these activities. 06.2 - Exhibit 01 (Project Plan)--Continued VII. Duration of the Project The project will be implemented no earlier than ninety (90) days from the date of this notice. The project will terminate at or before the end of the five year period beginning on the date on which the project takes effect, unless otherwise extended or terminated at an earlier date in accordance with 5 U.S.C. 4703.(e). VIII. Evaluation Plan A. Introduction An evaluation methodology is established in order to comply with the requirement that the demonstration project be evaluated in terms of the impact of project results against stated objectives as well as to determine whether or not permanent changes in law and/or regulation should be considered or proposed. The Agricultural Research Service has entered into a cooperative agreement with the Pennsylvania State University for the purpose of conducting the evaluation. B. Methodology The formal evaluation will be conducted by the Pennsylvania State University and is expected to follow a modified Action Research Model process. The utilization of that model in this situation means that the evaluators, researchers, and research participants share information about the research results to the extent that interventions detrimental to the organization can either be modified or aborted. This is to prevent the continuation of an intervention with obvious and severe impact deficiencies. The evaluation effort will be carried out in four distinct phases, as follows: 1. Design phase - includes the development of the experimental model, selection of test and comparison sites, and the collection of baseline data prior to implementation; 2. Implementation phase - includes actual project implementation, and monitoring to assure that each of the project interventions has been operationalized as originally conceived; 3. Evaluation phase - includes data collection and analysis. Periodic reports and annual summaries will be presented throughout the life of the project; and, 4. Concluding phase - summary evaluation and overall assessment of the impact of the project; conclusions and final recommendations. 06.2 - Exhibit 01 (Project Plan)--Continued C. Model There are two objectives, five interventions, one constraint, and fifteen related hypotheses. It is operationally not feasible to apply interventions (a), (b), and (e) independently, nor can interventions (c) and (d) be broken into separate experiments. It was further decided to apply all interventions as one experimental design because to do otherwise would create inadequate sample sizes and become too difficult to manage. Hence, all interventions will be introduced at each of the experimental sites, although the authorization of recruitment and retention incentives is discretionary on the part of management. The utilization and results of experimentation with these incentives will be separately identified to the extent possible. 1. Objectives (a) Increase the flexibility and responsiveness of the recruitment and hiring system. (b) Increase the reliability of the decision to grant career tenure for employees in scientific positions. 2. Interventions (a) Decentralize the decision to authorize direct hire in shortage categories. (b) Implement an alternative candidate assessment method which uses categorical grouping instead of numeric score. (c) Provide the option of awarding monetary incentives for recruitment purposes. (d) Provide the option of reimbursing relocation travel and transportation expenses beyond those currently authorized for travel to first post of duty. (e) Extend the one-year probationary period to three years for employees in scientific positions. 06.2 - Exhibit 01 (Project Plan)--Continued Table 1. Expected Effects, Measures, and Data Sources Overall Project Constraint ___________________________________ Constraint Measures Data Sources ___________________________________ Fair representation of # of women & minorities Personnel Records protected groups will and individuals with (PR) not be adversely affected disabilities in the workforce file # among applicants Recruitment Case File (RCF) # available for RCF selection # offered positions RCF # hired PR _____________________________________ Objective 1: Increase the flexibility and responsiveness of the recruitment and hiring system Interventions: (a) Decentralize the decision to authorize direct hire in shortage categories. (b) Implement an alternative candidate assessment method using categorical grouping instead of numeric score. (c) Provide the option of awarding monetary incentives for recruiting purposes. (d) Provide the option of reimbursing relocation travel and transportation expenses beyond those currently authorized for travel to first post of duty. _____________________________________ Hypotheses Measures Data Sources A. Managers will perceive Managers' perceptions Annual Survey of the new system as more managers (AS) responsive to local recruitment needs. B. Managers will be more Managers' attitudes AS satisfied with the new recruitment and hiring system than with the traditional system. 06.2 - Exhibit 01 (Project Plan)--Continued C. Under the experimental Time required to fill employee intake process, positions RCF candidates will receive job offers more quickly than Managers' perceptions AS under the traditional New hires' perceptions Pre-employment system. survey (PES) D. The experimental Activities required to RCF employee intake process fill positions will require equal or less Staff hours required RCF staff time than the to fill positions traditional system. E. Quality of new hires GPA PES under the experimental Education level PES system, as measured by Year of degree PES appropriate indicators, Field of Study PES will be equal to or Previous Salary PES greater than that of Level of experience PES employees hired through Relevant experience PES traditional methods. Competing Offers PES F. Level of performance Performance appraisal PR of new hires will be equal Performance awards PR to or greater than that Adverse actions PR of those hired through Managers' perceptions AS traditional means. G. Competitive recruitment Managers' perception of AS position will improve for competitive position sites using the Candidates lost to RCF experimental recruiting competition and hiring system relative Number of qualified RCF to those using traditional candidates who apply procedures. for USDA positions Ability to hire AS candidates whom the managers perceive as best qualified H. Recruitment incentives Declination/acceptance RCF will increase acceptance ratio rates above and beyond any Number, type, and effects of the new $ value of incentives PR recruitment and hiring system. 06.2 - Exhibit 01 (Project Plan)--Continued I. Retention will improve Rate at which service for appointees for whom a agreements are met PR recruitment incentive is Turnover rates PR authorized relative to Number of voluntary PR those hired without separations with recruitment incentives. outstanding deferred or incremental payments Number, type, and $ value of incentives PR _________________________________________________________________ Objective 2: Increase the reliability of the decision to grant career tenure for employees in scientific positions. Intervention: (e) Extend the one-year probationary period to three years for employees in scientific positions. Hypotheses Measures Data Sources _________________________________________________________________ A. Managers will have more Managers' perceptions AS confidence in career tenure decisions with an extended probationary period. B. Turnover patterns and Turnover rates/reasons PR the reasons associated with those patterns will differ between experimental and comparison sites. C. Recruitment efforts Managers' perceptions AS will not be hampered by the New employees' PES extended probationary perceptions period for employees in Declination/acceptance scientific positions. ratio (scientists) RCF 06.2 - Exhibit 01 (Project Plan)--Continued _________________________________________________________________ Hypotheses Measures Data Sources _________________________________________________________________ A. Supervisory responsi- Managers' perceptions AS bility and accountability Senior managers' for the integrity as well perceptions Interviews as the success of the Documentation of recruitment and hiring accountability Performance program will increase. Standards RCF Personnel Management RCF review evaluation B. Management advisory Managers' perceptions AS Interviews role for personnel specialists will increase. C. Total operating costs Administrative costs for for recruitment and hiring recruitment and hiring To be identified will not increase due to Amount spent for the new recruitment and recruitment incentives hiring process. D. Procedures Experimental results will be evaluated annually (1) against baseline data collected before the implementation of the interventions, and (2) between the experimental and comparison populations in each block and in toto. While the introduction of the interventions will continue throughout the five year life of the project, the collection of data and consequent evaluation efforts will continue until reasonably stable results can be identified. It is not necessary to wait until the results of the last experimentally hired employee can be fully evaluated before declaring the evaluation completed if sufficient valid data have been accumulated to draw such a conclusion. E. Model Evaluations The evaluation plan shown in Figure 1 represents the consensus of USDA, FS, ARS, and OPM regarding project constraints, objectives, and expectations. USDA has entered into a cooperative agreement with the Pennsylvania State University (PSU) to conduct the evaluation of this project. PSU's efforts will be guided by, but in no way restricted to, the hypotheses, measures, and data sources in this model. This demonstration project is a complex experiment, to be conducted in a dynamic environment over a 5-year period. Based on OPM experience with previous demonstration projects, we expect that modifications to the evaluation model will be required in response to mid-course project changes; following statutory, regulatory, and policy changes related to project interventions; and based on further exploration of proposed data sources. 06.2 - Exhibit 01 (Project Plan)--Continued All additions, deletions, and modifications to the current evaluation model will be fully documented and explained as part of the evaluation reporting process. If at any time during the course of the project, experimental sites are added or deleted, we will publish a notice in the Federal Register explaining the reasons for the change. F. Implementation Evaluation The evaluation model presents a framework for evaluating the success of the demonstration project in meeting its stated objectives. An equally important component of the overall evaluation is the description and monitoring of the implementation of the project. Implementation monitoring will provide a qualitative context in which to understand and interpret evaluation results. This facet of the evaluation will help to answer the "why" questions that are likely to arise, whether the project is a success or failure, and provide the documentation of actual implementation necessary to replicate the results of this demonstration project in other settings. Through examination of project-related documents, ongoing contact with key players in the participating agencies, interviews with project participants, and case studies of selected units, the implementation evaluation will address such questions as: 1. When are project changes implemented? 2. What training and orientation is delivered to facilitate implementation? 3. What operating procedures/guidelines are developed to manage project implementation? How do these guidelines differ from the project plan? 4. How does actual practice differ from the project plan and/or operating guidelines? 5. To what extent are recruiting and hiring practices being carried out in a manner consistent with the merit principles outlined in 5 U.S.C. 2301? 6. To what extent do users of the new system understand it? 7. What differences in actual practice exist with respect to different types of candidates and/or new appointees? 8 To what extent are discretionary interventions (i.e., recruitment incentives) actually used? 9. How similar are the experimental and the comparison sites? How are they different? 06.2 - Exhibit 01 (Project Plan)--Continued 10. What other changes occurring at experimental/comparison sites might provide competing explanations for observed changes? 11. What events in the external environment might provide competing explanations for observed changes or lack of change? 12. What unintended consequences of the demonstration project initiative may be observed? 13. What impact does the application of veterans preference under the project have on the examining and selection process? How many preference eligibles are being hired at experimental and comparison sites? 14. To what extent do the project interventions affect perceptions of equity among employees at experimental sites? IX. Experimental Design The experimental design uses experimental and comparison groups matched for characteristics but not necessarily for population numbers in ten population blocks. The populations were classified into five occupational categories: scientists, professionals, administrative employees, technicians, and all others (includes clerical and Federal Wage System). Each participating unit of the two agencies, the Agricultural Research Service and Forest Service, appeared in as many of these five blocks as the occupational category distribution represents. The occupational categories were then further subdivided into two situational categories: difficult and simple. In the difficult situation, the working site or environment, and/or the local personnel supply pool, make it very difficult to attract candidates in a particular occupational category. This situational condition is not necessarily constant for all five occupational categories at any one location. It was postulated that this random assignment to the experimental and comparison groups within these ten groupings would result in ten matched groups. Except for limited administrative pre-selection, the assignments to experimental and comparison groups within each of the ten blocks were by random draw. A detailed description of the selection process is available upon request. 06.2 - Exhibit 01 (Project Plan)--Continued Appendix A. Required Waivers to Law and Regulation Waivers to Title 5 United States Code 1104(a)(2) Delegation of authority for personnel management 3309 Preference eligibles; examinations; additional points for 3312(b) Preference eligibles; physical qualifications; waiver 3313 Competitive service; registers of eligibles 3317(a) Competitive service; certification from registers 3318(a),(b) Competitive service; selection from certificates 5723 Travel and transportation expenses of new appointees and student trainees; manpower shortage positions 5724a, 5724b, 5724c Relocation expenses 7501(1), 7511(a)(1)(A) Adverse actions; definitions, "employee" Waivers to Title 5 Code of Federal Regulations 2.1(b) Competitive Examinations and Eligible Registers. 315.801 Probationary period; when required. 315.802 Length of probationary period. Part 332 Recruitment and selection through competitive examination (except 332.101) 337.101 Rating applicants (except (c)) Part 352 Reemployment Rights 06.2 - Exhibit 01 (Project Plan)--Continued Appendix B. Experimental and Comparison Sites. Agricultural Research Service: Experimental Sites Experimental Sites Comparison Sites Akron, CO Peoria, IL Gainesville, FL Albany, CA Phoenix, AZ Georgetown, DE All Hawaiian Islands Pullman, WA Greenville, TN Ames/Ankeny, IA Raleigh, NC Headquarters, MD Athens, GA Reno, NV Houma, LA Beaumont, TX Riverside, CA Jackson, TN Beckley, WV San Francisco, CA Kearneysville, WV Beltsville, MD Shafter, CA Kimberly, ID Bozeman, MT Sidney, MT Laramie, WY Byron, GA St. Paul, MN Lewisburg, TN Canal Point, FL St. Croix, VI Lincoln, NE Charleston, SC Stillwater, OK Mandan, ND Clemson, SC Stoneville, MS Mayaguez, PR Columbia, MO Suffolk, VA Mississippi State Corvallis, OR Temple, TX Morris, MN Davis, CA Tifton, GA Newark, DE Dawson, GA Tucson, AZ Pendleton, OR East Grand Forks, MN University Park, PA Poplarville, MS East Lansing, MI Watkinsville, GA Prosser, WA Fairbanks, AK Weslaco, TX Salinas, CA Fargo, ND Woodward, OK Savannah, GA Florence, SC Wooster, OH Stuttgart, AR Frederick, MD Yakima, WA Tuxtla, MX Fresno, CA Urbana, IL Fort Collins, CO Comparison Sites Wenatchee, WA Geneva, NY Aberdeen, ID West Lafayette, IN Grand Forks, ND Auburn, AL Winter Haven, FL Griffin, GA Baton Rouge, LA Wyndmoor, PA Houston, TX Boise, ID Ithaca, NY Booneville, AR Kerrville, TX Boston, MA Lane, OK Brawley, CA Las Cruces, NM Brookings, SD Lexington, KY Brooksville, FL Logan, UT Brownwood, TX Lubbock, TX Burns, OR Madison, WI Bushland, TX Manhattan, KS Cheyenne, WY Miami, FL Clay Center, NE Miles City, MT College Station, TX New Orleans, LA Columbus, OH Orient Point, NY Coshocton, OH Orlando, FL Delaware, OH Orono, ME Dubois, ID Oxford, MS Durant, OK Oxford, NC El Reno, OK Pasadena, CA Ft. Lauderdale, FL 06.2 - Exhibit 01 (Project Plan)--Continued Forest Service: Experimental Sites Comparison Sites Region 1 Regional Office Beaverhead NF (includes (includes MTDC) Gallatin NF) Clearwater NF Bitterroot NF Custer NF (includes Deerlodge NF Helena NF) Lewis and Clark NF Flathead NF Nez Perce NF Idaho Panhandle NF Kootenai NF Lolo NF Region 2 Big Horn NF Regional Office Black Hills NF Medicine Bow NF (includes Nebraska NF) San Juan NF Grand Mesa, Uncompahgre, White River NF and Gunnison NF Pike and San Isabel NF Rio Grande NF Routt NF Shoshone NF Region 3 Apache Sitgraves NF Regional Office Cibola NF Carson NF Coconino NF Gila NF Coronado NF Prescott NF Kaibab NF Lincoln NF Santa Fe NF Tonto NF Region 4 Regional Office/ Payette NF (includes Intermountain Station Boise & Sawtooth NF) Targhee NF (includes Utah-Nevada Cluster: Bridger-Teton, Challis, Fish Lake, Dixie, Caribou, & Salmon NF) Humboldt, Toiyabe, Uinta, Wasatch-Cache, Manti-LaSal, Ashley NF, and Geometronics Service Center 06.2 - Exhibit 01 (Project Plan)--Continued Region 5 Regional Office Cleveland NF Angeles NF Eldorado NF Inyo NF San Bernardino NF Klamath NF (includes Six Rivers and Modoc NF) Los Padres NF Plumas NF (includes Lassen and Mendocino NF) Shasta-Trinity NF Sierra NF (includes Sequoia and Stanislaus NF) Tahoe NF/Basin Region 6 Colville NF Regional Office/PNW Deschutes NF (includes Gifford Pinchot NF Malheur & Ochoco NF) Rogue River NF (includes Mt. Baker/Snoqualmie NF Siskiyou NF) (includes Seattle Lab) Mt. Hood NF Olympic NF Siuslaw NF Umpqua NF Wallowa-Whitman NF (includes LaGrande Lab & Umatilla NF) Wenatchee NF (includes Wenatchee Lab & Okanogan NF) Willamette NF Winema NF (includes Fremont NF) Region 8 Regional Office Daniel Boone NF (includes (includes Macon Seed Cherokee NF) Lab & Caribbean NF) Ouachita NF (includes Chattahoochee & Oconee Ozark-St. Francis NF) NF (includes Frances Marion & Sumter NF) Jefferson NF (includes George Washington NF) Kisatchie NF (includes Texas NF, part of Alexandria Lab) Mississippi NF (includes NF's in Alabama and Florida) 06.2 - Exhibit 01 (Project Plan)--Continued Region 9 Regional Office Green Mountain & Finger Lakes NF Allegheny NF Shawnee NF Chequamegon NF White Mountain NF Chippewa NF Hiawatha NF Huron-Manistee NF Mark Twain NF Monongahela NF Nicolet NF Ottowa NF Superior NF Wayne-Hoosier NF Region 10 Regional Office Stikine Area Chatham Area Chugach NF Ketchikan Area Research Units Forest Products Lab North Central Station Intermountain Station/ Pacific NW Station/R-6 RO R-4 RO Rocky Mountain Station Northeastern Station/Area (includes Arapahoe & Pacific Southwest Roosevelt NF) Forest and Range Experiment Station Southeastern Station (includes North Carolina NF) Southern Station Headquarters Washington Office Billing Code 6325-01-M