Monitoring Contract Performance

Kelly LeRoux, International City Managers Association

To monitor a contract effectively, each member of the contract administration team must understand the duties associated with his or her position on the team and must understand a number of basic issues that are common to public and private service delivery: customer satisfaction, timeliness, productivity, and performance problems. The team must understand the importance of pre-performance conferences in laying the groundwork for efficient and effective performance and monitoring. Despite the care that has gone into the preparation of the contract documents, most contracts require amendments, and the team must understand when and how to issue them.

To monitor a contract effectively, the team must select appropriate measures of output or outcome, choose suitable monitoring methods, carefully document performance, and work cooperatively with the contractor.

Preparing for contract monitoring

This section delineates what the contract administration team needs to know and do before performance begins. It begins with the structure of the contract administration team and the responsibilities of the members; looks at several issues of basic importance in monitoring a contract; discusses the first step in contract administration, the pre-performance conference; and concludes with an overview of the contract amendment process.

The contract administration team

Chapter 2 described several organizational models for a service contracting program. Under the centralized model, a single contract manager is responsible for contracts in several or all departments and is assisted by a contract administrator and field manager for each contract established. In the decentralized model, each department has its own contract manager and supporting teams that draw on the resources of the legal, purchasing, and finance departments for technical direction. In the more common combination model, some contracts are handled centrally and others by individual departments. In the network model, a local government department contracts with one or more core providers who may in turn rely on subcontractors to meet the complex service needs outlined in the scope of work. Whether they function in a centralized, decentralized, combination, or network model, all contract administration teams have the same goal: to work with the contractor(s) to develop and maintain an environment in which the contractor can succeed in delivering high-quality public services at the agreed-upon cost.

Because of the variety and flexibility of organizational structures used in service contracting, the term "contract administration team" used throughout this chapter does not refer to a large static team that handles all service contracts but to individual teams composed of a contract manager and/or a contract administrator and/or a field manager and persons who do not bear these titles but perform similar duties. For the sake of consistency, the emphasis in this chapter is on the duties of the contract manager, contract administrator, and field manager.

Contract manager responsibilities The contract manager reports to the local government manager, project manager, or-in a decentralized model-a department head. As the team leader, the contract manager is responsible for the success of the contract administration activities for each contract. The contract manager functions as coordinator, organizer, evaluator, mediator, and enforcer of the contract administration program.

As coordinator, the contract manager

· Coordinates the activities of contract administration teams

· Conducts scheduled meetings at least monthly with each team to review the status of individual contracts and the progress of the contracting program

· Reviews and acts upon the requests of all parties involved in a contract, working closely with the contractor and with the auditing, finance, legal, and purchasing departments.

As organizer, the contract manager

· Establishes what is to be evaluated for each contract and devises rating systems and monitoring methods to be used

· Establishes evaluation criteria, rating systems, surveys, and other monitoring methods to be used for each contract

· Assembles and trains the contract administration team

· Establishes procedures for review of contract problems or disputes

· Responds to management requests for new services or changes in existing services

· With the team, prepares default contingency plans for each contract

· Prepares and updates a contract administration procedures manual.

As evaluator, the contract manager

· Determines the acceptability of reports or other deliverables provided by the contractor

· Reviews contractor payment requests and approves or rejects them as required

· Monitors the performance of the contractor in person when possible, working closely with the field manager

· Reports to the local government manager on contractor performance

· Evaluates the effectiveness of the contract administration teams in reaching their goals

· Establishes inspection schedules for performance monitoring

· Plans and manages surveys of citizens and department clients

· Analyzes information collected during the contract term to identify ways to improve the effectiveness and efficiency of the service, implements the improvements with the current contractor either by mutual agreement or by contract amendment, and incorporates the improvements into future specifications for the service.

As mediator, the contract manager

· Resolves conflicts between the contractor and citizens or local government clients when such conflicts cannot be resolved by the field manager or contract administrator

· Monitors team members, departments, and user agencies to prevent the development of an adversarial relationship that could affect the contractor's performance

· Negotiates and approves amendments to the scope of work and attempts to resolve disputes with the contractor

· Negotiates and resolves agency demands on contractors that may be beyond the contractor's responsibilities.

As enforcer, the contract manager

· Interprets contract provisions for the contractor, contract administration team members, and departments

· Decides what liquidated damages provisions or other default actions will be taken if serious service problems arise.

Contract administrator responsibilities The contract administrator functions at the department level and reports to the department director or the contract manager or both. The contract administrator also functions as the contract manager in the manager's absence. There is usually one contract administrator in a department; this person is responsible for providing administrative support to the department field managers and administrative oversight for all the service contracts in the department. A department with many contracts may require more than one contract administrator, each providing administrative support to several field managers and oversight for their assigned contracts.

The contract administrator

· Directs, monitors, and reviews the field manager's performance

· In cooperation with the field manager, assesses the contractor's performance

· Ensures timely submission of required reports from the contractor and field manager

· For cost reimbursement and unit price contracts, monitors the contractor's expenditures to ensure that they stay within expenditure forecasts and budget

· Reviews the contractor's payment requests and the field manager's payment recommendations, processes approved payment requests, and forwards requests that have been disapproved by the field manager or the contract administrator to the contract manager for review and disposition

· Tracks payment requests as they travel through the accounts payable system to ensure that the local government meets the promised payment schedule

· Assists the field manager in reviewing the contractor's requests for amendments and forwards to the contract manager recommendations for action on the changes

· Makes recommendations to the contract manager regarding enforcement of liquidated damages provisions or default procedures

· Maintains the master contract files and ensures that insurance certificates, bonds, licenses, and other contract-related documents are current

· Monitors the contractor's compliance with small and minority-owned business plans and Equal Employment Opportunity goals if applicable

· Helps prepare default contingency plans.

Field manager responsibilities The field manager reports to the contract administrator and is the team's principal link with the contractor. Field managers may be responsible for one or more service contracts. The field manager is in more frequent contact with the contractor's crews than any other team member and must be alert to identify service problems in the early stages. Of the three team members, the field manager is generally the most familiar with the operations of the contract and the exact duties and responsibilities of the contractor.

The field manager

· Checks the qualifications of the contractor's personnel against contract requirements

· Conducts scheduled and unscheduled inspections of the contractor's work to ensure that it is done as required

· Reports on contractor performance

· Monitors the contractor's quality control program

· Reviews payment requests to be sure that items or hours charged match the contract prices and accurately reflect the services that were received

· Alerts the contract administrator to developing problems

· Helps develop default contingency plans

· Advises the contract manager of improvements that could be made in the scope of work in current and future contracts.

Service delivery issues

The key to effective contract administration is to identify and address all problems before they become unmanageable. The following discussion addresses issues common to both public and private service delivery and offers guidelines to assist the contract administration team in dealing with them.

Customer satisfaction All customers deserve satisfactory service. To help ensure that they receive it, the contract administration team should assess customer satisfaction. If evaluation systems and procedures were used to monitor in-house service delivery, they should remain in place to evaluate customer satisfaction with private service delivery, particularly when a service is delivered to private citizens. If no systems are in place to measure and evaluate customer satisfaction, incorporate customer surveys, interviews, customer service lines, or some other similar mechanism into contract administration activities.

TimelinessMonitoring delivery schedules to prevent or minimize lapses in service is a critical function of the contract administration team. The team reviews, approves, and monitors contractor work schedules to ensure that acceptable services are delivered when and where promised. When a service is delivered in house, schedule slippage can often be corrected by shifting employees from other assignments to the problem area until the service is back on schedule. In the case of contracted services, slippage must be addressed by the contractor, and a contractor is seldom amenable to spending the money necessary to bring a slipped schedule up to date. It is not uncommon for a contractor to demand extra payment for correcting slippage, claiming that the local government contributed to the problem. The contract administration team must be alert to correct slippage problems immediately.

Productivity Specific productivity standards-for example, the amount of work to be performed by a work crew during one shift-should be part of every contract. These standards must be at least as high as those established for local government employees who formerly provided the service.

Record-keeping

Record-keeping increases significantly under service contracting. Preparing performance reports, keeping bond and insurance files current, tracking contract payments, and responding to contractor claims and disputes consume a considerable amount of the contract administration team's time.

To simplify record-keeping, the contract manager should maintain an individual master file for each contract that contains the complete history of the contract, including all correspondence, reports, evaluations, insurance certificates, and so forth. Do not mingle correspondence relating to several contracts in a general correspondence file. Place correspondence regarding each contract in its own file. To keep paperwork to a minimum, do not require a standard report or procedure if it is not essential for the monitoring of a specific contract.

The contract administrator (or the person who fills the equivalent position) is responsible for preparing and maintaining the master contract file. Although the field and contract managers may maintain their own files, everything in those files must be copied to the master file. The master contract file is the central reference for the contract and should contain (at a minimum) all contract documents, including the contractor's bid response, the scope of work, amendments, bonding documents, current insurance certificates, and required licenses and permits, and all performance documents, such as correspondence, evaluations, reports submitted by contractor and the contract administration team, payment requests, and copies of invoices

Careful monitoring is necessary to make sure that the contractor meets productivity requirements. When productivity falls below the established standard, the contract administration team must act quickly to prevent further decline and to ensure that productivity is brought back to contract minimums.

Performance problems Most problems that arise with public or private delivery of a service are based on poor performance on the part of individuals. It is important to be aware that when problems arise with a contractor, the attempt to resolve even simple issues can potentially result in a formal contract dispute. A contract manager's request to a contractor to correct a problem or perform a task may have little or no effect unless it is made clear to the contractor that there has been a breach of a specific term or condition of the contract or that the task is part of the contractor's obligations under the contract. If the contractor does not agree with the contract manager, the problem may eventually require resolution by the governing body or the courts.

The contract administration team, particularly the field manager, should intervene to prevent problems or conflicts from developing into formal disputes. Examples of intervention include meeting with a contractor whose repairs to equipment are being rejected at an unacceptable rate or informing a contractor that an employee has behaved belligerently toward a client.

Pre-performance meeting

Pre-performance meetings clarify the procedures to be used throughout the contract term and enable all parties to better understand one another's perspective and expectations. These meetings are especially important because members of the contract administration team may have different views of the contract due to their different responsibilities. The contract manager, for example, focuses on the relationship between the contractor's performance and service delivery goals. Both the department and the contract administrator are principally concerned with whether the contractor's performance meets department goals, contract requirements, and the project budget. A major part of the contract manager's job is to ensure that these sometimes-conflicting views do not interfere with the administration of the contract, that team members understand that the contract document alone should govern their actions, and that contract administration must be a team effort to succeed.

The contract administration team holds two separate pre-performance meetings: the first with the department or departments involved and the second with the contractor.

Department meeting The department meeting is facilitated by the contract administration team and attended by staff from all the departments involved. In this meeting, the contract manager defines the roles of the team members, reviews the forms and procedures to be used, and responds to any concerns or procedural questions department staff may have. Particularly when contract administration is centralized, this meeting helps assure department managers that although the details of monitoring and administration may be handled by team members outside the department, the department will continue to influence how the contract is administered and the quality of service delivery.

The meeting allows a department to question, comment upon, or recommend changes to the procedures that have been developed before the team commits itself to them at the pre-performance meeting with the contractor. This is especially important when the contracted service is closely related to the central mission of a department. For example, a contract for temporary clerical help is usually assigned to a central contract administration office and may be monitored by each department using the service. Although the contract is seldom monitored exclusively by the personnel department, it represents a service that is like the service provided through that department. Although not charged with overall contract administration and monitoring responsibilities, the personnel department is concerned about such issues as the minimum qualifications of personnel hired under the temporary help contract, the rate of pay for temporary personnel, and benefits of temporary personnel. The personnel department wants assurance that all employees, including temporary contract help, are qualified to perform their assigned tasks, are aware of their rights and responsibilities, and are treated fairly. Involvement of the department in the pre-performance meeting enables it to provide input during the contract administration process and establishes it as a resource for the team.

Contractor meetingAfter the department meeting but before the start of contract performance, a pre-performance meeting is held with the contractor. The purpose of this meeting is to establish a common understanding of contract requirements and the responsibilities of both the local government and the contractor under the contract. The meeting also helps make service start-up as smooth as possible.

The participants and the structure of the meeting can vary considerably. Typical participants include the contract administration team, staff from the departments involved, the contractor, and the contractor's staff person responsible for monitoring service delivery. In contracting for trade services (plumbers, carpenters, etc.), equipment repair services, or mowing services, a meeting between the contractor, the department involved, and a member of the contract administration team is sufficient. Services that affect essential operations, such as public transportation, refuse collection, or recycling, may require the attendance of the entire contract administration team, representatives from the departments involved, a representative of the local government manager, and possibly a member of a citizens' advisory committee.

An agenda should be prepared that includes topics such as the following: the identification of participants and a description of their authority, review of the contract documents, the local government's responsibilities, the contractor's responsibilities, performance expectations and monitoring, quality control, resolution of service complaints and contract disputes, contract enforcement, and payment procedures. When appropriate, combine the contractor's conference with a site tour to review field conditions that could affect performance. The accompanying sidebar describes these agenda items in more detail.

Amendments to the contract

The local government and the contractor are required to perform only those duties described in the contract documents. Because even the most comprehensive contract documents may fail to address clearly some aspects of service delivery, the responsibilities of the parties, or other important issues, amendments are almost inevitable.

Either the local government or the contractor may request an amendment, but both parties must agree to it unless the contract specifically allows unilateral changes by the local government (e.g., the incorporation of new legislation that does not affect the contract price, the time of performance, the quality of service, or the establishment of a new price structure for subsequent years on the basis of a formula set forth in the contract). Amendments must be approved by the legal department to be sure that the rights of both parties are protected. They are binding only if signed by the principals of the contractor's firm and local government officials authorized to execute contracts for the parties to the contract. Local government department heads, contract managers, contract administrators, and field managers are seldom, if ever, authorized to approve or sign contracts or contract amendments.

Although minor changes or clarifications that do not affect price, time of performance, or quality of service are sometimes made informally through a letter of understanding, the safest course is to issue a formal amendment to the contract. The letter of understanding-which is usually signed by only one party to the contract-has considerably less legal status than a formal amendment, which must be executed by both parties. A seemingly inconsequential issue that is dealt with through a letter of understanding may open the door to later contract disputes.

It is not unusual for a contractor to attempt to use the amendment process to correct a pricing error in the bid response. Requests from the contractor for additional compensation or a change in the level of service must be examined carefully to determine whether the request is valid and reasonable. If the request for an amendment stems from the fact that the contractor underestimated the effort or expense required to do the work or overestimated its capacity to provide the service, the amendment is not justified and should not be agreed to.

Valid amendments have a common element-the work covered is beyond the scope of work defined in the original contract documents. If the local government requests work that is not clearly within the scope-for example, adding a new building to a custodial services contract-an amendment is justified. If an amendment affecting the contract price is determined to be valid, the contractor must provide a detailed breakdown of the proposed cost increase to demonstrate that there are no unwarranted extras built into the amended price.

Pre-performance conference agenda items

Authority of participants Identify representatives of both the local government and the contractor and state the limits of their authority.

Review of the contract documents Review the contract documents in detail. Clarify any ambiguities and make the clarifications part of the written record of the meeting. If necessary, amend the contract to avoid future problems of interpretation. Review the contract amendment process. Stress the need for written amendments and remind attendees that verbal changes are not binding on either of the parties. Define what the local government considers adequate justification for an amendment.

Local government responsibilities Review the local government's obligations under the contract. Assure the contractor that the local government will abide by the contract and will provide an environment in which the contractor can provide the service with the least interference.

Contractor responsibilities Review the contractor's responsibilities described in the contract. They generally include following contract requirements: following the established staffing plan; providing a responsible on-site manager; assigning qualified personnel to the contract; providing the equipment required to do the assigned tasks; maintaining records of expenditures, deliverables, and progress; submitting required reports on schedule; and maintaining a clear line of communication for prompt resolution of problems.

Performance expectations and monitoring Describe the level of performance expected and provide the contractor with copies of the forms to be used for evaluation and monitoring. Review the forms in detail so that the contractor understands the performance standards and the evaluation methods. Encourage the contractor to recommend improvements in service delivery methods or productivity during the contract term. Establish a schedule for follow-up meetings to review both contractor and local government performance.

Quality control If the contract requires the contractor to have a quality control program, review and discuss the program. A contractor's quality control program may be similar to a local government's contract administration and monitoring or quality improvement program. It should have the same elements: inspection and observation on a scheduled and unscheduled basis; procedures for early identification of service delivery problems; and provisions for written records and reports.

Resolution of service complaints Review the procedures for prompt resolution of complaints generated by citizens or local government clients. Review any complaint forms devised for the contract and the information required from the contractor to complete them. If the contractor is responsible for managing the complaint process independently, describe the nature and frequency of any reports required.

Contract disputes Review the procedures for handling disagreements between the local government and the contractor. Provide the contractor with copies of ordinances, regulations, or procedures governing disputes. Review the time limits for dispute resolution, identify the local government staff responsible for settlement of disputes, and describe the administrative appeals procedure.

Contract enforcement Explain the liquidated damages provisions of the contract and describe the conditions that trigger their enforcement or enforcement of other similar provisions of the contract.

Payment procedures Because many contractors are not familiar with local government payment procedures, review the payment process from the time payment is requested by the contractor to the issuing of a check. Explain how processing of the check can be delayed by errors such as inserting incorrect payment terms on an invoice, failing to properly identify the department for which the work was performed, or failing to give a purchase order number. Discuss the level of detail or format of the invoice required under the contract.

Performance monitoring

This section covers the basics of performance monitoring, focusing on two approaches: measuring outputs and measuring outcomes. Measuring outputs involves a relatively straightforward method of counting service components produced. This might include number of intakes conducted in a given month, number of acres mowed, number of meals served, number of curb miles paved, number of health screenings performed, and so on. Outcome monitoring, also known as performance measurement, is a much more difficult task. Assessing outcomes involves asking whether clients were helped and what impact the service is having.

Information on output is generally gathered by the contract administration team. Information on outcome, or results, may be derived from multiple sources, including data collected from users of the service. [1]Both output and outcome monitoring are based on information that must be gathered and recorded, and this chapter covers a number of approaches to obtaining data and documenting performance, including observation records, discrepancy reports, field diaries, surveys, and customer hotlines.

Any monitoring method or combination of methods may be appropriate for a given contract. Part of successful contract administration is selecting the method that will provide the most timely and accurate information to the local government at the least cost.

Whatever the method used, it is important to remember that a single monitoring report provides only a snapshot view of performance; inspections and observations over time are required to create a comprehensive picture of average contract performance.

Getting started

The first thirty days of the contract term influence the character of the long-term relationship between the contract administration team and the contractor. For any contractor new to the local government, erratic performance during the first month is to be expected. Being patient and working with the contractor to resolve contract start-up problems will help create a long-term relationship characterized by cooperation and mutual respect.

Contract documents usually establish a range of acceptable performance, but contractors seldom perform at the upper end of the range, particularly when beginning a new contract. For example, even an experienced contractor may have underestimated the effort required to perform some aspects of the service and overestimated others. These miscalculations may cause performance to be uneven until the contractor makes the necessary adjustments in personnel and equipment to balance service in all contract areas.

During the start-up period, the contract administration team needs to monitor intensively but leniently. The point is to help the contractor get on the right path but not to interfere with the contractor's work. Performance problems are sure to arise, but the team should assist rather than criticize the contractor. Whatever idealized notions the team may have had about service contracting will quickly be tempered by the realization that the contractor faces the same service delivery problems that the local government faced with public service delivery-and may make the same mistakes in trying to resolve them.

Monitoring output

There are several choices for gathering information on outputs. One option is direct monitoring, which occurs while the work is being performed. Another option is follow-up monitoring, which occurs after the work is completed. Monitoring by exception is another approach, which may incorporate either direct or follow-up monitoring but is triggered only by specific complaints about service. Both direct and follow-up monitoring can be conducted according to a schedule or at random. The next five sections explore these options in detail.

Direct monitoringDirect monitoring is used mainly during contract start-up and occasionally during the contract term to ensure that the work is being performed according to established procedures or standards. Although it can be helpful in determining exactly how the contractor delivers the service, excessive use of direct monitoring can antagonize the contractor's employees and create tension between them and the observer.

Follow-up monitoringFollow-up monitoring is the inspection method used most often. Its purpose is to answer general and specific questions such as the following:

· Does the work meet contract requirements?

· Is the contractor on schedule?

· Have the required number of units of service been delivered?

· Are clients satisfied with the service?

· Has the work area been left in acceptable condition?

· Is the grass mowed to the correct height?

· Are all the rest rooms clean?

· Has all trash been removed from the curbside?

·

The monitor's monitor

Responsible contractors place as much or more value on effective monitoring as the local government and their internal quality control measures may be more stringent. For some contractors, elaborate internal monitoring mechanisms may be a requirement of accreditation, certification, or membership in industry-specific associations. In addition, to protect their investment in the contract and to prevent undue interference, contractors often familiarize themselves with contract law, local ordinances, and state and federal regulatory requirements. Essentially, they monitor the local government monitors to be certain that they do not overstep their authority

Monitoring by exceptionMonitoring by exception is a common and often necessary part of a contract administration program. Monitoring all contracts with the same attention to detail is costly, and the local government that can afford to do so is rare. For every contract requiring intensive monitoring there are probably several candidates for monitoring by exception.

If a service is essential to the community or if there is a risk that a client can actually be harmed by services of poor quality (refuse collection and social services are examples), then monitoring by exception cannot be relied on as the sole monitoring method. However, monitoring by exception is usually acceptable for contracts that meet two criteria: (1) the work performed is easy to measure and is conducted primarily at a contractor's site rather than on local government property (fleet repair and blueprinting services are examples); and (2) the delivered product (e.g., the work of a painting or plumbing contractor) is the criterion by which the contractor's proficiency is evaluated. Monitoring by exception is also appropriate for services that users evaluate continuously as a matter of course. For example, in the case of personal services (those delivered by a specified individual), a user will simply not rehire an unsatisfactory contractor; in the case of temporary help, a dissatisfied user will dismiss the assigned employee.

The principal disadvantage of monitoring by exception is that the contractor and the contract administration team may be caught off guard by a problem that might have been picked up by more intensive monitoring. For example, if a contract for repair of a small bus fleet used for supplemental public transportation is monitored by exception and a vehicle breakdown occurs as a result of the contractor's use of rebuilt transmission or engine components rather than new components, as called for in the contract, the entire fleet could require extensive repairs to correct prior unauthorized work-a situation that might have been prevented by periodic examination of the contractor's repair work orders. Periodic examination can be helpful in identifying a problem early enough to prevent a crisis later.

Scheduled monitoring Scheduled monitoring for some types of services is necessary to ensure systematic review of performance. Generally, scheduled monitoring occurs on a monthly or quarterly basis. The contract may have a plan for scheduled monitoring built in from the beginning or may initiate scheduled monitoring if service problems begin to develop or if the local government receives significant complaints from service recipients.

When an increase in productivity creates a problem

Although one goal of service contracting is to improve productivity, increased productivity can signal trouble. In lump-sum, fixed price contracts, it is in the best interest of the contractor to increase productivity because all productivity increases contribute to profit. But when productivity increases are achieved by lowering the quality of service, what benefits the contractor may harm service recipients. Thus, the contract administration team must examine unusual improvements in productivity as carefully as it would a drop in productivity.

Scheduled monitoring is arranged in advance with the contractor's project officer or field supervisor and can occur during or after performance. At the specified time, the local government's field or contract manager inspects or reviews the service with the contractor's representative. Because scheduled inspections allow the contractor to alert employees beforehand, the service is often an example of the best the contractor can deliver. This can be a benefit: problems identified during a scheduled inspection-when the contractor has presumably made a strong effort-indicate that the contractor may be misinterpreting contract requirements and should review them with the field or contract manager. Similarly, if the level of quality achieved for the scheduled inspection is higher than that noted in previous reports, it may indicate that the contractor is capable of improving performance, particularly when monitored closely. The contractor should be advised that the performance observed during the inspection is a welcome improvement and will be the new standard against which future work will be measured.

Random monitoringRandom monitoring is direct or follow-up inspection of the contractor's performance undertaken without the contractor's advance knowledge. Although random monitoring should not be relied on as the sole indicator of performance quality, it does provide a view of performance that is not usually obtainable through scheduled inspections. Another advantage of random monitoring is that it encourages the contractor to maintain acceptable service levels even when local government observers are not present. The sidebar on this page provides an illustration of random monitoring using "mystery shoppers."

Monitoring outcome

As noted earlier, outcome monitoring is the analysis of the results of a service. In a street maintenance contract, for example, the number of potholes repaired is a measure of output, whereas an outcome analysis might focus on the smoothness of the repaired roadway or on the reduction of hazards to pedestrians. As another example, social services contracts commonly examine recidivism rates, employment attainment, client satisfaction, and quality of life indicators as outcome measurements. The sidebar above provides more examples of outcome measures. Criteria for measuring outcome should be included in the contractor evaluation section of the original scope of work. The sections on surveys and hotlines later in the chapter are particularly pertinent to outcome monitoring.

Few contracts today are established without detailed performance expectations. However, local governments must exercise caution in the way these expectations are structured. Outcomes and performance expectations must be carefully structured so as not to stifle innovation with excessive rigidity or create incentives for the contractor that are at odds with the interests of clients. The sidebar on page 79 highlights some strategies for avoiding these risks.

Documenting performance

Because no single method of data collection can present a complete picture of a contractor's day-to-day performance, objective evaluation of performance must be based on information received from several sources. A random inspection, for example, may provide a snapshot view that does not necessarily represent average performance. A complaint may be groundless or unrelated to actual service delivery. Contractor reports and deliverables required under the contract may or may not be reliable.

This section discusses a number of approaches to gathering and recording information on performance-from forms to diaries, surveys, and hotlines. Whatever the source of information, the goal of the contract administration team is to tie together all available information to create a comprehensive picture of a contractor's performance. The forms and techniques described can be adapted for use in any service contracting program.

Mystery shoppers

To ensure that contractors' interactions with clients fulfill contractual performance requirements, Genesee County Community Mental Health (GCCMH) in the state of Michigan uses a "mystery shopper" program to randomly monitor its contractors. The Customer Services unit of GCCMH enlists the help of client volunteers who are provided with training and a set of protocols for randomly assessing how the contractor responds to client requests for help. The client is assigned a set of expectations to evaluate during a single encounter with the contractor. For example, the client may contact the agency by phone or visit the site in person and make a request for service. The client may be asked to report on whether they were treated with respect; how long they were put on hold if they phoned; whether they were given an appropriate response to an emergent care request; and, if they presented a request in a foreign language, whether the contractor followed the appropriate protocol for accessing a translator.This program has been highly effective in helping GCCMH monitor contractors and involve clients in the process, and it keeps contractors "on their toes." Contractors find the feedback helpful and use it as a performance improvement tool.

What does an outcome measure look like?

Outcome measures assess the extent to which a program has achieved its intended results. The principal question in outcome measurement is: Has the program or service made a difference? Outcome measures should assess some aspect of the effect, result, or quality of a service. They are outcomes the public would likely be interested in, not merely the amount of work done or resources consumed. Typically, outcomes are broad program goals that are measured through a series of measurable indicators. Some examples include:

Outcome: Timely access to services

Indicator 1: Intake appointments scheduled within 48 hours of initial request for services

Indicator 2: Clients assigned a caseworker within 7 days of initial intake

Indicator 3: After-hours emergent care calls answered within 15 minutes

Outcome: Economic revitalization

Indicator 1: Number of new businesses start-ups

Indicator 2: Reduction in unemployment rate

Indicator 3: Clearance of 25 percent of blighted land area

Outcome: Client satisfaction

Indicator 1: Number of complaint calls received per month

Indicator 2: Number of repeat complaint calls

Indicator 3: Average client satisfaction rating of service on annual survey

Source: Adapted from Compassion Capital Fund National Resource Center, Measuring Outcomes (Washington, D.C.:

National Resource Center, 2005), available at www.acf.hhs.gov/programs/ccf/resources/gbk_pdf/om_gbk.pdf.

One important note to remember: good record-keeping procedures are as essential to service contracting as they are to any other local government function. For example, any written material relating to a contract-a memorandum issued by the field manager or contract manager, for example-must be forwarded to the contract administrator and stored in the master file for that contract. Similarly, if the contract administrator receives a document that affects the contractor's performance or amends the contract documents, the administrator should forward a copy to the field manager for information and reference.

Overcoming the problems of performance contracting

Specifying the type and level of performance expected from contractors can go a long way toward structuring the monitoring process, but it is not a foolproof method. Performance contracting has several downsides, including the fact it may inhibit contractors' creativity, stifle overachievement, and encourage innovation in cost-cutting but not in service delivery. Robert Behn and Peter Kant offer eight strategies for avoiding these and similar pitfalls associated with performance contracting:

1. Link performance measures that are used to monitor contractors to the local government's mission. This reminds the contract administration team of the underlying purpose of contracting.

2. Create contracts based on outputs that are linked to mission; are easy to measure, understand, and reproduce; and facilitate benchmarking.

3. Start with measures that are simple and adjust the complexity with time and experience.

4. Monitor many indicators of performance and monitor them frequently

5. Be prepared to learn, change, and adapt. Use knowledge gained through the process to
improve future versions of the contract as well as new contracts.

6. Work collaboratively rather than adversarially with contractors.

7. Pay contractors not just for final outputs but also for significant and well-defined progress.

8. Choose contractors with a track record of desirable performance.

Source: Adapted from Robert D. Behn and Peter A. Kant, "Strategies for Avoiding the Pitfalls of Performance Contracting,"
Public Productivity and Management Review 22, no. 4 (June 1999): 470-489.

Contract administrator's checklist The contract administrator's checklist (Exhibit 1 at the end of the chapter) lists the licenses, certificates, bonds, and insurance and the reports, forms, and other deliverables required of the contractor during the contract term. The checklist is prepared by the contract administrator at the beginning of the contract term and is reviewed at specific intervals-anywhere from monthly to annually-to ensure that all items are current. Although the contractor's failure to provide the required items on time is a factor in the evaluation of performance, the checklist is more an aid for the contract administrator than an evaluation form in the strict sense.

Observation record An observation record (Exhibit 2) is used to document observations of contractor performance, primarily by the field manager but also by the contract administrator and contract manager whenever they observe service delivery.

Complaint record The complaint record (Exhibit 3) is used to record and follow up on complaints received from citizens or local government clients. To permit easier tracking of a complaint from receipt through final disposition, a separate form is used for each complaint.

Discrepancy report A discrepancy report (Exhibit 4) is used to document a condition that is or could lead to a breach of contract. It is prepared by the field manager or contract administrator, approved by the contract manager, and forwarded to the contractor for corrective action with a cover letter if necessary. If the contract manager thinks that a written response is necessary, he or she recommends in the discrepancy report that the contractor respond to the complaint in writing. A copy of the original complaint report may also be sent to the contractor with the discrepancy report. If the local government does not have an official discrepancy report form, a simple memorandum citing the discrepancy should be issued to the contractor and a copy placed in the contract file. The key is to put the problem in writing so that it becomes part of the performance history.

On the discrepancy form, the contract manager requires the contractor to respond within a specified time. After reviewing the contractor's response, the contract manager decides whether further action should be taken to prevent recurrence of the problem. A copy of the completed form is then returned to the contractor with a cover letter expanding on the entries, if necessary.

Depending on the nature of the discrepancy, it may be appropriate to ask the contractor to submit a brief corrective action plan. The corrective action plan does not need to be lengthy but should address why the discrepancy occurred and describe action taken to prevent future discrepancies.

Summary evaluation report The summary evaluation report (Exhibit 5) is a brief evaluation of performance during a given period. It should be prepared 30 or 60 days after contract start-up and every 90 to 120 days after that. Summary reports are prepared more frequently when performance threatens to fall below contract standards.

The report is prepared by the contract manager using information received from all available sources during the evaluation period (personal observations, team member observation reports and memoranda, complaints received, etc.).

Contract status reportDepending on the type of service, the local government may require status reports from contractors on a routine basis as part of the performance monitoring process. If the local government is paying the contractor for the service fully or in part through federal funds, status reports from contractors are probably required. A monthly narrative status report prepared by the contractor may fulfill a grant requirement or provide management or the governing body with information on contracts of special interest to them. All reports are reviewed thoroughly by the contract administration team and the affected department for completeness and accuracy.

Do not require a report without a valid reason. Reporting is not necessary or practical, for example, for contracts for trades services or any short-term personal service or consulting contract. When in doubt about whether to include a reporting requirement, require a quarterly report "at the discretion of the contract manager." It is much easier to waive an existing reporting requirement than it is to ask for a report that was not required in the contract.

Establish an outline of the topics or areas that must be addressed in any contract status report required and let the contractor choose the response format. The outline may include items such as the following:

· Specific tasks, units of service provided, number of clients served, or deliverables provided since the last reporting period and a summary of activities to date

· Problems encountered in service delivery or in adhering to budget or project schedules and how they were resolved

· In cost reimbursement contracts and contracts under which work is to be completed in phases, a comparison of the progress to date with targeted and actual contract dollars spent

· A forecast of tasks, deliverables, or services to be provided during the next reporting period

· An assessment of the quality of cooperation received from the local government staff

· Recommendations for improvements in service delivery, level of local government involvement, and so forth.

Field diaryField notes include information on weather conditions, favorable and unfavorable observations of contractor performance, and records of telephone and face-to-face conversations with the contractor. Because the observations recorded in field notes are included in more formal documentation of performance such as discrepancy reports and evaluation forms and may be required to justify the assessment of liquidated damages or to support legal action against the contractor, it is essential that they be accurate.

One way to help ensure accuracy and protect the integrity of field notes is to record them in a field diary. Traditionally, a blank hard-covered book has served this purpose. Increasingly, construction inspectors rely on electronic systems to record observations in the field. Specialized software such as Field Manager allows inspectors to record inspection details on an electronic reporting pad while in the field and later transfer the information to the local government's main computer system.[2]

The field diary provides a dated record of each inspection, observation, and problem encountered during the contract term. Every entry must be accurate and defensible: the diary is not to be used to express opinions or repeat hearsay but to record events or activities directly observed. Like most contract administration documents, the diary may be a part of the public record under freedom of information laws.

The sidebar below lists recommended procedures and typical entries for a field diary. The field manager should use a separate diary for each contract. Contract administrators and contract managers, who have less frequent contact with individual contractors, do not have to use a separate diary for each contract. They can insert their observations and comments into a single diary, identifying the contract to which each comment applies. However, when monitoring highly visible or sensitive contracts, it is advisable for each member of the contract administration team to have a separate diary for each such contract.