Inclusive Planning Guide Phase 4. Evaluate

Introduction

The following activities are necessary to measure performance and evaluate inclusive transportation planning efforts.

  • Have a Baseline: First Assess the State of Inclusion
  • Establish Performance Measures
  • Create a Model to Logically Link Activities with Results
  • Track Inclusive Performance Over Time
  • Analyze
  • Sustainability: Knowing When You Are There

Let’s look at each one in more detail.

Have a Baseline: First Assess the Status of Inclusion

One of the keys to evaluation is to see it as a continuous process that begins before new inclusive transportation planning efforts start. It is therefore useful to collect baseline data as you begin. This is how you will know where exactly you are starting. Visit the Start section, above, of the toolkit webpage for details about this critical step.

Establish Performance Measures

An inclusive strategic planning approach starts with setting goals, establishing objectives, deciding what performance to measure, collecting existing data, and determining the activities to conduct based on the collectively identified goals and objectives. Put in place a strategic plan that can be occasionally revised and that will set your inclusive transportation-planning project in a realistic and achievable direction. Involving participants and partners will enhance your ability to communicate about your work.

Components of a Performance Measurement System

Goals

These are a broad statement of the purpose of an initiative. They may take several years to achieve.

Objectives

These are incremental signs of progress towards the established goal. Make sure that objectives are specific and measurable. Measurable means that quantitative measures of progress in achieving the objective can be established. For example, measuring the percentage increase in the number of participants actively involved in inclusive transportation planning is a typical quantitative measure of progress. Qualitative measures such as measures of participant or partner satisfaction may also be acceptable, but should be summarized so results can be reported as a number, for example the percentage of participants who indicated a process was inclusive.

Activities

These answer the question, what will we do? Activities must align with the identified goals and objectives and must be realistic given the resources available (funding, staff, volunteers). The desire to change the world can often lead to unreasonable expectations, so an occasional reality check during the process is useful.

Inputs

What resources are being used? Inputs identify the funding, staff resources, volunteers and in-kind resources used to as part of the inclusive coordinated transportation-planning project.

Outputs

What will the activities produce? Typical outputs for inclusive transportation planning projects include the number of participants serving on advisory committees, number of partnering organizations and number of planning meetings held.

Outcomes

Provide numerical information documenting progress towards achieving project objectives and goals. The key question with an outcome is, how will conditions change as a result of the activities we are conducting? The link between program activities and outcomes observed can be complex and may be influenced by many things, so it is important to understand these linkages and to be realistic about the extent to which activities are responsible for changes being observed. Outcomes specific to inclusive planning practices include the number of participants assuming leadership roles, the percentage of participant transportation issues addressed or resolved, or the number of transportation improvements implemented. In the long run, one would hope to see more trips by the members of your target population[s] and higher quality transportation services.

Impacts

In the long run, what difference do you want to make in your community, or in the lives of people affected by your program initiatives? If your program can be sustained, the outcomes that your program is achieving on a regular basis will hopefully result in improvements for participants, partnering organizations and your community.

Customer Satisfaction

Measures should go beyond whether participants and partners are happy or satisfied. These measures are an opportunity to understand, what do participants and partners think? Examples of measures include participant satisfaction with the planning process, stakeholder/partner satisfaction with the planning process, percent of participants who felt their opinions had an impact and the percent of participants who were highly satisfied with leadership opportunities.

Targets

These are quantitative estimates of what will be accomplished by your project within a specific time period, typically a year. Longer-term target setting is great as long as the target is realistic and monitored regularly. Targets should be determined based on prior experience, such as a baseline measurement before activities started against which progress would be compared. Targets (as opposed to goals and objectives) must be quantitative measures of progress.

Records

It is critical to keep a written record of the goals, objectives, activities and performance measures. Don’t try to measure everything! Select measures that are meaningful for communication, decision-making and accountability. Measurement data should be collected regularly using a consistent, objective process that prevents reporting distorted or biased results. The key is to describe an objective method that someone else could use or audit to arrive at the same results. Reporting records should be openly shared with participants, partners and funding agencies.

Make note of conditions and issues that may have positively or negatively affected progress towards targets. A strategic inclusive transportation plan that is widely distributed and subjected to feedback will provide the framework for the work and a record of what you hope to accomplish.

Create a Model to Logically Link Activities with Results

A model is a great tool for communicating what your inclusive transportation partnership is trying to accomplish. The logic model links resources (inputs) to activities (what are you doing with the resources) to what you are producing (outputs) and to what you are accomplishing (outcomes). The logic model also will help you be precise in selecting relevant, useful performance measures. The W.K. Kellogg Foundation has some excellent resources on logic models. The Logic Model Development Guide does a nice job of explaining the relationship between different elements of a logic model.

In a logic model, one element is expected to lead to the next element in sequence. Logic charts are often used. Use arrows or other visual links to specific activities, outputs, and outcomes. Logic models often include a list of potential performance measures. Logic models may not be as effective in laying out the sequence of events that will lead to the changes your initiative envisions.

Monitoring Perceptions

In dealing with the diverse stakeholders represented in most inclusive transportation planning projects, attitudes and perceptions are important considerations. While producing outcomes is important, it can be a long-term process. Oftentimes, monitoring the perceptions of stakeholders to provide continuous information and feedback is the key to sustaining longer-term activities to change attitudes and improve services.

For example, simply asking if people with disabilities and older adults are satisfied with inclusion may be insufficient. It is critical to ask these stakeholders if others seek out their opinions and feedback and whether they feel that they are being heard. Collecting feedback is not just the right thing to do, but also a critical step in the inclusion process.

Track Inclusive Performance Over Time

  1. Set up a system to collect objective data. You’ll want to routinely and regularly collect measurement data. If you develop a survey to collect feedback on inclusiveness, use it regularly, such as at steering committee and other meetings attended by people with disabilities and older adults.
  2. Monitor results regularly. Regular monitoring, supported by detailed data collection, will help you see when things are going well, and when some of your project activities might need some attention to improve performance.
  3. Track both inclusiveness as well as program results. Inclusiveness is a means to improving performance, not just an end in and of itself. Continuously monitor your inclusive activities. Share the results with stakeholders to let people with disabilities and older adults know that you care about their involvement.
  4. Investigate irregularities in your results. Things don’t always go as you expect. If you get unexpected results when regularly collecting performance measures, look into it. Performance measurement produces regular feedback. Be sure to use the feedback to monitor programs and make decisions.

On a regular basis (quarterly or annually), consider the trends in your performance measures, and if appropriate, revise your approach. If you are greatly exceeding your targets, or having trouble reaching the targets, the problem may be target setting, not the results. Consider revising your targets, but don’t do it so often that the targets become meaningless.

Analyze

Analysis helps you assess if you’re doing the right things and doing them well. Analysis of performance data supports management decision-making. Such analysis can lead to the modification of policies, operational procedures, and processes. An added benefit is that the evaluation process will help you prove to others that you’re doing the right things and doing them well.

Periodically assess performance, measure progress to determine the achievement of goals and objectives, and to consider future actions. Be sure to identify reasonable expectations and interim milestones that can be pointed to as evidence of forward movement. These victories, often, will keep participants and partners engaged.

Agencies – at the state or local levels – may want to develop a system of collecting data regarding inclusive coordinated transportation planning practice that can be used by organizations as a self-scoring tool, or, at the state level, as a mechanism to enable organizations to compare themselves with each other.

Examples from the Field

Hopelink (Redmond, WA) established a process to continuously evaluate stakeholder participation and identify underrepresented.

People for People (Yakima, WA) built evaluation into sustainability plans to:

  • Determine current level of inclusive transportation planning in South Yakima County and how to improve it.
  • Continuously evaluate the public outreach process for successful participation of the target populations.
  • Evaluate how to sustain involvement of target populations.
  • Endorse and advocate project decisions and assist with fund raising.

Greater Portland Council of Governments (Portland, ME) developed draft recommendations using feedback collected at three months of steering committee meetings and focus groups.