Evaluating System Change Initiatives for Improving Youth Outcomes

No.3, October 2007

The Challenge:

Youth grantees and service providers are becoming more comfortable and adept at conducting

• impact evaluations, which measure ultimate youth outcomes such as improved high school graduation and youth employment rates, and
• process evaluations which generate data to be used for continuous improvement efforts over the course of the grant or program.

However, evaluating systems change, especially multi-systems change, is much more difficult because of the inherent complexity of systems and the long time frames (five to ten years) before impacts may become apparent. Consequently, identifying and selecting the indicators for measuring systemic change can be problematic.

Proposed Solution:

Identification of progress indicators and effective practices over the full course of the initiative are especially important. Funders and other stakeholders often cannot wait until final impact evaluations are complete to justify continued funding or support. Multiple data collection approaches will be needed from the early planning stages of the project to the dissemination of final results, and must be thoughtfully selected. System change indicators may be selected from six systems change focus areas identified by Westat: 1) capacity, 2) coordination, 3) customization (personalization of programs and services), 4) development/adaptation/evaluation of new practices, 5) dissemination of effective practice, and 6) sustainability of effective program elements.

Promising Practices

• The Annie E. Casey Foundation New Futures Evaluation http://www.cssp.org/publications/index.php
  This initiative was one of the first multi-systems change experiments for improving youth outcomes. Its experiences and lessons learned in evaluation, published in 1995, are still valuable today.
• Canadian Mental Health Service Systems http://www.phac-aspc.gc.ca/mh-sm/services_e.html
  This project evaluated 140 sites across Canada to identify innovations in health care delivery systems and related services between 1997 and 2001. Twenty-five sites focused on mental health care. Documents include best practices focused on client outcomes and system change, as well as accountability and performance indicators.

Who needs to act?

A full range of stakeholders should be involved in the design and implementation phases of evaluation including policymakers, agency heads, program and grant administrators, community members, families, and youth.

Action Plan

• Review the resources identified in this Short Cut in order to identify challenges and best practices in conducting multi-system change evaluations.
• Know where the interests of funders and potential funders lie and assemble a team of knowledgeable stakeholders to develop a realistic and comprehensive evaluation plan that addresses their needs. Involve an external evaluator if adequate expertise is not available among the stakeholders.
• Identify continuous improvement and impact indicators to be monitored throughout the course of the initiative for improving youth outcomes and effecting systems change. Initiative goals and the Westat indicators are good places to begin the identification of indicators.
• Determine data collection requirements. Accurate and timely data is critical for effective evaluations. If current systems cannot provide the necessary data, determine if they can be upgraded or if other sources of data are available. Developing new data collection system(s) is expensive and time-consuming and is not a decision to be made without full consideration of the costs, both financial and programmatic.
• Collect baseline data for critical impact and systems change indicators before beginning systems change interventions or program activities.
• Finalize your evaluation plan and begin implementation.
• Synthesize and report results. Create a process to reflect on and disseminate evaluation findings.

More Information

National Collaborative on Workforce and Disability for Youth & Westat. (2006). Improving outcomes for people with disabilities: Evaluating system change. Washington, DC: Authors. Available at [insert web address when document is loaded on Collaborative website].

Wertheimer, D.M. (2001, April). Evaluating progress. (The Change Agents’ Toolbox, No. 7). Alexandria, VA: National Technical Assistance Center for State Mental Health Planning. Available at http://www.nasmhpd.org/general_files/publications/ntac_pubs/toolbox/agent7.html

Shaw, K. Challenges in evaluating systems change. (1995, Winter). The Evaluation Exchange, 1, 1. Available at http://www.gse.harvard.edu/hfrp/eval.html

This document was developed by the National Collaborative on Workforce and Disability for Youth, funded by a grant/contract/cooperative agreement from the U.S. Department of Labor, Office of Disability Employment Policy (Number #E-9-4-1-0070). The opinions expressed herein do not necessarily reflect the position or policy of the U.S. Department of Labor. Nor does mention of tradenames, commercial products, or organizations imply the endorsement by the U.S. Department of Labor.

NCWD/Youth | c/o Institute for Educational Leadership
4455 Connecticut Avenue NW, Suite 310 | Washington, DC 20008
Telephone: 1-877-871-0744 (Toll Free)
TTY: 877-871-0665 (Toll Free)