Evaluation

This section includes 20 resources on school accountability and afterschool program outcomes. information on methods of evaluation, data-collection procedures, standards, and testing is provided.

Afterschool Initiative: Evaluation (New)

Sabrina Arrendondo Mattson

This resource presents the program evaluation findings of the Colorado Trust's 5-year, $11-million afterschool initiative. This initiative began in 2000, provided support to more than 30 afterschool providers in Colorado, and served more than 12,000 students in grades 4–9. The National Research Center conducted the independent program evaluation and found that the participating youth reported improvements in their positive life choices, sense of self, core values, cultural competency, life skills, community involvement, and academic success. Chapters on the program evaluation methodology, results, conclusions, and recommendations are included. (48 pages)

© 2005 Web Resource http://www.coloradotrust.org/repository/publications/pdfs/ASI/ASI-EvalAssessmentreport9-05.pdf

The Colorado Trust 1600 Sherman St. Denver, CO 80203 Phone: 888-847-9140 Fax: 303-839-9034 E-mail: receptionist@coloradotrust.org Web: http://www.coloradotrust.org

Analyzing Qualitative Data

Ellen Taylor-Powell and Marcus Renner

This guide explores the many ways of examining narrative data, which is often referred to as content analysis. It describes the analysis process, which includes the following steps: getting to know the data, focusing on the analysis, categorizing the information, identifying patterns, and interpreting the data. The guide also discusses the pitfalls of the analysis process. (10 pages)

© 2002 Print $1.25 Web Resource http://www.uwex.edu/ces/pdande/evaluation/evaldocs.html

Cooperative Extension Publications University of Wisconsin 45 N. Charter St. Madison, WI 53715 Phone: 877-947-7827

Analyzing Quantitative Data

Ellen Taylor-Powell

This resource discusses various types of descriptive statistics that can be used to make evaluation data more understandable. These statistics include numerical counts or frequencies, percentages, measures of central tendency (mean, mode, and median), and measures of variability (range, standard deviation, and variance). The guide offers specific examples of each of these descriptive statistics and explains how to calculate them. It also shows how to rank and work with the data to allow evaluators to explore their findings further. (11 pages)

© 1996 Print $2.00 Web Resource http://www.uwex.edu/ces/pdande/evaluation/evaldocs.html

Cooperative Extension Publications University of Wisconsin 45 N. Charter St. Madison, WI 53715 Phone: 877-947-7827

Charting the Benefits of High-Quality After-School Program Experiences: Evidence From New Research on Improving After-School Opportunities for Disadvantaged Youth (New)

Elizabeth R. Reisner, Deborah Lowe Vandell, Ellen M. Pechman, Kim M. Pierce, B. Bradford Brown, and Daniel Bolt

This research focuses on improving afterschool programs and opportunities for youth. The paper examines the policy implications for recent findings from the Promising Program study, which explores how high-quality afterschool programs affect youth. Findings from the study suggest that plans for high-quality afterschool programming should span entire communities, uniting to engage the largest number of at-risk youth. Working alone, afterschool programs offer limited choices; therefore, youth look to other, sometimes less beneficial, settings for more desirable activities. The study finds that working together, afterschool programs can provide a wider array of opportunities for youth. Implications for policy planning are also included in this study. (16 pages)

© 2007 Web Resource http://www.gse.uci.edu/docs/PASP%20Charting%20the%20Benefits.pdf

Policy Studies Associates, Inc. 1718 Connecticut Ave. NW, Ste. 400 Washington, DC 20009 Phone: 202-939-9780 Fax: 202-939-5732 Web: http://www.policystudies.com

Collecting Evaluation Data: An Overview of Sources and Methods

Ellen Taylor-Powell and Sara Steele

This resource provides a general overview of the types and methods of evaluation data available. It groups the most common sources of evaluative information into three categories: existing information, people and pictorial records, and observations. The guide also discusses methods for the collection of evaluation information, including surveys, case studies, interviews, observation, tests, logs, and document analysis, and it outlines various types of evaluation instruments. (11 pages)

© 1996 Print $2.00 Web Resource http://www.uwex.edu/ces/pdande/evaluation/evaldocs.html

Cooperative Extension Publications University of Wisconsin 45 N. Charter St. Madison, WI 53715 Phone: 877-947-7827

Collecting Evaluation Data: Direct Observation

Ellen Taylor-Powell and Sara Steele

This publication discusses direct observation as a method of collecting evaluation data. The authors highlight situations when direct observation is useful, including when you want direct information or when you are trying to understand an ongoing behavior or process or unfolding event. The guide lists the program components that should be observed, including behavior, physical surroundings, and interactions, and it discusses what an evaluator should look for during observations and how to record what is observed. The publication also explores requirements and training methods for observers. The appendix includes examples of observation guides that are used to record information. (8 pages)

© 1996 Print $2.00 Web Resource http://www.uwex.edu/ces/pdande/evaluation/evaldocs.html

Cooperative Extension Publications University of Wisconsin 45 N. Charter St. Madison, WI 53715 Phone: 877-947-7827

Collecting Evaluation Data: Surveys

Ellen Taylor-Powell and Carol Hermann

This guide offers an extensive overview of conducting surveys to collect evaluation data. It discusses the initial planning that needs to take place before administering a survey, the situations for which a survey is an appropriate evaluation tool, and the process of choosing the most appropriate survey method (e.g., mail surveys, telephone surveys, face-to-face surveys). It also discusses the planning that needs to take place before conducting a survey, the actual implementation of the survey, ways to ensure a good response to the survey, and the interpretation of survey results if the survey response is low. The appendix includes extensive references and samples of telephone survey materials, press releases, and other documents. (27 pages)

© 2000 Print $5.00 Web Resource http://www.uwex.edu/ces/pdande/evaluation/evaldocs.html

Cooperative Extension Publications University of Wisconsin 45 N. Charter St. Madison, WI 53715 Phone: 877-947-7827

Documenting Progress and Demonstrating Results: Evaluating Local Out-of-School Time Programs

Priscilla Little, Sharon DuPree, and Sharon Deich

This brief provides afterschool practitioners with the techniques, tools, and strategies they need to improve their programs and track their effectiveness longitudinally. The authors also present information about many areas of evaluation so that programs can ultimately conduct self-evaluations. The guide is divided into four parts. Part 1 reviews the key issues for conducting a program evaluation, including the program's stage of development, the program's target population, and the information funders expect to glean from the evaluation. Part 2 looks at the elements of a logic model and explains how it can be a useful instrument in program design and evaluation. Part 3 examines the five-tiered approach to evaluation, which involves early planning, documentation of program services, clarification of the program's intent, modification of the program, and demonstration of program impact. Part 4 discusses how to creatively disseminate information about the evaluation to stakeholders. The appendix includes evaluation terminology and a review of common data-collection methods. (48 pages)

© 2002 Web Resource http://www.financeprojectinfo.org/Publications/OSTlocalevaluation.pdf

The Finance Project 1401 New York Ave., Ste. 800 Washington, DC 20005 Phone: 202-628-4200 Fax: 202-628-1293 E-mail: info@financeproject.org Web: http://www.financeproject.org

Field Guide to Nonprofit Program Design, Marketing, and Evaluation, Fourth Edition (Revised)

Carter McNamara

This updated reference provides thorough, yet concise, guidelines for all critical aspects of a nonprofit program. It is written in an easy-to-implement style that includes 25 worksheets in an appendix and online. The highly practical resource can be used at any stage of development of the nonprofit program. For example, it can be used to conduct preparatory strategic planning, perform a market analysis, plan finances, develop credible fundraising proposals, and evaluate performance against goals and outcomes. (252 pages)

© 2006 Print $32.00

Authenticity Consulting 4008 Lake Drive Ave. North Minneapolis, MN 55422-1508 Phone: 800-971-2250 Fax: 763-592-1661 Web: http://www.authenticityconsulting.com

Measuring Youth Program Quality: A Guide to Assessment Tools (New)

Nicole Yohalem and Alicia Wilson-Ahlstrom with Sean Fischer and Marybeth Shinn

Quality assessment tools for afterschool programs are important for policymakers wanting to make sure funds are allocated to programs likely to have an impact and succeed. Assessment tools are also important at the practice level because they help practitioners mark what effective practices look like and allow them to assess and improve on their programs. This guide was designed to compare the purpose, structure, content, and technical properties of several youth program quality assessment tools. This resource provides useful guidance to practitioners, policymakers, researchers, and evaluators on what tools are available and under what conditions they would be most appropriate. (83 pages)

© 2007 Web Resource http://www.forumfyi.org/files/Measuring_Youth_Program_Quality.pdf

The Forum for Youth Investment 7064 Eastern Ave. NW Washington, DC 20011 Phone: 202-207-3333 Fax: 202-207-3329 Web: http://www.forumfyi.org

Out-of-School Time Evaluation Snapshot: Measurement Tools for Evaluating Out-of-School Time Programs: An Evaluation Resource (New)

Christopher Wilmer, Suzanne Bouffard, and Priscilla Little

This resource describes the academic, youth development, and prevention performance measures currently being used by out-of-school time programs to assess their progress and demonstrate results. It lists data sources for these measures and provides bibliographic citations, Internet links, and sample items from the different instruments. Practitioners can use this resource to help find an evaluation instrument that matches their program and needs. (29 pages)

© 2006 Web Resource http://www.hfrp.org/publications-resources/publications-series/out-of-school-time-evaluation-snapshots/measurement-tools-for-evaluating-out-of-school-time-programs-an-evaluation-resource

Harvard Family Research Project Harvard Graduate School of Education 3 Garden St. Cambridge, MA 02138 Phone: 617-495-9108 Fax: 617-495-8594 E-mail: hfrp@gse.harvard.edu Web: http://www.hfrp.org

Planning a Program Evaluation and Planning a Program Evaluation Worksheet

Ellen Taylor-Powell, Sara Steele, and Mohammad Douglah

This guide, organized into four major sections, helps groups plan program evaluations. The first section covers topics such as the purpose, target audience, and need for the evaluation. The second section discusses sources of information and data-collection methods. The third section addresses how the data will be analyzed, interpreted, and communicated to others. The final section examines time lines, the division of responsibilities, and budgets. Each section presents a series of questions and considerations for you to adapt to your own program needs and situations. The four-page worksheet describes each of the major sections of the guide and enables you to complete the information as it relates to your particular program evaluation. (31 pages)

© 1996 Guide $2.00 Worksheet $1.00 Web Resource http://www.uwex.edu/ces/pdande/evaluation/evaldocs.html

Cooperative Extension Publications University of Wisconsin 45 N. Charter St. Madison, WI 53715 Phone: 877-947-7827

Program Evaluation: Forms and Approaches, Third Edition (New)

John M. Owen

This resource provides recent research on evaluation and includes timely examples from worldwide sources. The text provides a comprehensive and useful overview of the evaluation process in a variety of settings, including the educational setting. The comparisons offer help in choosing and tailoring the appropriate techniques for any program. In addition, the book describes and presents examples based on the five major forms of evaluative inquiry: proactive, clarificative, interactive, monitoring, and impact. This edition includes a chapter on evaluation management and additional sections discussing negotiation theory, evidence-based practice, performance auditing and management, and realistic evaluation. (298 pages)

© 2007 Print $38.00

The Guilford Press 72 Spring St. New York, NY 10012 Phone: 800-365-7006 Web: http://www.guilford.com

Questionnaire Design: Asking Questions With a Purpose

Ellen Taylor-Powell

This resource assists in constructing an evaluation questionnaire and in completing an analysis of the type of data you want to collect from your questionnaire. In doing this, it examines knowledge, beliefs, attitudes, opinions, behaviors, and attributes. The publication offers advice on how to word questions and discusses the most effective questions to elicit the responses and data you need. The guide also shows how to most appropriately format and pretest a questionnaire. The appendix provides additional information on how to request attribute information (age, income, etc.). (20 pages)

© 1998 Print $3.00 Web Resource http://www.uwex.edu/ces/pdande/evaluation/evaldocs.html

Cooperative Extension Publications University of Wisconsin 45 N. Charter St. Madison, WI 53715 Phone: 877-947-7827

Sampling

Ellen Taylor-Powell

This evaluation resource provides guidance on conducting sampling. It first addresses whether sampling is an appropriate evaluation tool for a particular program. Then it presents sampling for generalizability, or probability sampling, and talks about specific sampling strategies, including simple random sampling, systematic sampling, and stratified sampling. The resource also explores sampling for other purposes, or non-probability sampling, including quota sampling and purposeful sampling. The appendixes list references, a table of random numbers, and recommended sample sizes. (12 pages)

© 1998 Print $2.00 Web Resource http://www.uwex.edu/ces/pdande/evaluation/evaldocs.html

Cooperative Extension Publications University of Wisconsin 45 N. Charter St. Madison, WI 53715 Phone: 877-947-7827

Selected Evaluation Terms

Priscilla Little

Evaluation terminology can be confusing at times. This resource provides commonly accepted definitions for terms used in the out-of-school-time field. It also discusses the difference between performance measurement and program evaluation, the difference between quantitative and qualitative data, and the main features and tradeoffs in study-design choices. (5 pages)

© 2002 Web Resource http://www.hfrp.org/publications-resources/browse-our-publications/selected-evaluation-terms

Harvard Family Research Project Harvard Graduate School of Education 3 Garden St. Cambridge, MA 02138 Phone: 617-495-9108 Fax: 617-495-8594 E-mail: hfrp@gse.harvard.edu Web: http://www.hfrp.org

Successful Program Evaluation (New)

Michael K. Wells

Successful Program Evaluation is designed to help grant writers create strong evaluation sections of proposals that may make the difference between getting funded or not. Helpful information is provided on topics such as what funders want, developing an evaluation plan, writing the evaluation section, and evaluation instruments. In addition, a chapter on resources and a comprehensive glossary are included. A well-developed and well-written program evaluation can strengthen an entire grant proposal. With this book, grant writers will have the tools necessary to take their grant development strategies to the next level. (142 pages)

© 2007 Print $29.95

Portland State University School of Extended Studies Continuing Education Press P.O. Box 1394 Portland, OR 97207-1394 Phone: 866-647-7377 Fax: 503-725-4715 Web: http://www.cep.pdx.edu/titles/program_evaluation

The ABCs of Evaluation: Timeless Techniques for Program and Project Managers, Second Edition (Revised)

John Boulmetis and Phyllis Dutwin

This updated resource shows how to select participants for an evaluation and how to deal with multiple goals and objectives, including those of an organization, its staff, and the client. The book describes different evaluation models, illustrates the circumstances under which each model can be used, and offers tips on identifying data sources and collecting data. Charts, graphs, models, and lists throughout the book help organize, extend, and facilitate the understanding of each evaluation concept. The authors, both professional educators and trainers, untangle the complexity of the evaluation process in a practical, accessible guide. (239 pages)

© 2006 Print $40.00

Jossey-Bass 989 Market St. San Francisco, CA 94103-1741 Phone: 800-956-7739 Fax: 415-433-0499 Web: http://www.josseybass.com

The Costs of Out-of-School-Time Programs: A Review of the Available Evidence (New)

Christianne Lind, Nanette Relave, Sharon Deich, Jean Grossman, and Andrew Gersick

Have you wondered about the cost of high-quality out-of-school-time programs or the cost of improving an existing program? This resource reviews the literature on this topic and provides key findings on various costs (startup, operating, capital, and infrastructure or system-building). Cost data from out-of-school-time program studies is included as well as a bibliography of resources on costs and quality. (28 pages)

© 2006 Web Resource http://www.financeproject.org/publications/litreview.pdf

The Finance Project 1401 New York Ave. NW, Ste. 800 Washington, DC 20005 Phone: 202-628-4200 Fax: 202-628-1293 E-mail: info@financeproject.org Web: http://www.financeproject.org

Using Research and Reason in Education: How Teachers Can Use Scientifically Based Research to Make Curricular and Instructional Decisions

Paula J. Stanovich and Keith E. Stanovich

This publication discusses three ways schools and teachers can provide evidence about the effectiveness of their instructional methods. Student achievement can be demonstrated by formal testing implemented by the teacher, school district, or state; by published findings of research-based evidence; and through proof of reason-based practice that builds on the research evidence. Each method has advantages and disadvantages. The authors encourage teachers to become more skilled independent evaluators of educational research. (42 pages)

© 2003 Print No Charge Web Resource http://www.sde.state.ok.us/Programs/ReadingFirst/resources/Stanovich_Color.pdf

National Institute for Literacy ED Pubs P.O. Box 1398 Jessup, MD 20794-1398 Phone: 800-228-8813 Web: http://www.nifl.gov/archive/pfr/about.html