Abstract:

The purpose of the project was the development of a “scorecard” or “blueprint” that could be used to improve the assessment of the effectiveness of academic outreach programs. The scorecard identifies the critical elements necessary for program effectiveness and meets methodological muster. In other words it meets both academic/researcher needs yet still meets the applied needs of funders, policy-makers, and practitioners. It adds information to the scholarly debate about academic outreach and college preparation while at the same time serves as a generic evaluative tool for all who want to verify the value or increase the impact of these programs.

The theoretical underpinning of the rests rests on nine propositions about academic outreach program activity:

1. Meaningful family involvement is critical to enabling students to get into a college or university.
2. Social activities are irrelevant to enabling students to get into a college or university.
3. Access to a college preparation curriculum is the critical variable for enabling students to get into a college or university.
4. College preparation programs need to begin no later than the 9th grade and have structured activities throughout the year in order to enable students to get into a college or university.
5. Having knowledgeable, available counselors that are at the core of the program is critical to enabling students to get into a college or university.
6. Mentoring is a helpful, but not critical, component for enabling students to get into a college or university.
7. Peer groups are a helpful, but not critical, component for enabling students to get into a college or university.
8. It is helpful, but not essential, for college preparation programs to emphasize the culture of the student in order to enable students to get into a college or university.
9. There is a positive relationship between the cost of program delivery and enabling students to get into a college or university.

By using these propositions and a largely qualitative approach to the work, the “richness of the tale”, not just numerical outcomes, emerge and with it, greater insight into the lives of the students and these programs.

The project focused on the University of California's Mathematics, Engineering, Science Achievement (MESA) program that for the last 40 years has helped to prepare underserved poor and minority students for academic and professional careers in math and science-based fields. It is a national K-16 model that has been replicated in at least a dozen other states in the US. While it has been cited numerous times for its success, e.g. as a winner of the Presidential Award for Excellence in Mathematics and Science Teaching and the Kennedy School/Ford Foundation's Innovations in American Government award, it had not previously been the subject of an externally-funded (U.S. Department of Education) evaluative study.

The results proved to be instructive formatively for MESA and the rubric based on the project is broadly applicable to other programs.

Keywords:

STEM, evaluation, self assessment rubric, academic outreach.