Abstract:

We often talk of college and career readiness but then focus solely on college preparation, paying less attention to workforce readiness. However, Project Lead the Way (PLTW) designs programs with both colleges and industry in mind. PLTW courses are followed by online exams. PLTW partnered with the University of Kansas (KU) as well as leaders from industry and institutions of higher education (IHE) to design these exams. The psychometric approach is overseen by a national technical advisory committee (TAC), and an independent consultant assists with validity studies and communicating the technical aspects of the exams to the public.

The team takes several steps to ensure the exams provide valid information about college and workforce readiness.
1: Embed industry and IHE professionals in each step of the test development process;
2. Develop inquiry-driving exams covering all aspects of science, technology, engineering, and mathematics (STEM).
3. Combine academic learning with practical knowledge and soft skills necessary for future success.

PLTW brought deans, faculty, executives, engineers, research scientists, data engineers, and more from multiple IHEs and industries such as NASA, FedEx, John Deere, and Eli Lilly to help develop blueprints, write and review test items, and set cutscores. They use their expertise to determine knowledge needed to enter their field and provide real-life examples of situations that can be turned into testable scenarios important to career learning and college preparedness.

An important component of this work is the focus on all aspects of STEM. While many programs address only science and math in secondary school, this program also includes engineering and computer science. Test items are grouped into task sets written to a scenario. In some tasks, students interact with computer simulations, manipulating variables and creating data tables, and then answer items measuring multiple standards.

In addition to academic skills, the exams also test transportable skills—competencies critical to success in college and the workforce and not specific to one subject. They include broad skills like problem solving, critical thinking, creativity, communication, and ethical reasoning. Situational judgment tasks—psychological measures that present test takers with hypothetical, real-world scenarios and ask what they would do—are embedded in the tested course (e.g., cybersecurity, medical interventions) that focus on skills such as communicating information, developing a creative solution, or behaving appropriately in groups. These competencies better differentiate students with similar subject-matter expertise on their preparedness for postsecondary success. Item writers develop the task, generate multiple responses, and assign points based on the effectiveness of each response.

The proposed presentation will have four components: design, psychometrics, technology, and validity. Industry needs and equity considerations drove much of the design and will be discussed more fully by a PLTW representative. A TAC member will describe how the item-cluster design with two skill types requires different psychometric modeling. The technology component, specifically the design of the simulations, will be discussed and demonstrated by a KU team member. Finally, the consultant will outline the validity evaluation with an emphasis on accessibility of simulations and equity issues for transportable skills items.

Keywords:

Online assessment, postsecondary readiness, computer simulations.