To answer these questions in the conceptual model, we used the experience of the projects' PIs, literature sources, input from youth development experts, and six still-running youth development programs. Roads Taken research team first created a model of influential factors and how they lead to the outcomes of interest. Short and Intermediate Outcomes assessed involve the nature of program experience, social-emotional development (SED), and STEM-ID post-program. Long-term outcomes include academic degree, career, community involvement with STEM, personal involvement with STEM, external context, and program influences on how alums responded to the COVID challenge.

Two hundred people who had participated in these programs between 1990-2005 responded to a questionnaire asking them about their experience with each of factors in the model. Results and descriptions of both structural equation modeling and fuzzy set qualitative comparative analysis (fsQCA) of questionnaire responses can be found under the "Findings and Conclusions" tab below.

Alum Liaisons collected about 135 resumés and LinkedIn profiles to extend our understanding of academic and career patterns. Coding for these documents identified academic patterns (e.g., number and types of higher education experiences) and career patterns (e.g., moving to positions of increasing responsibility and career changes). It also included additional information on STEM and non-STEM fields that former program participants have entered.

December 2022 – April 2024;.
In this phase, we collected qualitative data to clarify and deepen our understanding of the Pathways identified in Phase 1. We collected data from 30 former program participants who submitted questionnaires in Phase 1. Our intent is to understand how youth program experience influenced the roads these people have taken to reach where they are today. This research phase uses a case study methodology using qualitative methods for data collection and analysis. This method allows researchers to elicit narratives from participants to provide rich and meaningful explanations of the program and its impact on their lives.

With each respondent, two interviews were conducted, recorded, and transcribed. Each respondent received a stipend for participation in these interviews. In the initial interview, we collected general background information to clarify questionnaire findings. We also introduced the purpose and format of Photo Journals, which participants were asked to complete prior to their second interview.

We asked them to submit ten photos referencing the youth program's effect on their current lives. The photo journal protocol prompted respondents to consider program impacts on their work life, community engagement, and personal decision-making. Data analysis involves coding recorded interview transcripts and documents to develop case studies for each respondent.

Phase 1. Questionnaire Findings
We analyzed the questionnaire data with two distinctly different methodologies. Following our original plan, we used confirmatory structural equation modeling to learn statistically about the most predictable pathways to STEM-related outcomes and the relative influences of each factor. To accommodate for the smaller-than-expected sample size and consequent low power, we augmented these analyses with Fuzzy Set Qualitative Comparative Analysis (fsQCA).

To estimate our structural equation models, we used the partial least squares estimator to accommodate composite rather than latent variables. In other words, typical SEM estimators measure the effect of latent variables, i.e., unmeasurable conditions that cause measurable responses. In contrast, the PLS-SEM approach allows for composite variables, a different type of unmeasured variable. Rather than causing measured indicators (as is the case with latent variables), these composite variables (e.g., program experience) emerge from combining them (e.g., perception of specific program strategies create the program experience; they are not caused by the program experience).

Click here for the results of our PLS-SEM analyses.

fsQCA uses Boolean algebra to identify multiple pathways to outcomes achieved—without having to model them a priori. The advantage of this method over structural equation models is that variables (or conditions as they are called in QCA) highly influential in one pathway but detracting in another, while averaging out to no significance in an estimated structural equation model, can be highly influential in QCA models. The disadvantage is that the degree of contribution of any single condition cannot be determined. fsQCA results are currently being analyzed next to SEM results, with analyses available June 2024.

Conditions used in our fsQCA models can be found at this link.

Phase 2. Interview results will be available in April 2024.

Triangulation across the three methods and answers to our research questions will be available in June 2024.

March 2020 – September 2024 and beyond
The Road Taken project includes efforts to provide information to stakeholders through methods that are accessible and useful. We target three primary audiences: researchers, practitioners, and funders of out-of-school time programs. Aims for each of these audiences include:

• Researchers – use research reports and journal articles to design or interpret their research and contact the research team to network ideas.
• Practitioners –make better informed (evidence-based) design decisions about OST STEM programs and use Road Taken findings in funding proposals.
• Funders – Make better informed/evidence-based decisions about funding OST STEM programs for underserved audiences.

The Association of Science and Technology Centers (ASTC) and the Afterschool Alliance participate as the Roads Taken dissemination partners. ASTC includes many STEM informal learning program practitioners among its institutional members. The Afterschool Alliance serves researchers and practitioners throughout the OST landscape. Dissemination activities began in the timeframe of Phase 1 of the project to raise awareness of the project. Lessons learned, and conference presents are available through the Dropbox link on the Homepage of this website. Also, we shared project information on Informalscience.org to raise awareness and share information about our research. We have intensified our dissemination efforts during the Phase 2 timeframe. We developed a website and social media pages on Facebook and Instagram. We decided to include our web pages as part of the CRE/COSI website to ensure that our findings in various formats would be available after project funding. The Facebook and Instagram presence raises website awareness, recruits users, and connects audiences to detailed information relevant to their use. These pages also create ways for the audiences to network with one another. Researchers will develop journal and newsletters articles to reach our target audience as we move ahead. In the last year of the Roads Taken project, we will focus on developing a report that includes both methods and findings. The project team will excerpt parts of this report into brief one-page summaries for informal learning researchers, program practitioners, and funders. Also, each partner institution will receive a summary of project findings and data analysis from respondents at thein institution.