The case management database was embedded in the same Sea Table cloud-based platform as the GAL. Core components of the case management database that informed both process and outcome evaluation measures included data from the Enrollment Form, Exit Form, and the Demographics Questionnaire. Upon enrolling a student, a RAYS program coordinator was responsible for recording the reason for referral to the RAYS program, date of enrollment, student status, school information, point of youth diversion, participation status , and any notes from the disciplinary incident report. During this same meeting, students were prompted to complete the Demographics Questionnaire, which collected information on race/ethnicity, gender identity, and age. Once the student completed RAYS, a program coordinator would submit an Exit Form where they would document the RAYS activities that the student was initially assigned via their Restorative Plan, which ones they completed, reason for exiting the program , and date of the program exit. Individual data from the Enrollment Form, Exit Form, and Demographics Questionnaire all formed each enrollee’s Student Profile which was linked to any RAYS activities that a student participated in.To inform the outcome evaluation, pretest and post test surveys were developed and administered to students to examine any changes in the enrollees’ sense of self-responsibility, past 30-day AOD use behaviors, perceptions of AOD use, and awareness of resources. Questions and scales were adapted from the 2019-20 California Student Tobacco Survey and the 2021-22 Mapping Youth Health Behavior Survey , 4×4 grow tray both of which are population-based survey instruments developed and utilized by Professor Shu-Hong Zhu’s research team with dimensions in substance use and relevant covariables .
Both pretest and posttest survey instruments were programmed and administered using the Qualtrics PlatformXM . The pretest included approximately 35 questions on the above mentioned variables while the posttest included an additional 6 questions on student experiences in RAYS. Students took approximately 8-10 minutes to complete either survey. Participants were asked whether they had used marijuana , vapes with nicotine or just flavoring, alcohol, or opioids to get “high” in the last 12 months. Questions pertaining to opioid use were not added to the pretest and posttest instruments until Fall 2022; as such, data on opioid use behaviors and perceptions are not reported here due to the low number of responses. Utilizing a skip logic design, participants were asked to indicate past 30-day use of any products they said they had used in the last 12 months. Participants were also asked product-specific follow-up questions on frequency of use and intentions to quit any product they reported using in the past 30 days . All students were prompted to indicate their perception of the harm of using each substance “some days” and “every day” with a 5-point scale ranging from “not at all harmful” to “extremely harmful”. A 4-item scale on self-responsibility and personal awareness included questions derived and adapted from Mergler and colleagues’ personal responsibility scale for adolescents. A 4- item Likert scale was used to assess student awareness of resources. For the posttest instrument only, students were also prompted to assess their overall experience in RAYS via a 3-item Likert scale reflecting on RAYS components and likelihood of recommending the program to others. Additional open-text questions prompted students to provide feedback on what they liked and disliked about the program, as well as what they believed could be changed.
RAYS program coordinators administered the pretest to students upon program enrollment, ideally prior to their first exposure to an intervention activity. The posttest was administered to students upon exiting the program . Each RAYS student was randomly assigned an alphanumeric passphrase upon enrollment which was linked to their student profile in SeaTable. Students were instructed to enter their assigned passphrase when taking the pretest and posttest surveys to allow for longitudinal linkage. Pretest and posttest responses were linked utilizing these passphrases in lieu of student names. Prior to analysis, data was deidentified by reassigning each linked pretest and posttest response pair with a new passphrase to ensure student confidentiality and by removing the linkage to their respective student profiles.Suspension data was extracted from the CDE’s public data repository, DataQuest. Multi-year, aggregate reports on suspension counts were exported for each of the four school sites in Nevada County. Four comparable school sites were matched to the Nevada County sites and used as comparators. Discipline data reports from comparable sites were included in this study to examine any differences in the number of suspensions over time from Nevada County sites. Data was categorized into overall suspension and drug-related suspension counts to allow for the examination of changes in drug-related disciplinary incidents from the 2017-18 to 2021- 22 academic year.Data from the SeaTable activity log and case management databases for the reporting period May 2021 to January 2023 were exported and converted to Microsoft Excel® files. Activity data was cleaned, filtered by implementation date, and tabulated to report the number of activity exposures by attendee type . Case management data, including demographics and enrollment and exit data, was cleaned and relinked to passphrase-matched enrollee profiles. Suspension data for Nevada County schools and matched comparable sites was examined for changes in overall and drug-related suspensions from the 2017-18 to 2021-22 academic years.
Comparable school sites from a neighboring county were identified utilizing school-level data from CDE site profiles. Each Nevada County school was matched with a comparable site based on enrollment size, racial/ethnic breakdown, and regional proximity. For reference, during the 2021-22 academic year, all four Nevada County sites reported a cumulative enrollment of 3,001 compared to 2,900 at the comparable sites. No drastic changes in enrollment numbers were reported for either the Nevada County or the comparable sites from the 2017-18 to the 2021-22 academic year. RAYS program coordinators approved the selection of these comparable sites. Differences in these suspension counts between Nevada County schools and comparable sites were examined to inform the evaluation of the potential impact of RAYS on the number of disciplinary incidents over time. Pretest and posttest survey data was exported and converted to Microsoft Excel® files from the Qualtrics PlatformXM® online survey database. Using the randomly assigned passphrases, each respondent’s pretest and posttest data was linked to analyze behavioral outcome changes from pretest to posttest. Descriptive analytical methods were employed to tabulate counts and percentages for each question response option at pretest and posttest. To evaluate changes in enrollee knowledge, perceptions, and behaviors, the percentages of participants who selected each response option for each question were compared from pretest to posttest. All percentages reported are of the total pretest and posttest survey sample . RStudio© statistical software was used to conduct all analyses while Microsoft Excel® was used to tabulate all activity and case-level data.The following sections report results from the RAYS pretest and posttest survey responses. The findings reported here should be interpreted with caution given that these results are from an intermediate evaluation. All findings are preliminary and should not be considered as a comprehensive assessment of the RAYS program. Additionally, greenhouse racking due to a small pretest and posttest survey sample size, data may not be inclusive of all students who participated in the RAYS program. Overall, a total of 21 out of the 48 participants who exited RAYS during this evaluation period submitted a matched pair of pretest and posttest survey responses, equating to an approximate 43.75% response rate. Discussions with program coordinators revealed unanticipated logistical challenges with ensuring all students who exited RAYS submitted both pretest and posttest survey responses. Nonetheless, current protocols are being revised to increase pretest and posttest survey response rates. Despite these limitations, the findings reported here may provide insight into the potential individual-level impacts of the RAYS program on select variables of interest.Differences in student responses to questions on self-responsibility and personal awareness from pretest to posttest are found in Table 5.
The proportion of students who strongly agreed or somewhat agreed with statements on self-responsibility did not significantly change from pretest to posttest. There was a slight increase in the percentage of students who agreed that before they do something, they think about how it will affect the people around them; however, agreement levels for other statements remained the same or did not change drastically.Table 6 presents the percentage of students at pretest and posttest who reported using a substance in the last 30 days. Out of the 21 students, 52.38% reported having used a marijuana product in the last 30 days at pretest whereas 38.1% said they had recently used marijuana at posttest. A similar reduction was seen with the proportion of students reporting past 30-day vape use, with 66.67% of students reporting using a vape with nicotine or just flavoring at pretest and 38.1% at posttest. There was a slight decrease in the percentage of students who said they had drunk alcohol, with 19.05% reporting past 30- day alcohol use at pretest and 9.52% at posttest.To measure changes in awareness of resources prior to and after going through RAYS, students were prompted to indicate how much they agreed with statements on identification of services and resources at their school. Table 8 reports the proportion of students who either somewhat agreed or strongly agreed with statements on student self-efficacy of being able to identify mental health and substance use services. There was a slight increase in the percentage of students who believed they could name at least one person or place that they could go to for support with mental health-related issues with 90.48% saying they could at pretest and 95.25% at posttest. Awareness of support or resources for substance use-related issues also increased, with 71.43% saying they would be able to name a place or person at pretest and 90.48% at posttest. When asked if they would be able to refer a friend or classmate to such services, 71.43% said they would be able to at pretest and 85.71% at posttest.To assess overall student experiences in RAYS, participants were prompted with both quantitative and open-text questions at posttest. Table 9 presents the proportion of students who either somewhat agreed or strongly agreed with statements regarding RAYS. Of the 21 participants who exited RAYS between August 2021 and January 2023, 80.95% believed that the program helped them to think about how their substance use affects others. The majority of students found the resources provided through RAYS to be available when they needed them. Additionally, 80.95% said that they would recommend the program to others.Students were also asked what they liked and disliked about RAYS and what they would change about the program via open-text questions. Overall, enrollees expressed their appreciation for the education received through RAYS, with some students highlighting their preference for substance use education in lieu of traditional forms of punishment. Some students also indicated that they enjoyed learning about drugs, what is and isn’t a drug, and how to identify harmful substance use behaviors. One student noted the information they learned through RAYS was “valuable” and that they could “take [it] with [them] in life” citing the program as a “second chance” to change their health behaviors. One student specifically mentioned how it “wasn’t just adults involved” but also other “kids going through things just like [them], with the same experiences that [they] have”. Enrollees also cited the people involved as important contributors to making the program helpful and enjoyable. Students described the individuals involved in administering and implementing RAYS as welcoming and open-minded, fostering a non-judgmental environment. One student noted how RAYS advocates and staff are “open and listen to what you have to say” and provide valuable support in helping the students obtain the resources they may need.The self-reflective nature of RAYS components was also brought up by multiple students. They noted how RAYS helped them to reflect on the mistakes that they had made in an educational rather than a punitive environment. These alternative to punitive approaches were highly regarded by students as something valuable and engaging. One enrollee said that they “liked that it gave [them] an opportunity to understand what [they] did wrong without just being taken out of school” while also giving them “the chance to take constructive criticism about [their] substance use”. Students also noted the community engagement piece in positive regards as something that allowed them to maintain a relationship with their peers and school community. When asked what they disliked about the program, most students noted that there wasn’t anything they felt that they strongly disliked or would like to see done differently. Nonetheless, a few enrollees cited aspects related to the logistics of program delivery and knowledge of peer advocates.