Furthermore, this evaluation may inform limited research of RJP program evaluation with a small, rural school district in a region with a prevalent cannabis cultivation industry. Findings from this project may help to fill the gap in the literature on restorative program evaluation for drug-related incidents in school-based settings. This may inform efforts for developing, implementing, and evaluating public school-based restorative programs with a substance use focus. Data from this project may also provide the grantee with crucial intermediate findings to inform the continuous development and optimization of RAYS program components and implementation strategies.The General Activity Log database was developed utilizing the SeaTable Cloud software service . UCSD researchers conducted training sessions with RAYS program coordinators where data recording protocols were reviewed to ensure accurate and efficient collection of activity data. Links between the GAL and the case management database were established to track participation in RAYS activities at the individual enrollee level. RAYS program coordinators recorded all relevant activity data in the GAL at the time of implementation or ideally within the same month the activity was implemented. Critical activity data for process evaluation included activity name, implementation date, number and type of attendees , engagement levels, vertical racking system and duration of the activity .The case management database was embedded in the same SeaTable cloud-based platform as the GAL.
Core components of the case management database that informed both process and outcome evaluation measures included data from the Enrollment Form, Exit Form, and the Demographics Questionnaire. Upon enrolling a student, a RAYS program coordinator was responsible for recording the reason for referral to the RAYS program, date of enrollment, student status, school information, point of youth diversion, participation status , and any notes from the disciplinary incident report. During this same meeting, students were prompted to complete the Demographics Questionnaire, which collected information on race/ethnicity, gender identity, and age. Once the student completed RAYS, a program coordinator would submit an Exit Form where they would document the RAYS activities that the student was initially assigned via their Restorative Plan, which ones they completed, reason for exiting the program , and date of the program exit. Individual data from the Enrollment Form, Exit Form, and Demographics Questionnaire all formed each enrollee’s Student Profile which was linked to any RAYS activities that a student participated in.To inform the outcome evaluation, pretest and posttest surveys were developed and administered to students to examine any changes in the enrollees’ sense of self-responsibility, past 30-day AOD use behaviors, perceptions of AOD use, and awareness of resources. Questions and scales were adapted from the 2019-20 California Student Tobacco Survey and the 2021-22 Mapping Youth Health Behavior Survey , both of which are population-based survey instruments developed and utilized by Professor Shu-Hong Zhu’s research team with dimensions in substance use and relevant covariables . Both pretest and posttest survey instruments were programmed and administered using the Qualtrics PlatformXM .
The pretest included approximately 35 questions on the above mentioned variables while the posttest included an additional 6 questions on student experiences in RAYS. Students took approximately 8-10 minutes to complete either survey. Participants were asked whether they had used marijuana , vapes with nicotine or just flavoring, alcohol, or opioids to get “high” in the last 12 months. Questions pertaining to opioid use were not added to the pretest and posttest instruments until Fall 2022; as such, data on opioid use behaviors and perceptions are not reported here due to the low number of responses. Utilizing a skip logic design, participants were asked to indicate past 30-day use of any products they said they had used in the last 12 months. Participants were also asked product-specific follow-up questions on frequency of use and intentions to quit any product they reported using in the past 30 days . All students were prompted to indicate their perception of the harm of using each substance “some days” and “every day” with a 5-point scale ranging from “not at all harmful” to “extremely harmful”. A 4-item scale on self-responsibility and personal awareness included questions derived and adapted from Mergler and colleagues’ personal responsibility scale for adolescents. A 4- item Likert scale was used to assess student awareness of resources. For the posttest instrument only, students were also prompted to assess their overall experience in RAYS via a 3-item Likert scale reflecting on RAYS components and likelihood of recommending the program to others. Additional open-text questions prompted students to provide feedback on what they liked and disliked about the program, as well as what they believed could be changed. RAYS program coordinators administered the pretest to students upon program enrollment, ideally prior to their first exposure to an intervention activity. The posttest was administered to students upon exiting the program . Each RAYS student was randomly assigned an alphanumeric passphrase upon enrollment which was linked to their student profile in SeaTable. Students were instructed to enter their assigned passphrase when taking the pretest and posttest surveys to allow for longitudinal linkage. Pretest and posttest responses were linked utilizing these passphrases in lieu of student names.
Prior to analysis, data was deidentified by reassigning each linked pretest and posttest response pair with a new passphrase to ensure student confidentiality and by removing the linkage to their respective student profiles. Suspension data was extracted from the CDE’s public data repository, DataQuest.19 Multi-year, aggregate reports on suspension counts were exported for each of the four school sites in Nevada County. Four comparable school sites were matched to the Nevada County sites and used as comparators. Discipline data reports from comparable sites were included in this study to examine any differences in the number of suspensions over time from Nevada County sites. Data was categorized into overall suspension and drug-related suspension counts to allow for the examination of changes in drug-related disciplinary incidents from the 2017-18 to 2021- 22 academic year.Data from the SeaTable activity log and case management databases for the reporting period May 2021 to January 2023 were exported and converted to Microsoft Excel® files. Activity data was cleaned, filtered by implementation date, and tabulated to report the number of activity exposures by attendee type . Case management data, including demographics and enrollment and exit data, was cleaned and relinked to passphrase-matched enrollee profiles. Suspension data for Nevada County schools and matched comparable sites was examined for changes in overall and drug-related suspensions from the 2017-18 to 2021-22 academic years. Comparable school sites from a neighboring county were identified utilizing school-level data from CDE site profiles. Each Nevada County school was matched with a comparable site based on enrollment size, racial/ethnic breakdown, and regional proximity. For reference, during the 2021-22 academic year, all four Nevada County sites reported a cumulative enrollment of 3,001 compared to 2,900 at the comparable sites. No drastic changes in enrollment numbers were reported for either the Nevada County or the comparable sites from the 2017-18 to the 2021-22 academic year. 19 RAYS program coordinators approved the selection of these comparable sites. Differences in these suspension counts between Nevada County schools and comparable sites were examined to inform the evaluation of the potential impact of RAYS on the number of disciplinary incidents over time. Pretest and posttest survey data was exported and converted to Microsoft Excel® files from the Qualtrics PlatformXM® online survey database. Using the randomly assigned passphrases, indoor grow facility each respondent’s pretest and posttest data was linked to analyze behavioral outcome changes from pretest to posttest. Descriptive analytical methods were employed to tabulate counts and percentages for each question response option at pretest and posttest. To evaluate changes in enrollee knowledge, perceptions, and behaviors, the percentages of participants who selected each response option for each question were compared from pretest to posttest. All percentages reported are of the total pretest and posttest survey sample . RStudio© statistical software was used to conduct all analyses while Microsoft Excel® was used to tabulate all activity and case-level data.Table 4 provides a breakdown of all RAYS activities implemented from May 2021 to January 2023. Data in each column presents the total number of exposures for each activity by attendee type – student, staff, administrator, parent, and “other”. The majority of student exposures was through informational presentations as these events reached a broader range of students, not just the youth advocates and students enrolled in RAYS. Exposure counts are reported in place of participant counts as some individuals may have attended multiple sessions or events of the same activity type. Therefore, the counts presented in Table 4 may reflect duplicate exposures to a specific activity.The findings reported here should be interpreted with caution given that these results are from an intermediate evaluation.
All findings are preliminary and should not be considered as a comprehensive assessment of the RAYS program. Additionally, due to a small pretest and posttest survey sample size, data may not be inclusive of all students who participated in the RAYS program. Overall, a total of 21 out of the 48 participants who exited RAYS during this evaluation period submitted a matched pair of pretest and posttest survey responses, equating to an approximate 43.75% response rate. Discussions with program coordinators revealed unanticipated logistical challenges with ensuring all students who exited RAYS submitted both pretest and posttest survey responses. Nonetheless, current protocols are being revised to increase pretest and posttest survey response rates. Despite these limitations, the findings reported here may provide insight into the potential individual-level impacts of the RAYS program on select variables of interest.Differences in student responses to questions on self-responsibility and personal awareness from pretest to posttest are found in Table 5. The proportion of students who strongly agreed or somewhat agreed with statements on self-responsibility did not significantly change from pretest to posttest. There was a slight increase in the percentage of students who agreed that before they do something, they think about how it will affect the people around them; however, agreement levels for other statements remained the same or did not change drastically.Table 6 presents the percentage of students at pretest and posttest who reported using a substance in the last 30 days. Out of the 21 students, 52.38% reported having used a marijuana product in the last 30 days at pretest whereas 38.1% said they had recently used marijuana at posttest. A similar reduction was seen with the proportion of students reporting past 30-day vape use, with 66.67% of students reporting using a vape with nicotine or just flavoring at pretest and 38.1% at posttest. There was a slight decrease in the percentage of students who said they had drunk alcohol, with 19.05% reporting past 30- day alcohol use at pretest and 9.52% at posttest.To measure changes in awareness of resources prior to and after going through RAYS, students were prompted to indicate how much they agreed with statements on identification of services and resources at their school. Table 8 reports the proportion of students who either somewhat agreed or strongly agreed with statements on student self-efficacy of being able to identify mental health and substance use services. There was a slight increase in the percentage of students who believed they could name at least one person or place that they could go to for support with mental health-related issues with 90.48% saying they could at pretest and 95.25% at posttest. Awareness of support or resources for substance use-related issues also increased, with 71.43% saying they would be able to name a place or person at pretest and 90.48% at posttest. When asked if they would be able to refer a friend or classmate to such services, 71.43% said they would be able to at pretest and 85.71% at posttest.To assess overall student experiences in RAYS, participants were prompted with both quantitative and open-text questions at posttest. Table 9 presents the proportion of students who either somewhat agreed or strongly agreed with statements regarding RAYS. Of the 21 participants who exited RAYS between August 2021 and January 2023, 80.95% believed that the program helped them to think about how their substance use affects others. The majority of students found the resources provided through RAYS to be available when they needed them. Additionally, 80.95% said that they would recommend the program to others.Students were also asked what they liked and disliked about RAYS and what they would change about the program via open-text questions. Overall, enrollees expressed their appreciation for the education received through RAYS, with some students highlighting their preference for substance use education in lieu of traditional forms of punishment.