Setting Up Systems for Data Collection
School districts maintain comprehensive records of student educational data such as attendance, grades, and test scores. Some of these systems also include information about student 504 plans and IEPs; however, most do not additionally track information such as needs identified by assessments or screenings that do NOT result in Special Education enrollment. There are also limited options for recording student participation in services that would be helpful for the school team and others to understand.
As California advances toward better integration of student mental health, this should change. There are likely to be a proliferation of vendors offering modules or add-ons to support this tracking and communication. School districts should be mindful of ways to make this information accessible to the right people on the team without broadcasting sensitive information. A good system should also allow tracking of outcomes over time, as with IEP supports.
Sharing Information
As discussed in Sections 6 and 8, there are legal safeguards in place that prevent sensitive student mental health information from being shared without a justifiable purpose and/or appropriate consents. These protections are different in the education system, governed by FERPA, than in health and mental health, as stipulated by HIPAA. This is indeed one reason it can be challenging to engage meaningful collaborations across the sectors – hence silos. However, many LEAs and their partners are finding strategies to share student mental health information, either on an individual basis as allowed within the law for care coordination, on a population basis with aggregated student information, or using technology to match student health and academic data and then de-identify the data.
For individual-level data sharing, it is key to train all staff on HIPAA & FERPA. When staff understand the privacy laws, best practices and compliance procedures, and how it applies to them in their role and for their partner agencies, it helps tremendously with supporting strong partnerships. When there is not clear understanding, it can sometimes lead to perceptions that someone is purposely withholding information, or sharing without holding privacy and following their HIPAA or FERPA duties, which leads to mistrust, and ultimately impacts collaboration. Comprehensive and consistent training ensures that staff know how to navigate the complexities and that they are following responsible data sharing practices. It is also essential to have a clear MOU in simple language that helps ensure everyone has clarity and is reviewed regularly. See Section 6 for an overview and resources for Coordination of Services Teams, including how to navigate sharing student information in the context of these teams.
The Los Angeles Trust for Children’s Health, a nonprofit organization that supports school-based health centers/wellness centers in the Los Angeles Unified School District, has recently created a database (the Data XChange) that allows LEA & healthcare partners to submit student level information that then links health and wellness data with achievement factors such as attendance, grades and graduation rates, and then de-identifies the data so that individual-level impact can be measured without sharing individual level data across sectors. This is the first-in-the-nation initiative that was designed to interoperate with other systems and databases and includes such metrics such as number of visits, unduplicated patients, co-morbid conditions, demographics and differences between the services received by students and community members. The database also tracks services provided as well as benchmarks on key school-based health center performance standards, including risk assessments, well-child exams, BMI screening, chlamydia tests and depression screening. Use of the database’s elements allowed for a recently published study which found that visiting an SBHC is significantly associated with improved attendance over time, with more dramatic improvements for students with a mental health diagnosis. The authors go on to state that understanding how student use of school-based health centers impacts academic performance is essential for estimating potential academic and health returns on investment for school-based health services, particularly mental health.
Tracking Outcomes
Schools can track the progress of students receiving mental health services using a range of systems – from traditional pen-and-paper methods, to spreadsheets to online forms or surveys to more sophisticated Electronic Health Record (EHR) systems. Any partners from the healthcare field will have their own EHR system, and any services that are going to be submitted for billing, through LEA-BOP traditional Medi-Cal will need to be collected electronically. Many districts contract with EHRs or Third Party Administrators (TPAs) to support documentation and billing, such as Paradigm or Welligent. Smaller districts often use the Google suite (especially forms and sheets) to document each student’s initial assessment and progress notes, and conduct progress monitoring on the individual level. The clinical supervisor then references this data to improve quality of care during clinical supervision and providing input on treatment decisions.
All SBHCs run by Federally Qualified Health Centers and many others use their electronic health records and incorporate assessment and screening tools like the PHQ9 for depression, the GAD7 for anxiety, and the Youth Outreach Questionnaire to assess clinical improvement over time.
Program Evaluation
Evaluations come in many forms, ranging from those run by a team of external evaluators to researchers who collect and analyze data over a period of several years to simple data collection efforts by school mental health staff and partners. The scope of a school mental health evaluation depends on the resources available, the questions you want to answer, the demands of any outside funders, and other factors and competing priorities. Because resources are limited, schools implementing school mental health programs will eventually want to know that the school mental health investment is a good value.
The most important thing to remember as you develop your evaluation plan is that you need to create a plan that is realistic for your team. You don’t have to measure everything! In fact, without a sufficient budget and staff capacity you are likely to get overwhelmed if you try to document everything. Instead, it is best to check in with your team and community members and prioritize what matters most to them and make sure that staff are properly trained in order to effectively capture the data identified. In addition, these questions may help start conversations amongst you and your team:
- What is going to be the most compelling evidence for them that you are being effective?
- What are you required to track for your funders?
- What data are already being gathered (e.g., service delivery) that can tell your story?
- How can you collect other evidence in a way that is the least burdensome but the most likely to capture your outcomes?
It is also essential to involve students and families in the evaluation planning process, as in all other elements of mental health planning.
The following set of outcomes are important to track and would likely be valuable to all stakeholders (schools, county mental health agencies, funders, students and families):
- Improved academic performance
- Improved student behavior
- Improved school climate
- Increased teacher satisfaction and reduced turnover
- Increased parent participation in school activities
- Increased parent and student satisfaction
- Increased attendance
- Graduation rates
- Decreased suspensions and expulsions
- Improved student mental health outcomes such as reduced rates of students reporting depression and anxiety
- Increased student report of knowing how to access services if they have a mental health need
- Increased teacher report of knowing how to access services and supports for their students
- Increased rates of students identifying a supportive relationship with an adult on campus
- Decreased student report of loneliness
- Decreased rates of students experiencing suicidal ideation
There are four main steps to developing an evaluation plan:
- Clarify program objectives and goals
Think about the main things that you want to accomplish and how you have set out to accomplish them. Develop SMART goals–an acronym that stands for Specific, Measurable, Achievable, Relevant, and Time-Bound.
- Develop evaluation questions
Examples of questions include: How well was the program planned out, and how well was that plan put into practice? How many students were screened and what were the outcomes? How has behavior changed as a result of participation in the program? Are participants satisfied with the experience?
- Develop evaluation methods
After developing the questions that you want to answer in your evaluation, decide on which methods will best address those questions. This can be done using surveys, key informant interviews, focus groups, web analytics, and observations. You can consider using a mix of qualitative and quantitative methods in your evaluation.
- Set up a timeline for evaluation activities
It is important to create a timeline on when to conduct evaluation activities, especially when are directly tied to program activities so you can prepare the tools you will need to use such as surveys. Think about when you expect to complete each phase of the evaluation and who is responsible for meeting each deadline.
More in-depth information on types of evaluation (process, outcome, etc.) and the importance of using logic models for evaluation can be found in the resources below.
