Preparing Schools for Responsible AI Use
Artificial intelligence tools are entering classrooms faster than most schools can develop policies to govern them. As schools prepare for the new academic year, school leaders should assess whether their policies, training and governance frameworks adequately address the growing use of AI by students, teachers and staff.
Protecting Student Data
AI-powered education technology tools collect enormous amounts of student data. Because a single vendor compromise can expose data from hundreds or thousands of school districts simultaneously, attackers are increasingly targeting education technology software suppliers and vendors rather than schools directly.
In addition, teachers and staff may use AI tools without fully understanding privacy considerations. This creates the risk that identifiable student information, such as names, grades, behavioral notes, or IEP details, is pasted directly into publicly accessible AI tools.
Steps to Strengthen Data Privacy and Security
Review and update vendor vetting processes to thoroughly evaluate AI vendors before approving their tools. The processes should enable every administrator to answer these questions:
- What student data is collected?
- Is student data used to train public-facing AI models?
- How long is the data retained?
- Which third parties have access to the data?
- What security protections are in place to prevent unauthorized access?
Use the information you obtain to incorporate data privacy, FERPA compliance, and cybersecurity requirements directly into procurement contracts.
Review and update the school’s incident response plan to account for vendor-related breaches involving AI tools. The plan should address how the team will proceed if an AI vendor experiences a security incident or data breach.
Provide teachers and staff clear training on acceptable use, including instructions never to input identifiable student data into publicly accessible AI tools.
Setting Boundaries for Student AI Use
Many schools lack clear policies defining the boundaries of AI use in the learning context.
Generative AI also makes it more difficult to distinguish between appropriate learning support (such as brainstorming, revising, and obtaining feedback) and conduct that may violate academic integrity expectations.
Steps to Protect Academic Integrity
- Adopt a clear, written policy on AI use that defines what is appropriate use from academic dishonesty.
- Require students to disclose their use of AI.
- Align consequences for AI-related academic integrity violations with existing disciplinary frameworks to promote consistent and fair enforcement.
- Communicate expectations clearly to staff, students and families before problems arise.
Managing Bias in AI-Assisted Decisions
AI systems learn from historical data. The quality and composition of that data can affect how a system performs across different populations. AI tools used for assessments, admissions, discipline, or student support services can potentially affect how certain student populations are evaluated or treated.
Steps to Mitigate Bias
- Request information from vendors regarding the AI tool's testing, validation, known limitations and performance across different educational settings and student populations.
- Maintain meaningful human review and oversight when AI-generated information supports decisions affecting students. AI should inform decisions, not make them.
- Carefully evaluate AI tools that generate recommendations, predictions, or risk assessments. Understand how the tools develop their output, which factors they consider and what limitations apply.
- Periodically assess AI tools for effectiveness, reliability and continued appropriateness for their intended purpose.
Building an AI Governance Framework
As AI tools become more common in educational settings, schools should continue to assess how policies, training and governance should address AI use. Some states have adopted or are considering requirements related to AI literacy, governance and responsible use.
Even where not required by law, many schools and school districts have developed policies or governance frameworks to address issues such as acceptable use, privacy, security and risk management. Clear expectations and guidance can help support consistent and appropriate use of AI tools by students, faculty and staff.
Steps to Strengthen Policies and Training
- Form a cross-functional AI task force that includes teachers, IT and cybersecurity employees, legal or compliance officers and other relevant stakeholders. The task force should evaluate and oversee AI-related initiatives.
- Pilot new AI tools on a small scale before approving them for school or district-wide use. Gather structured feedback before a full rollout.
- Develop or refresh written AI policies and procedures covering staff use, student use, tool vetting and procurement and incident response.
- Provide ongoing, practical training and development for staff and students that addresses both the opportunities and the risks associated with AI.
- Revisit and update policies and training periodically.
- Monitor legal requirements.
Preparing Schools for Responsible AI Use
AI is not going away, and a blanket ban is not a realistic or productive strategy for most schools. As with any high-stakes technology tool, AI should inform decisions, not replace the judgment of the educators and administrators who remain accountable for them.
Schools that establish clear expectations for AI use, provide appropriate training, and periodically review their policies will be better positioned to address emerging issues as the technology and legal landscape continue to evolve.
If you have any questions about preparing for the upcoming school year, please contact Lucy Porter, Erin Malone or any member of Phelps’ education or artificial intelligence teams.