Distinct Functions for Artificial Intelligence in Healthcare Systems
Improving AI Safety in Healthcare: The Importance of Separation of Concerns
In the rapidly evolving world of AI, the healthcare sector is increasingly embracing large language models (LLMs) to automate tasks such as clinical documentation, managing prior authorizations, and even steering clinical decisions. However, as these advanced systems are integrated into the high-stakes realm of healthcare, ensuring their safety, reliability, and ethical use becomes paramount.
One approach that is gaining traction is the Separation of Concerns (SoC), an engineering strategy that breaks complex systems into distinct, independent components, each responsible for a clearly defined task. In the context of LLMs, SoC allows different aspects of AI systems—such as data management, model reasoning, user interaction, and safety checks—to be designed, implemented, and maintained independently.
Enhancing Safety and Ethical Use
By separating model reasoning from clinical validation components, SoC allows for mandatory human-in-the-loop checks and bias-aware workflows, ensuring outputs are verified before clinical use and reducing risks to patients and clinicians.
Improving Maintainability and Scalability
Distinct components for data ingestion, clinical context understanding, reasoning, and output generation can be updated or improved without disrupting other parts, facilitating continual refinement and regulatory compliance.
Supporting Trust and Transparency
Modular design makes it easier to audit specific parts of the AI system, improving transparency and fostering trust among clinicians and patients.
Mitigating Risk through Clear Protocols
SoC enables the implementation of structured harm reduction strategies by clearly defining roles and responsibilities within the system and aligning processing steps with institutional policies and safety guidelines.
In practice, SoC in AI systems involves using software engineering methods (APIs, programming language-driven AI control, templates) to separate concerns such as state management, prompt engineering, and response formatting. This not only addresses challenges like unpredictable AI responses but also improves robustness and reproducibility in clinical applications.
However, even with these measures in place, LLMs may still perform well in testing environments but fail catastrophically in production. To mitigate these failures, it's necessary to build independent systems outside the model itself that audit LLM outputs, detect performance drift, and revalidate the vector embeddings driving their "knowledge."
A living rulebook that evolves alongside the technology is essential for these systems. It's also crucial to establish independent audit systems, involving professionals beyond the core team who can audit model outputs, embedded data sources, and prompt structures. Checkpoints that automatically trigger reviews should be embedded in the system, such as after every model update, when outcomes deviate from clinical baselines, and upon changes to clinical guidelines or billing codes.
Co-creation, co-constraint, and co-correction are necessary for building systems jointly with clinicians, patients, and regulators from the outset. Co-correction means establishing mechanisms for rapid, cross-functional intervention when errors emerge, without waiting for retraining cycles. Co-constraint means defining explicit boundaries where the model cannot operate.
Every LLM-based system in healthcare now needs an LLM-sitter, a dedicated monitoring layer tasked not with tracking uptime, but with actively testing for policy deviations. The people using these models often don't notice the error until the damage is done.
Amber Nigam, the CEO and cofounder of basys.ai, a Harvard-based company that streamlines prior authorization for health plans using generative AI, has shared lessons about building agentic AI systems at MIT, Stanford, and TED. She has been recognized by groups like Rock Health's Top 50 in Digital Health and 40 Under 40 in Healthcare by the Boston Congress of Public Health. Basys.ai is supported by organizations like Mayo Clinic and Eli Lilly, both as collaborators and investors.
In conclusion, employing Separation of Concerns in healthcare LLMs improves the reliability, safety, and ethical use of AI by enabling modular, verifiable, and auditable systems that are easier to manage and align with clinical standards and policies. This leads to enhanced patient safety, better clinician trust, and more effective integration of AI tools in health care practice.
Amber Nigam, a renowned figure in AI and healthcare, has emphasized the importance of building agentic AI systems with co-creation, co-constraint, and co-correction, ensuring their alignment with clinical standards, policies, and ethical practices.
Embracing technology such as artificial-intelligence backed by Separation of Concerns can significantly improve healthcare by promoting safer, more reliable, and ethically sound AI systems, thereby making healthcare more trustworthy and improving the integration of AI tools in the medical field.
