OpenAI’s recent collaboration with the Pentagon on cybersecurity and veteran suicide prevention projects has sparked important conversations about the ethics and implications of artificial intelligence in critical, high-stakes domains. While OpenAI assures responsible development and a ban on weaponry, there’s one crucial consideration that deserves further attention: explainability.
When dealing with applications that carry life-altering consequences, opaque AI algorithms simply aren’t good enough. Take, for example, a predictive model informing veteran suicide risk. A black box churning out a binary “high risk” verdict without explanation is ethically unjustifiable. Imagine the immense psychological burden on the flagged individual, the potential harm of unwarranted interventions, and the erosion of trust in the system (see Dr. Pershing in Season 3 Episode 3 of the Mandolarian for an example of a patient receiving unhelpful care from an AI robot/droid).
Explainability transcends the “right to know.” It’s a moral imperative in critical applications. In healthcare, understanding why a diagnosis is reached guides treatment decisions. In law enforcement, knowing the reasoning behind suspect identification ensures fairness and accountability. These principles extend to the military, where AI-powered algorithms might contribute to targeting decisions or risk assessments (To be clear, DARPA has been thinking about these sorts of issues).
The stakes are simply too high to rely on blind faith in an algorithm’s output. We need models that not only deliver accurate predictions but also offer clear, human-interpretable insights into their reasoning. This allows for:
- Accountability: When wrong decisions are made, explainability facilitates tracing the error back to its source, enabling improvement and mitigating future harm.
- Building trust: Transparency fosters trust between humans and the AI systems they interact with, crucial for long-term acceptance and effective collaboration.
- Human oversight: Even with advanced AI, critical decisions ultimately lie with humans. Explainability empowers humans to understand the AI’s reasoning, challenge its conclusions, and ultimately make informed judgments.
Fortunately, advancements in AI research are paving the way for more explainable models. From feature importance analysis to bread and butter machine learning approaches, various techniques offer glimpses into the inner workings of algorithms. While challenges remain, the pursuit of explainable AI is not only a possibility, but a pressing necessity. I provide an overview of explainable AI in my textbook chapter, “An Overview of Explainable and Interpretable Artificial Intelligence”.
This blog post is just a starting point for a broader discussion. Do you think explainability is truly achievable in complex AI systems? How can we balance accuracy with transparency in critical applications? Let’s work together to ensure AI, with all its potential, is used for good, and explainability plays a key role in that journey.
Note: Bard was used to help write this article. Midjourney was used to help create the image(s) presented in this article.