The Informatics e-Learning Hub: Monitoring AI Algorithms

Importance of Testing and Monitoring of AI Models

By: Woojin Kim, MD

Even after FDA clearance and pre-market validation, ongoing monitoring is necessary to uphold the accuracy of any algorithm. Ongoing monitoring is essential to ensure the AI model doesn’t decay over time. This video reviews why it’s important to monitor your AI models and includes steps developers, vendors and radiologists can take to uphold their models.

In this video, radiologsits will learn:

The importance of post-implementation surveillance.
The definition and impacts of model decay.
How to define concept drift.
How to define data drift.
Steps for radiologists in evaluating AI algorithms.

Monitoring AI Performance in Clinical Practice

By: Bibb Allen Jr., MD, FACR

After evaluating and selecting an AI model, how can we be sure it will continue to work well in our practice? How do we know when a model breaks? This video explains the importance of continuous monitoring and how data collection can improve the performance of current models, as well as future versions.

In this video, radiologists will learn:

Common causes of model drift and decay.
How to use Assess-AI and AI-LAB™ for real-world performance monitoring.
How monitoring AI can help developers improve model performance.
The benefits of continuous learning.

Assess-AI and AI-PROBE: Monitoring Algorithm Performance in Clinical Practice

By: Axel Wismüller, MD, MSc, PhD

Assess-AI provides monitoring of algorithm performance in clinical practice by capturing real-world data during clinical use in a clinical data registry. This video introduces AI-PROBE (Artificial Intelligence Prospective Randomized Observer Blinding Environment) and how it can contribute additional study-specific information such as turnaround times, application specific outcome measures, and cost-effectiveness measures to the existing Assess-AI data repository.

In this video, radiologists will learn:

How Assess-AI works in post-market surveillance.
An overview of AI-PROBE and how it is used for performance evaluation.
How AI-PROBE can augment data repositories, such as Assess-AI.

Bias and Fairness in AI Models

By: Monica J. Wood, MD

Unwanted bias may be incorporated unwittingly into AI models at points throughout an algorithm’s lifecycle — including the creation of training datasets, selection of model architecture and refinement of the algorithm post-deployment. And just as radiologists perform quality assurance on all our imaging modalities, we should aim to incorporate bias evaluation in routine assessment of AI algorithms.

Learning Objectives

How biases make their way into AI algorithms.
Examples of bias affecting the training data.
How to engage in meaningful conversations on bias.

Brittleness of AI Models

By: Woojin Kim, MD

“Brittleness” is the term for an AI model which works well where it was trained but fails when taken outside that area and subjected to new data. There are no guarantees that an algorithm developed elsewhere will function well at your institution — even if it has FDA-clearance. The video explores issues relating to brittleness including concept drift, where the goal of the algorithm changes over time, and data drift, which involves changes created by using new data sources to generate medical imaging data.

Learning Objectives:

The limitations of AI and what can be done to mitigate them.
How validation and training issues contribute to brittleness.
What radiologists should look for in evaluating an AI algorithm.