| To jump to a topic, click your choice below: | ||||
|---|---|---|---|---|
| Protocol Submission | AI FDA FAQs | AI Glossary of Risk | AI Video Series | Additional Resources |
Artificial Intelligence/Machine Learning in Human Research
What's New
New AI Adventures in Medical Research videos are now available!
Meet Clara and Vigil as they explore two different ways artificial intelligence may be involved in medical research: AI being developed for future clinical use and AI being used to support research.
Episode 1: Developing AI in Medical Research
Meet Clara, an AI tool being developed and tested for future clinical use. Follow Clara as she learns how her intended use may affect the IRB and FDA requirements that apply to her development as Software as a Medical Device (SaMD).
Episode 2: Using AI in Medical Research
Meet Vigil, an existing AI tool being used to support safety monitoring during a clinical trial. Vigil shows how AI can support a research team without being the investigational product or replacing clinical judgment.
AI Protocol Submission Guidance for Researchers
The AI Protocol Submission Guidance for Researchers offers guidance to researchers conducting research that uses or develops Artificial Intelligence (AI). In this guidance, “AI system” refers broadly to any AI-based tool, model, or software used or developed as part of a human research study.
Not all sections will apply to every study. Researchers should use their judgment to determine which areas require AI-related explanation and what level of detail is appropriate. For example: A study using an AI transcription service to transcribe recorded interviews may only need to include the AI system name. Then, confirm whether the third-party vendor has access and maintains ownership of the data being transcribed and include this information in the informed consent.
AI Study Design Description
In the Study Design section of a protocol application, researchers are asked to describe and explain the overall study design. If using or developing AI, researchers should describe how the AI system fits within the study design. This information helps IRB reviewers determine whether the technology introduces additional risks or may require additional regulatory review or oversight.
Use the following guidance to clearly describe the AI system and how it functions within the study design:
GENERAL INFORMATION
AI System Name and Version: Including the system name and version helps clearly document what system was used, which supports transparency, reproducibility, and review of the research. (e.g., “GPT-4.1,” “custom convolutional neural network v2.3”)
Whether the AI System is Being Used or Developed: Your description should reflect whether you are using an existing AI system or developing one as part of the research.
- Existing AI system is being used “as is”
- Existing AI system is being used in a way that it was not intended to be used
- A new AI system is being developed as part of the research
In general, studies using an existing system should focus on how the system is applied, while studies modifying the use of AI or developing AI should also describe how the system is built, trained, or updated.
For AI development, indicate what development stage the AI system is in (e.g. record review for data testing and training).
What the AI System Does or What the AI System Will Do: Describe the AI system’s function in plain language:
- Recruitment (e.g., pre-screening electronic records based on enrollment criteria)
- Data transcription (e.g., focus group audio recordings)
- Data collection (e.g., chatbot administering survey questions)
- Data analysis (e.g., identifying patterns in imaging data)
- Prediction or classification (e.g., predicting hospital readmission risk)
WHEN ADDITIONAL DISCLOSURE IS IMPORTANT
The following situations do not necessarily require additional detail, but they should be clearly disclosed in the protocol, as they may raise important ethical, regulatory, or oversight considerations.
General Use:
- Direct Interaction with Participants: (e.g., an AI chatbot administers survey questions, asks follow-up questions based on participant responses, or conducts an interview in real time)
- Participant Intervention: (e.g., participants wear a device that continuously collects physiological data—such as heart rate or activity level—and the AI system prompts them to increase the intensity of their activity in real time)
Non-Standard or Higher Risk Use:
- Use Outside Intended Purpose: (e.g., repurposing a diagnostic system for screening in a new population)
- Downstream Clinical Intent: Indicate whether the non-standard use of an AI system has a downstream intent to diagnose, treat, cure, mitigate, or prevent disease; and whether it may influence clinical care.
- Use in a Clinical Trial: Clarify whether the AI influences endpoints, treatment decisions, or stratification. (e.g., in a drug trial, an AI system monitors patient response data and flags participants for dose adjustment or identifies when clinical endpoints—such as disease progression—have been met)
Developing or Integrating AI Systems in Clinical Care:
- Role in Clinical Decision-Making: Explain if the AI system will be informing decisions (decision support) or making automated decisions
- Regulatory Context for Development of an AI System: Indicate whether the software may meet the FDA definition of Device Clinical Decision Support Software, and reference applicable guidance when relevant. FDA Digital Health Policy Navigator Tool
- Adaptive or Evolving Algorithms: If the system updates during the study, describe how those changes are monitored or controlled.
COMMON PITFALLS
- Describing the AI in purely technical terms without explaining what the system does or how it will be used
- Not clarifying whether the system is new, modified, or pre-existing
- Omitting whether the AI interacts directly with participants or influences care decisions
AI Data Description
AI Data Description: In the Data Collection section of a protocol application, researchers are asked to describe what data are collected and why. Here, researchers should focus on describing what collected data will go into the AI system. Because AI systems depend on data to function, the type and source of these data are key to understanding potential risks related to privacy, confidentiality, consent, and group harms.
Use the following guidance to clearly describe what data the AI system will access:
GENERAL INFORMATION
An AI data description should indicate whether:
Collecting Data from Living Individuals: (e.g., through surveys, chatbots, wearable devices)
De-identified Secondary Data: (e.g., commercial, licensed, or publicly available datasets used as input to the AI system)
Electronic Health Records or Imaging Data: Specify whether data are retrospective or prospective, and how they are accessed or integrated into the AI system.
Student Records: (e.g., educational data protected under FERPA or institutional policies)
Combining Data with Other Datasets or Auxiliary Information: (e.g., linking datasets, enriching with external sources, or merging across systems)
Known limitations or biases in the dataset(s): See AI Glossary of Risks
COMMON PITFALLS
- Assuming “de-identified” data eliminates all privacy risk
- Not describing how datasets are sourced, accessed, or combined
- Omitting whether data originate from living individuals
AI Data Security
In the Records, Privacy, and Confidentiality section of a protocol application, researchers are asked to describe how data are stored and who has access. Because AI systems may process data in external or cloud-based environments, understanding how data move through and are controlled within these systems is essential for assessing risks to privacy, confidentiality, and unauthorized access.
Use the following guidance to clearly describe where the AI system is hosted and who may access, retain, or use the data:
GENERAL INFORMATION
Compliance with Institutional and Regulatory Requirements: Per AR 10.7, Protected Health Information (PHI) or Personally Identifiable Information (PII) should not be entered into a generative AI tool or other software unless the UK HealthCare InfoSec Data Sharing Committee has confirmed the tool is HIPAA-compliant and supports PHI
Where the AI system is Hosted: Clearly indicate whether the AI system is:
- Hosted in a third-party or cloud-based environment, or
- Running locally within a UK-managed or protected environment
If using a cloud-based or third-party system, describe whether data entered in the system may be stored, accessed, or used by the vendor (e.g., for service improvement or system training).
When AI systems are used within a UK-protected environment (e.g., UK Centered for Applied AI), both the system and the data remain under institutional control. A helpful way to think about this is with a Hotel California analogy. The AI system can “check in” to analyze the data, but it cannot leave the environment with the data.
Data Access and Ownership: Indicate whether the AI vendor has access to the data and whether the vendor retains any ownership, usage rights, or licensing permissions over the data.
UK ITS/GRC: Indicate whether a data privacy review was conducted by UK Information Technology Services (ITS) Governance, Risk and Compliance (GRC) team.
Additional Data Security and Privacy Controls: Describe safeguards in place to mitigate AI-related risks (e.g., access controls, encryption, data minimization, monitoring)
AI Sample Informed Consent Language
AI Food and Drug Administration (FDA) Frequently Asked Questions (FAQs)
How do I find out if the software/application/algorithm I'm developing is a medical device?
Consult the FDA Digital Center for Excellence and use the Digital Health Policy Navigator to determine if subject to FDA oversight and applicable FDA legal and regulatory requirements.
How do I find out if the Clinical Decision Support (CDS) software/application/algorithm I'm developing is a medical device?
Consult the FDA website and Graphic, Final FDA Guidance, and/or consult the Digital Center for Excellence with questions.
According to the FDA Digital Health Policy Navigator tool, the software/application/algorithm I'm developing may be subject to FDA oversight as a medical device. How does that impact my IRB submission?
If you are in the Test or Validation phase and the study will involve intervention or interaction with human subjects, complete the medical device section of the IRB application. Indicate the applicability of Investigational Device Exemption (IDE) regulations (e.g., exempt, non-significant risk, or significant risk).
According to the FDA Digital Health Policy Navigator tool, the software/application/algorithm I'm developing may be subject to FDA oversight as a medical device. How do I classify or engage FDA to determine procedure for clearance or approval?
For formal classification of a product, a 513(g) request for information to the applicable Office of Health Technology (OHT). You may also engage the FDA or request feedback through the Pre-Submission program.
How is the FDA regulating the clearance and approval of Artificial Intelligence (AI) medical devices that use Machine Learning?
Is there an FDA list of AI/ML enabled medical devices?
How do I determine what other laws and regulations apply to the software or mobile application I'm developing?
Use the Federal Trade Commission (FTC) Mobile Health App Interactive Tool for determining which laws and rules may apply.
AI Glossary of Risk
AI risks are grouped by where they arise: data and inputs, outputs and performance, privacy and security, human interaction, generative AI and Copilot limitations, governance, and transparency.
Data and Input Risks
- Bias: AI bias occurs when an AI system produces outcomes that reflect or amplify unfair patterns, stereotypes, or disparities. Bias may originate in training data, data selection, labeling, model design, or the way the system is used.
- Example: An AI recruitment tool trained primarily on data from younger adults may identify fewer potentially eligible older adults, resulting in unequal recruitment opportunities.
- Data Drift: Data drift occurs when the characteristics of data encountered by an AI system change over time and become different from the data used to develop or validate the model. Even when the model itself remains unchanged, these differences can reduce its accuracy, reliability, or usefulness.
- Example: An AI model validated using data from one healthcare system may become less reliable when patient demographics, clinical practices, or documentation methods change.
- Data Fusion/Combining: Data fusion is the combination of information from multiple data sources (e.g., de-identified data combined with publicly available demographic or location data). Although combining data can improve analysis, it can also create inaccurate profiles, increase re-identification risk, or make it more difficult to identify data conflicts and errors from different datasets.
- Data Minimization: Data minimization is the practice of collecting, using, and retaining only the information necessary for a defined purpose. Failure to minimize data can increase privacy, confidentiality, security, and secondary-use risks. However, over-minimizing data can limit results, reduce accuracy, or mask bias if relevant data points, such as race, gender, or age, are excluded.
- Data Provenance: Data provenance describes where data originated, how data were collected, and how data were cleaned, labeled, transformed, combined, or reused. Incomplete or unclear provenance can make it difficult to evaluate data quality and appropriateness, trace errors or bias, and reproduce research results.
- Example: Researchers may be unable to determine whether data used to train an AI model were collected with appropriate consent or permission for secondary research use.
- Data Quality: Incomplete, inaccurate, outdated, inconsistent, mislabeled, or poorly representative data can produce unreliable AI results. Data quality problems may be carried into model training, evaluation, or AI-assisted decisions.
- Poisoned Data: Data that have been deliberately manipulated, inserted, mislabeled, or corrupted to influence an AI system’s training or behavior. Data poisoning can cause a model to produce inaccurate, biased, unsafe, or attacker-directed results. Researchers using internet, third-party, or externally supplied datasets may not know that the data have been intentionally altered.
- Example: An externally obtained image dataset may contain deliberately mislabeled medical images intended to reduce the accuracy of a diagnostic model.
- Skewed Data: Skewed data does not adequately represent the population, setting, or conditions in which an AI system will be used. Models trained on skewed data may produce inaccurate or inequitable results for underrepresented groups.
- Example: A model developed using data from adults at an academic medical center may not perform reliably for pediatric patients or people receiving care in rural settings.
- Synthetic Data: Artificially generated data designed to reproduce selected characteristics or patterns of real-world data. Although synthetic data may reduce the need to use identifiable records directly, it may still reveal information about individuals represented in the source data, particularly when privacy protections are weak. Synthetic data may also inaccurately represent real-world populations or reinforce demographic inequities present in the original data.
Output and Performance Risks
- Confabulation (Hallucination): Confabulation, commonly called hallucination, occurs when generative AI produces false, inaccurate, fabricated, or unsupported information and presents it as if it were reliable. Hallucinations may result from insufficient context or poor-quality data and may also occur when the AI attempts to fill gaps in available information.
- False Negative: A false negative occurs when an AI output incorrectly indicates that a condition, characteristic, or finding is absent when it is actually present. In research or clinical settings, false negatives can cause important cases or risks to be overlooked.
- False Positive: A false positive occurs when an AI output incorrectly indicates that a condition, characteristic, or finding is present when it is not. False positives can lead to unnecessary follow-up, exclusion, intervention, or misclassification.
- Information Integrity: Information integrity refers to the accuracy, authenticity, completeness, and reliability of information. AI-generated, summarized, or modified content can make it harder to determine whether information remains faithful to the original source.
- Example: An AI-generated summary of participant interviews may omit qualifying statements or combine comments in ways that alter their original meaning.
- Misclassification: Misclassification occurs when an AI system assigns information, people, records, or outcomes to the wrong category. Misclassification can result from biased data, unclear categories, poor model performance, or use outside the intended context.
- Example: An AI tool used during recruitment may incorrectly classify an eligible individual as ineligible or assign a participant to the wrong study group.
- Misinformation: Misinformation is false, inaccurate, or misleading content that may be generated, repeated, or amplified by an AI system. Users may spread misinformation when AI output is accepted without verification.
- Overfitting: Overfitting occurs when a model learns the training data too closely, including noise or idiosyncratic patterns, and performs poorly on new data. Strong training performance may therefore conceal weak real-world generalizability.
- Over-optimization: Over-optimization occurs when a model is tuned too narrowly to a specific metric or target. Performance may improve on the selected measure while accuracy, fairness, safety, usefulness, or other important outcomes deteriorate.
- Prompt Injection: Prompt injection is when instructions embedded in a prompt, document, email, webpage, or other source attempt to override an AI system's intended instructions or cause the system to reveal information or perform unintended actions.
- Example: A document uploaded for AI-assisted analysis may contain hidden or embedded instructions that cause the system to ignore the researcher’s original task or disclose information from another accessible source.
Privacy, Confidentiality, and Information Security Risks
- Data Leaks: A data leak is unintended or unauthorized exposure of private, confidential, proprietary, or sensitive information. Information may be exposed through prompts, uploaded documents, generated output, logs, connected systems, or insecure storage.
- Re-identification: Re-identification is the process of linking de-identified or coded data back to a specific individual. Combining datasets, indirect identifiers, or auxiliary information can make re-identification possible even when direct identifiers have been removed.
Generative AI and Human Interaction Risks
- Anthropomorphism: Anthropomorphism, also called over-personification, occurs when people attribute human understanding, judgment, intent, or emotion to an AI system. This can create misplaced trust and reduce appropriate human oversight.
- Automation Bias: Automation bias is the tendency to favor an automated recommendation or AI-generated answer even when other evidence or human expertise suggests that it may be incorrect.
- Conversational Drift: Conversational drift occurs when an AI interaction gradually moves away from the original task, source material, or constraints. Responses may become broader, more speculative, repetitive, or less grounded as the conversation develops.
- People Pleasing or Agreeableness: Generative AI is designed to be helpful and responsive and may reinforce a user's assumptions, framing, or preferred conclusion instead of challenging it. Leading prompts can increase the risk of receiving an answer shaped toward a predetermined result.
- Personalization and Context Variation: Generative AI responses may be influenced by the user's accessible information, conversational history, provided context, and prompting choices. Different users may receive different responses to similar questions.
- Example: Two members of a research team may receive different answers to the same protocol question because their conversational histories, available files, permissions, or prompts differ.
- Over-reliance: Over-reliance occurs when users place too much trust in AI output or delegate judgment that requires human expertise. Over-reliance can allow errors, omissions, unsupported claims, or inappropriate recommendations to influence decisions.
Generative AI and Copilot Limitations
- Context Dependence: Generative AI output depends on the instructions, attachments, accessible information, permissions, and conversational context available when the response is produced. Changes in any of these factors can change the result.
- Document Interpretation and PDF Readability: AI may extract a document's text without fully preserving checkboxes, columns, tables, alignment, page layout, or relationships among visual elements. Information that depends on document structure should be verified against the original file.
- Grounding Limitations: Grounding connects an AI response to accessible source material. If relevant sources are missing, outdated, inaccessible, ambiguous, or poorly selected, the response may be incomplete or inaccurate. Citations and critical details should be checked against the original sources.
- Permission-Based Oversharing: Microsoft 365 Copilot respects existing user permissions, but inaccurate or overly broad permissions may allow sensitive information to be discovered more easily. Copilot does not correct underlying access-control problems.
- Output Variability: Generative AI may produce different wording, emphasis, findings, or levels of detail across users or repeated prompts. A similar request does not guarantee an identical or fully repeatable response.
Transparency, Explainability, and Generated Content Risks
- Deepfakes: Deepfakes are audiovisual or other media that have been synthetically generated or altered to falsely represent a person, event, or source. Deepfakes can spread false information, undermine authenticity, and contaminate research or public evidence.
- Excessive Agency: Excessive agency is when an AI system or agent is given more functionality, permissions, access, or autonomy than necessary. Excessive agency increases the chance that the system can take unintended, unauthorized, or harmful actions.
- Example: An AI agent connected to recruitment systems could contact potential participants, change records, or initiate workflow steps without sufficient human review.
- Interpretability: Interpretability is the degree to which people can understand how an AI system processes information or produces output. Limited interpretability can make it difficult to identify errors, evaluate fairness, or justify decisions.
- Shadow AI: Shadow AI is the use of AI tools, models, agents, or AI-enabled features outside an organization's approved, visible, or governed technology processes. Shadow AI can result in organizational or research information being processed by tools that have not undergone appropriate privacy, security, compliance, contractual, or institutional review. In research, Shadow AI can also obscure where AI was used and how research information was handled.
- Example: A researcher may paste identifiable participant information into a publicly available AI tool that has not been approved by the institution for handling research data. Or a researcher may download or use a third-party AI tool without realizing that its terms of use allow the provider to retain or reuse research or participant data submitted to the tool, including for product or model improvement.
- Synthetic or AI-Generated Content Disclosure: Failure to disclose that content was generated, altered, summarized, or translated by AI can mislead readers about authorship, authenticity, evidence, or the role of human judgment.
Sources Consulted in Developing the AI Glossary of Risk*
- NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (2024; updated 2026).
- OWASP Top 10 for LLM Applications and Generative AI (2025).
- Microsoft guidance on Microsoft 365 Copilot grounding, permissions, oversharing, and prompt-injection protections.
- ORI staff-training limitations developed for the AI Pre-Screening Tool: personalization, people pleasing, conversational drift, document readability, output variability, and required human oversight.
*The categories and definitions in this glossary were developed for University of Kentucky research education purposes. Recognized AI risk terminology and concepts were informed by guidance from NIST, OWASP, Microsoft, and other authoritative sources. Definitions have been adapted and written in plain language for a research audience and are not direct quotations unless specifically indicated.
AI Adventures in Human Subjects Research Video Series
The AI Adventures in Human Subjects Research Video Series offers researchers responsible-use strategies for mitigating the risk of employing AI in human subjects research.
Episode 1. AI in Human Subjects Research
Episode 1 provides the purpose for creating the video series, presents the beneficence conundrum, and introduces the data ancestors. [7:10]
Episode 2. Grandfather Data
Episode 2 offers strategies for mitigating the risk of inputting potentially biased historical data sets into AI models. [4:54]
Episode 3. Aunt Data
Episode 3 offers strategies for mitigating the risk of combining de-identified data with publicly available auxiliary information. [4:53]
Episode 4. Sister Data
Episode 4 offers strategies for mitigating the risk of mining open-source social media content that may or may not be publicly available. [6:43]
Episode 5. Uncle Data
Episode 5 offers strategies for mitigating the risk of unintentionally including hallucinations or fabricated data in research involving human subjects. [6:10]
Additional Resources
Commonwealth AI Transdisciplinary Strategy (CATS AI)
CATS AI operates as an initiative of the Advancing Kentucky Together (AKT) Network, connecting UK with partners statewide to position the Commonwealth as a national leader in AI adoption and innovation.
UK Center for Applied AI
Since 2023, the UK Center for Applied Artificial Intelligence has been at the forefront of AI, making investments in people and technology to empower others to explore AI and create meaningful solutions. Today, the Center is a specialized community who empowers faculty, staff, researchers, and clinicians to use AI to advance research, improve health outcomes, enhance student experiences, and drive productivity. Our team includes early and late-career software developers, project managers, data scientists, and advisors who guide our collaborators to overcome common AI barriers, such as technical expertise and access to secure compute resources. Without requiring a technical background or costly investments, we help our collaborators quickly turn ideas into prototypes and prototypes into solutions that make an impact for the communities they serve. Since our inception, we’ve supported more than 100 projects, have a network of 43 partners and over 120 individual collaborators. Please fill out our Collaboration Intake Form to connect.
UK ADVANCE
Recommendations on the Use of Generative AI in Research and Scholarly Activity. UK ADVANCE offers guidance in response to frequently asked questions (FAQs) about the use of artificial intelligence from the UK research community.