AI is rapidly integrating into daily operations such as customer service, fraud detection, software development, analytics, and decision support. This broad adoption also makes AI a prime target for attackers, who may manipulate inputs, poison training data, steal models, or use generative AI to scale social engineering and fraud. NIST notes that while AI offers significant benefits, it also introduces risks that can adversely affect individuals, organizations, and society.
Fortunately, AI risk and security can be managed through disciplined governance, clear threat modeling, measurable controls, and continuous monitoring, particularly when aligned with established guidance such as the NIST AI Risk Management Framework (AI RMF 1.0) [1] and the AI RMF Playbook. [2]
How does AI security differ from traditional security?
AI systems don’t just run code—they learn patterns from data and often operate probabilistically. This introduces new failure modes and attack vectors, including:
- Adversarial inputs and prompt injection: crafted inputs that cause unsafe or unintended outputs.
- Data poisoning: contaminated training or retrieval data that changes model behavior.
- Model and data exfiltration: stealing model weights, prompts, training data, or proprietary knowledge through system endpoints. [1]
- Operational drift: performance changes over time as data and contexts shift.
- Explainability gaps and hidden bias: outputs may be hard to justify; bias can emerge if models aren’t tested, validated, and used appropriately. [5]
Additionally, criminals are leveraging generative AI to make fraud more convincing and scalable by using AI-generated text, images, audio vocal cloning, and video impersonation. [3]
Leverage the NIST AI RMF to structure your defense
NIST’s AI RMF organizes AI risk management into four functions—GOVERN, MAP, MEASURE, MANAGE—so organizations can apply consistent rigor across the AI lifecycle. [2]
1) GOVERN: Set accountability, policy, and oversight
Governance is foundational. The AI RMF Playbook emphasizes that effective oversight requires organizational commitment and accountability mechanisms, not solely technical controls.
Practical GOVERN controls (from the Playbook) include:
- Policies that promote regular communication among AI actors involved in risk management
- Separation of AI development from AI testing functions to enable independent course-correction
- Policies to identify and prevent conflicts of interest
- Training and role clarity across technical and oversight stakeholders
Financial-services lens: FFIEC guidance for technology operations governance highlights board/senior management responsibilities, enterprise risk management (ERM), policies, standards, procedures, internal audit, independent reviews, and reporting. To further clarify these principles in relation to AI governance, consider the following comparison:
Traditional IT:
- Policies and standards that define IT operations
- Routine internal audits and independent reviews for IT infrastructure
- ERM frameworks focusing on operational and financial risks
- Board accountability predominantly for IT strategy and oversight
AI-specific:
- Enhanced policies that include AI governance and ethics
- Regular audits for AI algorithms and data usage
- ERM that integrates AI-based risks like data bias and algorithmic malfunction
- Board accountability expanded to AI technology impact and compliance assessments
This contrast highlights how traditional and AI-specific practices can overlap and where new considerations need to be made.
2) MAP: Know what you built, where it runs, and what could go wrong
Mapping involves documenting AI use cases, stakeholders, data flows, dependencies, and areas of impact. In practice:
- Maintain an AI system inventory (including shadow/embedded AI)
- Classify use cases by criticality and exposure (customer-facing, financial decisions, safety-impacting, etc.)
- Identify likely threats: data leakage, fraud enablement, unsafe output pathways, and third-party concentration
Financial-services lens: Treasury’s financial services report highlights that firms are cautious about broadly deploying GenAI in customer-facing use cases due to new risks and notes that smaller firms are more dependent on third parties.
3) MEASURE: Test trustworthiness and security—continuously
AI security measurement extends beyond traditional vulnerability scans. NIST identifies common AI security concerns such as adversarial examples, data poisoning, and exfiltration, all of which require targeted evaluation. [2]
A practical measurement program includes:
- Red teaming for prompt injection, jailbreaks, and harmful output
- Data quality and provenance checks for training and retrieval corpora
- Privacy testing (sensitive data leakage, memorization risks)
- Bias and fairness tests for regulated decisions
- Resilience monitoring for drift and degradation over time
4) MANAGE: Reduce risk with controls, monitoring, and incident response
Risk management is the stage where controls are implemented:
- Implement guardrails (policy engines, content filters, tool-use constraints)
- Secure the AI “supply chain” (models, datasets, plugins/tools, agents, MLOps pipelines)
- Monitor runtime behavior (anomalous queries, data access spikes, unusual tool calls)
- Create AI-specific incident playbooks (data leakage, model compromise, fraudulent use)
FFIEC specifically warns that AI/ML can introduce risks tied to large-scale data use (breaches → misuse, fraud, consumer harm), human errors in algorithm development, bias if not tested/validated, lack of explainability, and challenges from dynamic updates. [5]
Address the human threat: GenAI-enabled fraud at scale.
The FBI’s IC3 alert describes how criminals use generative AI to improve the believability of scams and reduce effort by using realistic fake profiles, persuasive phishing, synthetic images, vocal cloning, and deepfake video impersonation. [3] This criminal ingenuity stands in stark contrast to the often routine methods employed by defenders. For instance, while scammers are innovatively crafting deepfakes to deceive individuals, many organizations still rely on basic password checks to secure sensitive transactions. This mismatch reflects an asymmetry that underscores the urgent need for more advanced and imaginative security strategies.
Immediate defenses include:
- Strengthen identity verification for high-risk requests (out-of-band call backs, known numbers)
- Add payment/change-of-bank controls (dual authorization; verification workflows)
- Train staff to expect AI-enhanced social engineering (not just “typos” and “bad grammar”)
- Promote verification rituals (IC3 even suggests a “family secret word/phrase” concept for identity checks) [3]
Key priorities for financial services (and any regulated environment)
Treasury’s report emphasizes that existing frameworks may need to be enhanced as AI accelerates, and it highlights support for actions such as clarifying data privacy/security/quality standards; expanding consumer protections; clarifying compliance expectations; and strengthening collaboration and information sharing (including monitoring concentration risk).
Also, classic third-party risk disciplines become even more important in the context of AI vendors and hosted platforms. FFIEC notes that externally hosted services can reduce control over infrastructure and changes by providers, and stresses contract provisions and risk assessment for potential interruptions. [5]
The following examples provide concrete breach and incident references (Samsung, Amazon, Arup, T-Mobile) that directly illustrate the AI security threat.
Real-world incidents that illustrate the AI security threat
AI security risk is not hypothetical; recent incidents demonstrate how quickly sensitive information can be exposed or exploited when AI systems interact with people, data, and business processes. To further leverage these incidents for organizational growth, consider the following questions: How robust were our current defenses against these types of breaches? What specific vulnerabilities were exploited in these examples, and do we have similar vulnerabilities? What immediate changes can be implemented to fortify our systems? Are our incident response plans updated to handle similar occurrences? By integrating these inquiries into internal audits, organizations can transform past incidents into opportunities for enhancing their adaptive risk culture.
Samsung: Sensitive data exposure via employee use of public GenAI
Reports in 2023 described Samsung employees pasting sensitive information (including source code and internal content) into ChatGPT, prompting Samsung to restrict/ban certain generative AI use on company devices and internal networks. [6]
Security takeaway: Treat public GenAI tools as untrusted external endpoints unless enterprise controls are in place. Apply DLP, prompt and clipboard controls, and enforce clear ‘no secrets’ policies for third-party AI.
Amazon: Internal warnings about confidential data leakage to GenAI
Amazon issued internal guidance warning employees not to share confidential information (including code) with ChatGPT after concerns that outputs could resemble internal Amazon data. [7]
Security takeaway: Even without a traditional breach, data leakage can occur through prompts, pasted code, and troubleshooting logs. The objective remains to prevent sensitive data from leaving the organization via external services.
Arup: Deepfake-enabled fraud (“synthetic identity” + impersonation attack)
Arup confirmed a 2024 incident in which a staff member was deceived by an AI-generated video call impersonating senior leaders and transferred roughly $25M to criminals. [8]
Security takeaway: This incident highlights AI risk at the people and process level. Countermeasures include out-of-band verification, payment controls, and deepfake-aware training, particularly for finance, treasury, and vendor payment teams.
T-Mobile: Recurring large-scale data breaches and regulatory action
T-Mobile has faced multiple data security incidents affecting millions; for example, AP reported a breach impacting 37 million customers that was discovered in early 2023. [9]
Regulators also took action: the FCC announced a $31.5M settlement (Sept 30, 2024) tied to investigations into significant breaches, including forward-looking cybersecurity commitments. [10]
Security takeaway: Traditional cyber hygiene remains essential for AI. Weaknesses in identity, access, logging, segmentation, and incident response can be amplified by AI systems, which increase data touchpoints and automate workflows.
A concise checklist for implementation within 30 to 60 days
Governance & inventory
- Name an executive owner for AI risk, with quarterly board reporting, and a cross-functional AI risk committee
- Inventory AI systems (including “shadow AI” and embedded vendor features)
- Require use-case approval for customer-facing or regulated decisions
Security controls
- Lock down data access (least privilege for retrieval tools, connectors, and logs)
- Add prompt injection defenses (input filtering, tool allowlists, output constraints)
- Protect secrets (no credentials in prompts; rotate keys; isolate environments)
Testing & monitoring
- Red team high-impact use cases monthly
- Track model drift, unusual query patterns, and sensitive data exposure
- Run tabletop exercises for “AI data leak” and “deepfake impersonation” scenarios
Vendor and compliance
- Update vendor questionnaires for AI (training data handling, logging, incident response)
- Map controls to NIST AI RMF functions and your existing enterprise risk framework
How DataComm can help
DataComm can help operationalize AI security while maintaining innovation:
- AI Security & Risk Assessment aligned to NIST AI RMF (GOVERN/MAP/MEASURE/MANAGE), producing an actionable risk register and prioritized control roadmap [1]
- Threat modeling & red teaming for GenAI apps (prompt injection, data exfiltration, tool/agent abuse)
- AI governance program buildout: policies, approval workflows, and training frameworks consistent with AI RMF Playbook governance recommendations
- Financial services–focused control mapping to support examiner-ready documentation and operational governance expectations
- Third-party and concentration risk support for AI vendors, hosted model platforms, and data providers [5]
- Fraud resilience uplift: procedures and training to reduce GenAI-enabled impersonation and social engineering losses [3]
Next steps
If your organization is currently using AI or plans to do so in the next quarter, consider this a prompt to begin immediate action:
- Inventory AI use cases and data pathways.
- Classify which ones are high-risk (customer-facing, money movement, compliance decisions).
- Adopt NIST AI RMF as the organizing framework and implement the top controls first. [1]
If desired, I can draft a one-page ‘AI Security Program Starter’ for DataComm to share with clients, outlining scope, deliverables, timeline, and outcomes.
FAQ
What’s the fastest way to reduce AI security risk?
Start with governance and access control: inventory AI systems, restrict access to sensitive data, and add approval gates for high-impact use cases. The NIST AI RMF’s GOVERN function is designed to anchor everything else. [1]
Are AI threats mostly “future risk,” or are they happening now?
They’re happening now. IC3 warns that criminals are already using generative AI to scale financial fraud using text, images, audio, and video impersonation. [3]
What are the top technical risks to plan for?
NIST highlights AI security concerns such as adversarial examples, data poisoning, and the exfiltration of models/training data through endpoints. [1]
Why do regulators care so much about explainability and testing?
FFIEC notes that AI can lack transparency/explainability, which can reduce confidence, create unintended consequences, and even lead to noncompliance or risk tolerance breaches—especially if models dynamically update. [5]
How should we manage AI vendor risk?
Use traditional third-party risk discipline plus AI-specific requirements (data handling, logging, model update controls, incident response). FFIEC notes that hosted services can reduce your control over infrastructure and changes, making contracts and risk assessments essential. [5]
Do we need an “AI incident response plan,” separate from cyber IR?
Yes—at least an addendum. AI incidents include model output hazards, prompt injection abuse, data leakage through AI interfaces, and deepfake-driven fraud scenarios that blend cyber + fraud + identity workflows. (These risk types are emphasized across NIST’s AI RMF and IC3’s fraud guidance.) [1]
