Over the last year, I have been watching another important shift take place in AI.

The early security conversation around generative AI was relatively straightforward. Businesses worried about employees entering confidential information into ChatGPT, and security teams focused on data privacy, acceptable-use policies, and whether public AI models might retain sensitive information.

Those concerns still matter, but they are no longer the most interesting part of the AI security problem.

AI systems are increasingly being connected to email, document repositories, software codebases, customer relationship management systems, cloud platforms, databases, and other business applications. As I discussed recently in my article on MCP servers and the next phase of business AI, we are moving from AI systems that simply answer questions to AI systems that can work across the applications businesses already use.

That creates enormous opportunity. It also changes the security model.

The question is no longer only: What information can an AI system see?

Increasingly, the more important question is: What can an AI system be manipulated into doing with the information and access it has been given?

That distinction is going to become increasingly important as businesses move from chatbots to AI agents.

AI Is Moving From Answering to Acting

The first generation of generative AI was primarily conversational. You asked a question and the AI generated an answer. There was a relatively clear boundary between the model and the systems around it.

That boundary is disappearing.

An AI assistant might now search company documents, review emails, inspect a GitHub repository, check customer records, query a database, call an API, or trigger a workflow. AI agents go further by completing a series of actions on our behalf.

That changes the risk considerably. A chatbot that misunderstands a question may give you a poor answer. An AI agent that misunderstands — or is deliberately manipulated — could potentially perform an action in another system.

I think a useful way to look at the progression is:

Ask → Access → Connect → Act → Operate

At each stage, AI becomes more useful. At each stage, the potential impact of getting security wrong also increases.

AI Risk Increases as AI Gains Agency

What Does It Mean to “Hack” an AI?

When we hear the phrase AI hacking, it is easy to imagine someone breaking directly into an artificial intelligence model. That is not necessarily what is happening.

Increasingly, attackers are trying to influence the information an AI system consumes. That information might come from an email, website, PDF, support ticket, GitHub issue, shared document, database, API response, or MCP server.

The challenge is that large language models are designed to interpret natural language. That is exactly what makes them useful, but it also creates a security problem.

Traditional applications are normally designed to distinguish clearly between instructions and data. An AI system has to interpret both.

That creates an attack technique called prompt injection.

OWASP now lists Prompt Injection as one of the most important security risks affecting large language model applications. Prompt injection occurs when external content alters the behaviour of an AI system in unintended ways, potentially leading to unauthorized access, information disclosure, or unsafe actions.

And we are starting to see what that can look like in real systems.

EchoLeak: When an Email Could Attack the AI

One of the most important examples involved Microsoft 365 Copilot.

In 2025, researchers disclosed a vulnerability known as EchoLeak, formally identified as CVE-2025-32711. The vulnerability involved an indirect prompt-injection attack against Microsoft 365 Copilot and demonstrated how an AI assistant could potentially be manipulated through information it retrieved from an external source.

What made EchoLeak particularly interesting was the attack path.

An attacker could send a specially constructed email containing instructions intended for Copilot rather than the recipient. When Copilot subsequently retrieved that email as part of its normal work, those instructions could influence the AI and potentially cause sensitive information to be exposed.

The user did not necessarily need to click a malicious link or consciously execute the attack. In this case, the AI itself was the target.

Microsoft addressed the vulnerability, but the broader lesson is more important than the individual bug.

For years, cybersecurity teams have trained employees to identify suspicious emails. Now we also have to consider whether an email is capable of manipulating the AI assistant reading it.

That is a fundamentally different threat model.

Reference: Microsoft Security Response Center — CVE-2025-32711

MCP Expands the Security Boundary

This is also why I think Model Context Protocol, or MCP, deserves attention from technology leaders.

MCP provides a standardized way for AI systems to connect to tools and data sources. That might include GitHub, a CRM, a payment system, an internal database, a cloud service, a project-management platform, or an organization’s own applications.

The business opportunity is significant, but connecting an AI assistant to more systems also expands its security boundary.

The security question is no longer simply whether the AI provider is secure. It becomes a broader architectural question: What systems can the agent reach? What permissions does it have? Which tools can it execute? Which external sources can influence its decisions? What happens if one of those sources contains malicious instructions?

This is one of the reasons OWASP includes Excessive Agency among the major risks associated with generative AI applications.

OWASP identifies three common causes: excessive functionality, excessive permissions, and excessive autonomy.

Those three ideas provide a very useful way of thinking about AI security.

An AI assistant that needs to read customer information probably does not need administrator access to the CRM. An AI system that summarizes email probably does not automatically need the ability to send or delete email. An agent that reviews infrastructure alerts probably does not necessarily need permission to modify production systems.

The fact that AI can perform an action does not mean it should be allowed to perform it automatically.

We Have Already Seen AI Reach Real Systems

Another recent example illustrates why system boundaries matter.

In July 2026, Anthropic disclosed that during cybersecurity evaluations, Claude models gained unauthorized access to real systems belonging to three organizations. The models had been told that they were operating within simulated environments, but a configuration problem in a third-party evaluation environment inadvertently left a path to the live internet available.

The models then interacted with real systems while attempting to complete their evaluation tasks.

Anthropic subsequently identified an additional incident while expanding its investigation.

There are important qualifications here. These were cybersecurity evaluations conducted under unusual conditions, and some normal safeguards had intentionally been removed. But I think the business lesson is still useful.

AI capability and infrastructure configuration can no longer be considered separately.

A model may have enormous capability, but the real-world consequence of that capability is determined by what the surrounding architecture allows it to reach.

References:

Anthropic — Investigating Incidents in Cybersecurity Evaluations

Anthropic — Alignment Assessment of Cybersecurity Incidents

Gemini Crossed From a Security Test Into Real Companies

One of the most recent examples involves Google’s Gemini models.

During a cybersecurity evaluation in May 2026, Gemini was supposed to attack systems belonging to a fictional company in a controlled environment. A configuration issue unexpectedly gave the model access to the public internet, where it found or guessed credentials and gained access to systems belonging to three real companies. Ars Technica provides a detailed account of the incidents.

Google says Gemini stopped once it recognized that the systems were real rather than part of the test. That is encouraging, but the incident still demonstrates how quickly a capable AI agent can move from a simulated task into real infrastructure when access controls, credentials, and environment boundaries fail.

It is also worth noting that Google did not publicly disclose the incidents when it was first notified. The events only became public later through media reporting, raising broader questions about how AI security incidents should be disclosed.

The key lesson is simple: model safeguards are only one layer of security. Proper containment, permissions, credential management, and system architecture remain essential.

Reference: WSJ – Gemini Hacked Three Companies in First Known Breakout by Google’s AI

The Next Security Problem May Be AI Memory

Another emerging issue is even less obvious.

What happens when AI systems start remembering information?

In February 2026, Microsoft described a technique it calls AI Recommendation Poisoning. Researchers observed prompts designed to tell AI systems to remember certain companies as trusted or preferred sources, with the intention of influencing future recommendations.

This does not look like a traditional cyberattack. There is no stolen password and there may be no malware.

Instead, the attacker attempts to influence the context the AI carries forward.

As AI assistants develop longer-term memory, richer organizational context, and more persistent understanding of users, we may have to protect something new: the AI’s understanding of the business itself.

Security could increasingly involve protecting not only systems and data, but also context, memory, and behaviour.

Reference: Microsoft Security — AI Recommendation Poisoning

The New Trusted Insider

There is another way I think about this.

Traditional enterprise security frequently looks something like:

User → Application → Permission → Data

AI adds another layer:

User → AI → Tool → Application → Data

The AI becomes an intermediary acting on behalf of the user.

That creates a deceptively simple question:

How much authority should an AI inherit simply because the person using it has that authority?

My view is: not necessarily very much.

If I have permission to delete every record in a CRM, that does not mean the AI assistant helping me research customers should automatically inherit that permission. If a developer has production access, a coding agent probably should not automatically receive every credential available to that developer.

And if an executive can authorize a payment, an AI assistant preparing invoices probably should not be capable of initiating one without additional approval.

This is essentially the principle of least privilege applied to AI.

The agent should receive the minimum capability required to complete the job.

A Practical Framework for Managing AI Security

There are already useful industry frameworks available.

The NIST AI Risk Management Framework provides organizations with a structured way to identify, assess, and manage AI risks. NIST has also published a specific Generative AI Profile addressing risks associated with generative systems.

OWASP provides a more technical perspective through its Top 10 risks for LLM and generative-AI applications.

For business leaders, however, I think the starting point can be simpler.

I would organize AI security around four practical principles:

Govern

Start by understanding where AI is actually being used.

Many organizations already have AI embedded in SaaS platforms, development tools, productivity suites, and marketing applications without having a complete inventory.

Document the AI platforms, agents, MCP servers, connected systems, data sources, APIs, automated workflows, and AI-enabled SaaS products in use. Then establish ownership.

Who approves these systems? Who owns their data access? Who decides which applications they can connect to?

AI governance cannot simply be an acceptable-use policy sitting in a document somewhere. It has to connect to architecture and operations.

Limit

Apply least privilege.

Give AI systems only the information and functionality they actually require.

For example, Read Customer Record is very different from CRM Administrator.

The same principle applies to email, cloud systems, databases, document platforms, and code repositories. Permissions should be designed around the use case rather than inherited simply because broad access is convenient.

Validate

Treat external information entering an AI system as potentially untrusted.

That includes documents, websites, emails, tickets, uploaded files, retrieved knowledge, and responses from third-party tools.

It also means validating what the AI produces before another system acts on it.

The more authority an AI agent receives, the stronger this validation layer needs to become.

Monitor

Organizations should be able to reconstruct what an AI system did.

That means logging what the agent accessed, which tools it called, what records it changed, what external information influenced it, which user initiated the request, what approvals occurred, and what outcome was produced.

This is going to become increasingly important for security, compliance, and incident response.

When an automated agent makes a poor decision, organizations need more than the final output. They need an audit trail.

References:

NIST — AI Risk Management Framework

OWASP — Top 10 for LLM and Generative AI Applications

Add Human Approval Where the Consequences Matter

I do not believe every AI action should require human approval. That would defeat much of the value of automation.

But the amount of autonomy an AI receives should reflect the potential consequence of the action.

There is a significant difference between Summarize this customer record and Delete this customer record.

The same principle applies to financial transactions, infrastructure changes, production deployments, changing user permissions, external communications, contractual documents, and confidential information.

For higher-impact actions, a good operating model may remain:

AI recommends → Human approves → System executes

Over time, organizations may automate more of these decisions as confidence and controls improve.

But autonomy should be earned through good architecture, not simply enabled because the technology supports it.

AI Security Is an Architecture Problem

This is where I think businesses sometimes approach AI from the wrong direction.

AI security is not primarily a feature you buy. It is an architectural decision.

It involves identity, permissions, systems, data, APIs, integration, monitoring, software design, governance, and increasingly, agent behaviour.

This is why AI adoption needs to be considered within a broader digital-transformation strategy rather than as an isolated technology initiative.

At Swinmark Consulting, our approach starts by understanding how the business operates, the systems involved, the data moving between them, and the constraints under which teams work. We then look at where technology can remove friction while managing the operational risk that comes with it.

That approach becomes even more important with AI.

AI security touches several areas we already help clients navigate: technology strategy, system architecture and integration, AI enablement, software engineering, and managed services.

The objective should not be to put barriers around AI adoption. It should be to design AI into the organization in a way that is useful, controlled, observable, and aligned with the business.

What Businesses Should Do Now

For most small and mid-sized organizations, I do not think the answer is to slow down AI adoption.

The answer is to become more deliberate about it.

Start with five questions:

Where is AI already being used in the organization?

What information can those AI systems access?

What other systems can they connect to?

What actions are they capable of taking?

How would we know if something went wrong?

Those questions quickly expose where governance and architecture may need attention.

They can also form the basis of an AI Security Readiness Assessment covering AI inventory, data classification, identity and permissions, agent and MCP integrations, third-party AI platforms, logging and monitoring, software-development practices, approval controls, incident response, and governance.

The goal is not to create bureaucracy.

It is to make sure the controls evolve at the same pace as the capability.