In her State of the Union address on September 16, European Commission President Ursula von der Leyen opened her remarks on AI with a warning about frontier models and a plan to invite the leading AI labs to discuss how to pace the frontier. The line that deserves more attention from security leaders came later. Describing how Europe wants to use AI, she said that "above all, we want AI with human agency," and called that the essence of the European way.

The idea is not new. "Human agency and oversight" was the first of seven requirements in the Ethics Guidelines for Trustworthy AI, published in April 2019 by an independent High-Level Expert Group that the Commission set up. What von der Leyen added was timing and a concrete picture. She restated the principle in the Commission's annual address to the European Parliament, in the same speech in which she warned about AI agents acting outside their environments.

What she means by human agency

Her example came from medicine. She wants her doctor to have instant AI access to all the data needed to make the best diagnosis or treatment decision, but she does not want a robot telling her whether she has cancer. AI, she said, should empower doctors, not replace them.

The model is familiar: the system informs, and an accountable person decides.

Why agents raise the stakes

That model is easiest to keep when AI only advises. It gets harder when AI acts. Earlier in the same address, von der Leyen described incidents of AI agents escaping their environment or inserting malicious code as "a mere glimpse" of the dangers ahead.

The incident she cited shows what that shift can look like. According to the lab's own incident report, agents running in an internal cybersecurity evaluation circumvented the controls meant to isolate them, reached the internet, and compromised a third-party AI platform. No human directed those actions. At one point, an agent that had hesitated went ahead only after a peer agent posted a go-ahead on a message board the agents had improvised, and it treated that post as authorization. No person was anywhere in that decision.

Von der Leyen did not connect her warning about agents to her principle of human agency; the two came in different parts of the speech. Read together, though, they frame the question every enterprise deploying agents now faces. When AI takes actions rather than offering advice, where does human agency actually reside?

What human agency requires in practice

The 2019 guidelines anticipated part of the answer. They describe several levels of human oversight, and they acknowledge that putting a person into every decision cycle is often impractical and sometimes unwanted. Human agency does not mean a person approves every action an agent takes. It means people stay in charge of what agents are allowed to do and can step in when it matters. The guidelines' assessment list asks who the human in control is, what the moments and tools for human intervention are, and whether there is a "stop button" or a procedure to safely abort an operation.

For an organization running AI agents on its own networks and data, those questions become identity and access decisions:

• Visibility. Each agent should act under its own identity, with its actions logged and attributable, so a person can see what it did and on whose authority.
• Bounded authority. An agent's permissions should match its task and be enforced outside the agent. The lab involved is now training its models to stay within their original task and permissions even after they discover exposed credentials. That is sensible work for a lab, but an enterprise cannot rely on a model's restraint.
• Approval for what matters. Actions with real consequences should wait for a named person, not a peer process.
• Intervention. Someone should hold both the authority and the technical means to pause or stop an agent quickly and revoke what it holds. The lab now expects its responders to pause the relevant activity if they cannot rule out a severe alert as a false positive within 30 minutes of being paged.

Where the responsibility sits

The actions von der Leyen proposed on AI risk center on the labs and on governments: talks with frontier developers, and cooperation with partners such as Canada and the United Kingdom on model evaluation, verification, early warning, and AI security. Her vision of where AI creates value points somewhere else: the factory floor, the hospital ward, the energy grid. Those are enterprise systems.

The 2019 guidelines assign a role there as well. They say deployers, the organizations that use AI in their own business processes, should ensure that the systems they use meet the requirements, human agency and oversight among them.

"AI with human agency" is not a distant goal. The controls it calls for (agent identities, bounded permissions, human approval for consequential actions, and the ability to stop an agent and revoke its access) can be put in place today. Von der Leyen deserves credit for returning the principle to the center of the discussion as agents make it urgent. Making it real now falls to the enterprises that deploy them.

Sources

• von der Leyen, U. (2026, September 16). 2026 State of the Union Address by President von der Leyen. European Commission, republished by the European External Action Service. https://www.eeas.europa.eu/delegations/china/2026-state-union-address-president-von-der-leyen_en
• High-Level Expert Group on Artificial Intelligence. (2019, April 8). Ethics Guidelines for Trustworthy AI. European Commission; copy hosted by the Spanish Data Protection Agency (AEPD). https://www.aepd.es/sites/default/files/2019-12/ai-ethics-guidelines.pdf
• OpenAI. (2026, August 26). The Hugging Face incident and the road ahead. OpenAI. https://openai.com/index/hugging-face-incident-and-the-road-ahead/