Security must start with the model itself 

OpenAI breaks out and hacks a third-party platform

The US company’s AI models gained access to the internet independently and hacked into Hugging Face’s infrastructure in order to achieve a test objective. OpenAI confirmed this in a blog post, thereby accepting responsibility for the attack.

Hugging Face is a platform for sharing AI models and datasets. The company reported an attack that was said to have been controlled entirely by an autonomous AI system. OpenAI subsequently announced that it would be strengthening its security measures.

Security through modelling

This recent incident demonstrates just how capable autonomous AI systems have become and what kinds of undesirable actions are now technically possible. At the same time, it is clear that such risks can be identified at an early stage – and thus mitigated more effectively – through systemic modelling of potential actions, interdependencies and attack vectors.

From SparxSystems Europe’s perspective, threat modelling is therefore not an isolated specialist discipline, but an essential component of professional software and systems development.

Enterprise Architect already enables the joint modelling of system architectures, components, interfaces, data flows, trust boundaries and potential attack surfaces. 

Threats, risks, security requirements and countermeasures can be directly linked to the relevant architectural elements and made traceable throughout the entire development process.

Particularly with AI agents, the following issues arise:

• Which systems and data can the agent access?
• What are the limits of its authorisations?
• Through which interfaces could it leave its intended environment?
• Which security measures must be incorporated into the architecture from the outset?

The key advantage of model-based development lies in the integrated view of architecture, behaviour, requirements and security. Security risks become apparent not only during final testing, but at a stage when the system design can still be modified.

For in-depth and automated analyses, Enterprise Architect can also be supplemented by specialised solutions – such as ThreatGet from AIT. SBA Research, as an independent partner and Austrian research centre, also makes an important contribution to the further development of Cyber Security and Threat Modelling.

The key message, however, remains:

Security by Design begins with a comprehensible, verifiable and traceable model.

The more autonomous our systems become, the more important it is to systematically model their capabilities and limitations before they are put into operation.