How Does Kernaro AI Work with an On-Premises LLM? 

Kernaro AI can connect to the organisation’s local LLM through an OpenAI-compatible API.

Here is a simplified example of how it works:

1. A user asks a question: An architect asks Kernaro AI, “Which applications depend on our customer management system?”
2. Kernaro AI accesses the repository: Kernaro AI retrieves relevant elements and relationships from the configured Enterprise Architect repository.
3. The local LLM processes the request: Kernaro AI communicates with the organisation’s configured LLM endpoint to interpret the request and analyse the relevant information.
4. Kernaro AI presents the results: The user receives an AI-assisted response based on information retrieved from the repository.

The key difference is where the AI model processes the request.

• With a cloud-hosted LLM, inference takes place in the selected cloud provider’s environment.
• With an on-premises LLM, inference takes place within the organisation’s own infrastructure.

When the deployment, network routing, and integrations are configured to remain internal, the AI inference process does not require sending repository information to an external LLM provider.

This provides organisations with an additional option for adopting AI while working within their existing infrastructure and security policies.

Which On-Premises LLMs Can Be Used with Kernaro AI? 

Kernaro AI is not restricted to a particular brand of locally hosted LLM.

Instead, compatibility primarily depends on whether the model can be accessed through an OpenAI-compatible API and whether it supports the capabilities required by Kernaro AI.  Many AI deployment and serving platforms provide this capability.

Examples include: 

• Ollama: A platform that makes it easier to run and serve language models locally.
• LiteLLM: A gateway or proxy that provides a unified interface for communicating with supported AI models and services.
• Enterprise AI gateways: Organisations can also use their own internal API gateways or proxy services, such as Azure API Management, to provide access to compatible LLM endpoints.

These platforms offer different approaches to hosting, exposing, or accessing an LLM. For instance, an organisation running its own model through Ollama can expose an OpenAI-compatible endpoint that Kernaro AI can access.

Similarly, an organisation with a centralised AI platform may expose its models through a corporate gateway, allowing Kernaro AI to communicate with them using the supported API format.

Examples of LLMs Tested with Kernaro AI 

As part of our internal evaluation of on-premises LLM integration, the Kernaro AI team has tested several models, including:

• Qwen 3.5 9B
• Meta Llama 3.1 70B
• GPT-OSS 120B

These models represent different sizes and capabilities, demonstrating the flexibility of connecting Kernaro AI to different self-hosted AI environments. They are examples of models evaluated by our team rather than an exhaustive list of supported models.

Compatibility and performance can vary depending on the model, serving platform, API implementation, and hardware configuration. Organisations should evaluate their preferred model against their intended Enterprise Architect workflows.

Is Every OpenAI-Compatible LLM Suitable for Kernaro AI? 

Not necessarily. An OpenAI-compatible API allows Kernaro AI to communicate with a model, but compatibility at the API level is only the first requirement. 

Enterprise architecture tasks often involve understanding repository structures, retrieving related information, and performing multiple operations to complete a request. 

For the best experience, organisations should evaluate three important model capabilities. 

1. Tool and Function Calling

• This is one of the most important capabilities for Kernaro AI.
• Tool calling allows an LLM to request specific functions provided by an application.
• For example, when a user asks Kernaro AI to identify all applications connected to a particular business service, the model may need to invoke repository operations to retrieve the relevant elements and relationships.
• A model with reliable tool-calling capabilities can help Kernaro AI coordinate these operations.
• Without adequate tool-calling support, many repository-aware capabilities may not function as intended.

2. Reasoning Capabilities

• Enterprise architecture questions can require more than generating a simple text response.
  Consider a question such as:
• “If we retire this application, which other systems could be affected, and what dependencies should we review first?”
• Answering this may involve analysing several interconnected elements, interpreting relationships, and presenting conclusions based on repository information.
• Models with stronger reasoning capabilities are generally better suited to these multi-step workflows.

3. Vision Capabilities

• Some AI use cases involve understanding images, diagrams, or other visual information. 
• For these scenarios, the selected LLM should support vision, allowing it to process image-based inputs. 
• Vision is not required for every Kernaro AI operation, but it is important when the intended workflow relies on visual content. 
• The LLM serving platform must also support passing these inputs through the compatible API. 

A useful point to remember: Even if an LLM supports an OpenAI-compatible API, it may not support tool calling, advanced reasoning, or vision equally well. 

Organisations should evaluate their chosen model against actual Enterprise Architect use cases before adopting it for regular use.

Why Choose an On-Premises LLM?

• Greater Control Over AI Infrastructure
• Alignment with Security and Compliance Requirements
• More Control Over Data Processing
• Reuse Existing AI Infrastructure
• Flexibility in Model Selection

How Do You Get Started?

If your organisation already operates an on-premises LLM, getting started with Kernaro AI begins with a few straightforward checks. 

Step 1: Confirm That Your LLM Exposes an OpenAI-Compatible API 

• Check with your AI or infrastructure team whether the existing LLM platform exposes an OpenAI-compatible chat completions endpoint, commonly using a path such as /v1/chat/completions. 
• If your model currently uses a proprietary API, an appropriate gateway or proxy may be able to provide a compatible interface. 

Step 2: Verify the Model’s Capabilities 

• Ensure that your selected model supports reliable tool or function calling. 
• Also assess its reasoning capabilities and, if your intended workflows involve images, vision support. 

Step 3: Configure the LLM Endpoint in Kernaro AI 

• Configure Kernaro AI to communicate with your organisation’s LLM endpoint, either directly or through an approved gateway. 
• Your infrastructure team should verify that the endpoint is accessible and that the required authentication and network security settings are correctly configured. 

Step 4: Test with Your Enterprise Architect Repository 

Start with common repository tasks, such as: 

• Retrieving elements from a selected package. 
• Identifying relationships between applications. 
• Exploring dependencies across architecture components. 
• Summarising information about a particular system. 
• Analysing potential impacts of architecture changes. 

These tests help determine whether the selected model can reliably support your intended Kernaro AI workflows.

A Quick Checklist for Your IT Team 

Before integrating an on-premises LLM with Kernaro AI, confirm the following:

1. API Compatibility: Does our LLM environment expose an OpenAI-compatible API? 
2. Tool Calling: Does the model support reliable tool or function calling? 
3. Reasoning: Can the model handle complex, multi-step architecture questions? 
4. Vision: Does the model support image-based inputs if required? 
5. Connectivity: Can Kernaro AI securely access the LLM endpoint from its deployment environment? 

If these requirements are satisfied, your infrastructure may be a suitable candidate for Kernaro AI’s on-premises LLM integration.

Cloud or On-Premises: The Choice Is Yours 

• Every organisation has different AI infrastructure and governance requirements. 
• Some prefer cloud-hosted LLMs for their accessibility, scalability, and managed services. 
• Others prefer locally hosted models to maintain greater control over infrastructure, model deployment, and information processing.

Kernaro AI supports both approaches

• Organisations can continue using supported cloud-hosted LLMs from providers such as OpenAI, Microsoft Azure OpenAI, and Anthropic Claude, or connect to their own on-premises LLM through an OpenAI-compatible API. 
• This flexibility allows organisations to adopt AI-powered Enterprise Architect capabilities in a way that aligns with their existing technology strategy.

Ready to Bring Your Own Model to Kernaro AI? 

• Your organisation’s AI strategy should work with your infrastructure, not require you to replace it. 
• With support for on-premises LLMs, Kernaro AI gives organisations more freedom to choose how they bring AI into their Enterprise Architect workflows. 
• Whether your organisation already runs an internal LLM or is evaluating one, the Kernaro AI team can help you understand the integration requirements and identify a suitable approach.