Building an On-Premises sLLM at SANDS Lab

[Contents]

1. Introduction

2. On-Premises sLLM

- 2.1 Differences Between On-Premises, Cloud, and Closed Networks

- 2.2 What Is an sLLM?

- 2.3 On-Premises sLLM

3. Building an On-Premises sLLM

4. SANDS Lab’s On-Premises sLLM Deployment Process

1. Introduction

This report describes the on-premises sLLM developed by SANDS Lab while carrying out the IITP project “Cybersecurity-Specialized On-Premises sLLM (small large language model).” The project aims to create a safer enterprise environment by applying AI technology to cybersecurity.

To discuss on-premises sLLMs, we first need to address the following questions.

First, what is an on-premises sLLM?

Second, how do you build an on-premises sLLM?

This report explains these two topics in detail.

2. On-Premises sLLM

Before discussing on-premises sLLMs, let us explain the terms “on-premises” and “sLLM.”

On-premises refers to installing and operating software and data on a company’s own servers and infrastructure. It is important to distinguish on-premises deployment from the concepts of cloud computing and a closed network.

2.1 Differences Between On-Premises, Cloud, and Closed Networks

What, then, distinguishes on-premises deployment, the cloud, and a closed network?

2.1.1 On-Premises

On-premises means building and operating infrastructure yourself—in other words, setting up and running servers within your company.

It is the approach most companies used before cloud computing became widespread. It offers a high level of security and direct control, but requires substantial resources and personnel. Initial setup costs are also high, and maintenance requires considerable time and expense.

However, it provides strong security by allowing companies to manage sensitive data directly.

Another advantage of on-premises deployment is the ability to customize the system. Companies can design systems around their specific requirements and add the functions they need. For example, they can apply particular security policies or adopt data management practices that comply with internal rules. Such customized systems can make a significant contribution to operational efficiency and stronger security.

2.1.2 Cloud

Cloud computing means subscribing to services provided by major companies such as AWS and Azure and renting the resources you need.

It is also the approach used by most companies today. Cloud computing is cost-effective because upfront setup costs are low and resources can be used as needed. It also offers excellent scalability and flexibility, with access available from anywhere at any time. However, because data is stored on external servers, potential security concerns need to be considered.

Cloud services provide tools and features that help companies develop and deploy applications easily. For example, they offer infrastructure for training and deploying machine learning models, as well as tools for data analysis. With these tools, companies can develop services quickly and efficiently without building complex infrastructure themselves.

2.1.3 Closed Networks

A closed network is a network environment completely isolated from the external internet.

A military intranet is one example. Because a closed network has no external connections, it offers a very high level of security. However, maintaining and operating it requires substantial costs and resources, and it does not allow data exchange with external networks.

A key strength of a closed network is its ability to protect data thoroughly from external threats. However, the lack of external connectivity also makes it difficult to apply the latest software updates and security patches. Closed networks therefore require careful planning for internal security management and maintenance, and their software and systems must be designed without external dependencies.

2.2 What Is an sLLM?

Since the introduction of the Transformer, advances in LLMs have astonished the world.

Starting with GPT-3.5, numerous LLMs have appeared. However, users’ hardware capabilities have not kept pace with the increasing size of these models. Users therefore began making LLMs more lightweight so that they could run them on their own hardware and perform only the functions they needed. These models, smaller than conventional LLMs, became known as sLLMs (small language models). An sLLM generally refers to a model with fewer than 100 billion parameters.

Compared with conventional large LLMs, sLLMs are characterized by their ability to offer similar performance while using fewer resources. This is particularly useful in environments with limited hardware resources. They can also be trained for a specific domain to optimize performance for particular tasks. For example, healthcare sLLMs can analyze patients’ medical histories to support diagnosis, while financial sLLMs can analyze transaction data to detect anomalous transactions.

Another advantage of sLLMs is faster training and inference. Large LLMs take considerable time to process vast amounts of data, whereas the lightweight architecture of an sLLM enables faster processing. This is particularly useful for applications that require real-time responses.

2.3 On-Premises sLLM

We can now understand what the term “on-premises sLLM” means.

It is a small LLM built directly within a company—on premises—rather than an LLM provided by a major company such as OpenAI.

Why, then, do companies want to build on-premises sLLMs?

The reason is to prevent the leakage of confidential internal information.

Most companies are reluctant to disclose internal information externally. Using cloud-based OpenAI services raises concerns about the exposure of internal company information. This creates demand for a customized GPT with OpenAI-like capabilities that can be used exclusively within the company.

An on-premises sLLM can integrate with internal company data to provide a better customized solution.

For example, training a model on data tailored to a particular task can produce more accurate and efficient results for that task.

When a company’s IT infrastructure and data security policies are taken into account, deploying an on-premises sLLM can therefore significantly improve productivity and efficiency.

This begins with analyzing the company’s data environment and designing the required hardware and software. It is followed by training the sLLM and integrating it into internal company systems.

3. Building an On-Premises sLLM

The company’s size and budget are key considerations when building an on-premises sLLM.

1. For small companies, we recommend using an open-source sLLM.

Various freely available open-source LLMs can be used to implement basic functions. For example, an open-source model such as Llama3 provides strong performance and is available free of charge, allowing a model to be deployed on a limited budget.

2. If you can build a VectorDB (vector database) using internal company information, you can introduce RAG (Retrieval-Augmented Generation). RAG combines retrieval and generation techniques to provide more accurate and useful information. A VectorDB enables efficient searches of large volumes of internal documents and the generation of accurate answers based on them.

3. For companies that can fine-tune an sLLM, we recommend building a RAG system using an sLLM customized for the company.

This approach trains the model for specific tasks and uses it to provide more accurate information. For example, fine-tuning an sLLM with a Q&A dataset tailored to the company’s requirements can yield more accurate answers.

Of these options, the author recommends option 2. For example, the open-source Llama3 model now delivers performance close to GPT-3.5, while the embedding models used to build VectorDBs are also improving.

Therefore, building an internal VectorDB and using an open-source LLM is, in practical terms, the option that requires the least cost and time.

This may raise the following question.

“If RAG can improve the accuracy of a model’s answers, does that mean fine-tuning is unnecessary?”

The answer is: “It depends on the company’s circumstances and the domain in which the RAG system is built”.

Let us illustrate this with a question posed to the cybersecurity-specialized on-premises sLLM developed by SANDS Lab.

Q: Analyze whether the test.exe file is malicious and summarize the analysis results.

As shown in the figure above, Llama3 uses the CTX.io cyber threat intelligence API to request analysis of test.exe, receives the results, and responds. However, CTX.io analysis results generally consist of lengthy JSON data.

Llama3 has a token limit, so it cannot provide an answer when it is supplied with a large volume of data like this.

Answering questions based on this kind of data may require fine-tuning the open-source LLM.

Fine-tuning requires a variety of Q&A (question-and-answer) datasets.

SANDS Lab has built a Q&A dataset based on extensive cyber threat analysis data. This enables us to deploy cybersecurity-specialized on-premises sLLMs tailored to a company’s needs. An on-premises sLLM can further improve the security of internal company data and help prevent issues such as external threats and data leakage.

4. SANDS Lab’s On-Premises sLLM Deployment Process

SANDS Lab’s on-premises sLLM deployment process is as follows.

Requirements analysis: Analyze the company’s requirements in detail and define the necessary functions and performance.

Infrastructure design: Design the optimal infrastructure for those requirements and select the necessary hardware and software.

Model training: Train the sLLM using the company’s data and domain knowledge.

System integration: Integrate the trained model with the company’s existing systems so that they operate smoothly together.

Testing and validation: Thoroughly test the deployed system to confirm that it meets the required performance and functionality.

Maintenance: Provide ongoing maintenance and updates to ensure stable system operation.

Through a systematic deployment process and our own analysis technologies, SANDS Lab provides the optimal on-premises sLLM solutions for our customers’ needs.

SANDS Lab will continue working to transform our customers’ businesses with AI technology and create a safer cybersecurity environment.