The fact that open source software can be downloaded free of charge is one of the many benefits it has over commercial products. Anyone can download the code, and if they have the correct hardware and software configuration, they can use it immediately.
There are two parts of being open with artificial intelligence (AI). The source code of the AI engine can be downloaded, inspected, and run on appropriate hardware just like any other open source code. But Open also applies to the data modelswhich means that it is possible for someone to run an AI model locally that has been trained.
With the right hardware, developers can download an AI model and run it locally, without worrying about query data being sent to a cloud AI service.
Since it is open-source, the AI models can be installed locally, so they do not incur the costs of cloud-hosted AI engines. These are usually charged based on how many tokens the AI engine receives.
How does an AI model that is open differ from one that is commercial?
Software must be licensed. Commercial products are increasingly being charged on a monthly subscription basis. In the case of large languages models (LLMs), this cost is correlated to the amount used, based on both the volume of tokens sent to the LLM, and the hardware consumption in hours of graphics processor unit (GPU time) when the model is queried.
An LLM that is open-source is subject to the licensing scheme’s terms and conditions, just like all open-source software. There are some restrictions on the use of this software, but there is no charge for running an open model locally.
There is a charge for the open model if it is run on public cloud infrastructure, or accessed via a cloud service. This is calculated based upon the Volume of tokens submitted by the LLM using APIs.
What is the benefit of open-source AI models?
Their openness, in addition to the fact that they are available for download and deployment on-premise at no additional cost, helps to advance the development of the AI model. This is similar to the way the open-source community can improve projects.
Like other open-source projects, an AI project that is open-source can be checked by anyone. This will help improve the quality of the model, remove bugs, and reduce bias when source data used to train a model are not diverse enough. The following podcast explores AI further.
How do I get started with open model
The majority of AI models provide free or low-cost web access to allow people to interact directly with the AI system. API access is usually charged based on tokens, such as words in a query. Output tokens can also be charged, which are a measure of data produced by a model when it answers a query.
An open model, which is open-source, can be downloaded via its open-source repository (“repo”) at GitHub. The repository contains different builds of the target systems, such as Linux, Windows or MacOS.
While this is the way developers use open source code, a data scientist might just want to “try out” the latest and greatest model without having to go through the sometimes arduous process of setting it up and running.
Enter Hugging Face, a platform that allows people to research AI models and test them using datasets from a single place. Hugging Face offers a free version but also offers an enterprise subscription for AI model developers and different pricing options for hosting and running models.
Ollama is another option. It’s an open-source, command-line program that makes it easy to download and run LLMs. Open WebUI is an open-source project on GitHub that provides a full graphical interface for LLMs.
Open source AI models to support corporate IT security.
Cybersecurity leaders have raised concerns about the ease of accessing popular LLMs by employees, which poses a risk of data leakage. Samsung Electronics, which uses ChatGPT to assist developers in debugging code, is one of the many leaks that have been widely reported. The code, which was Samsung Electronics’ intellectual property, was uploaded to the ChatGPT LLM public and became part of the model.
The technology giant took immediate steps to ban ChatGPT. However, the growth of so-called copilots as well as the rise of agentic artificial intelligence could lead to data leakage. Software providers who use agentic technology often claim that they keep private data of customers separate. This means that such data is not being used to train AI models. The model will quickly become outdated if it is not updated with the latest thinking, shortcuts and best practices.
A model of AI that is open can also be customized. Run in a secure sandbox, either on-premises or hosted in a secured public cloud. This model is a snapshot of an AI model that was released by the developer. It will quickly become outdated and irrelevant, just like AI in enterprise software.
But, any information that is fed into the model will remain within its confines, if organisations invest the resources necessary to retrain it using this information. In essence, new enterprise content and structured information can be used to train the AI model on the specifics of the business.
What do you need?
YouTube videos demonstrate that an LLM like the Chinese DeepSeek R1 modelis compatible with an Nvidia Jetson Nano embedded device or even on a Raspberry Pi using a GPU card that is relatively new and a suitable adapter. If the GPU is supported then it will also need a lot of video memory (VRAM). For best performance, it is important that the LLM runs in memory on GPU.
For inference, less memory is required and fewer GPU cores. However, the faster a model can respond, the more tokens it will be able to process per second. The number of GPU cores required for training LLMs increases dramatically, resulting in the need for expensive on-premise AI servers. Even if GPUs are used in the public cloud, with a metered usage model, the costs of running inference workloads continuously will remain high.
However, the sheer computing power of hyperscalers may make it cost-effective to upload training data into an open LLM hosted in a cloud.
How to make open-source AI models more affordableAs its name implies, a large language is large. LLMs require massive datasets for learning and vast farms of powerful servers to train. Even if the AI model is open-source, the cost of the hardware makes it impossible for organisations to fully operationalise LLMs unless they are willing to invest in hardware upfront or reserve GPU capacity on the public cloud.
However, not everyone requires an LLM. This is why models that run on cheaper hardware are so popular. These small language models (SLMs) require less computing power and some can even be run on smartphones, personal computers, and edge devices.
