/
AI Agent - Installation and configuration

AI Agent - Installation and configuration

Documentation valid as of Datafari 6.2

This documentation explains how to install the AI Agent, and how to configure it. If you want to read more about AI Agent, check the links above.

Find the project on our Gitlab: https://gitlab.datafari.com/sandboxespublic/datafari-ai-agent

Functional and API documentation: AI Agent - API documentation

Technical documentation: AI Agent - Technical documentation

Requirements

The AI Agent can be installed on a Virtual Machine, or on a server. This section presents the hard and soft requirement to run the services.

Hardware requirements

Minimal configuration:

  • Memory (RAM): 32 GB. (64 recommended)

  • Storage: 100 GB

Graphics Procession Unit (GPU):

  • Not required, but highly recommended if running GPU-accelerated models.

For developments and tests, we have been using the g5.4xlarge Virtual Machine, from AWS. This is the recommended configuration.

  • Processor (CPU): AMD EPYC 7R32 3.3 GHz

  • Memory (RAM): 64 GB.

  • Storage: 600 GB (SSD)

  • GPU Memory: 24 GB

  • vCPU: 16

Software requirements

AI Agent services should be able to run on Windows or any Linux distributions. However, monitoring scripts are only available for Debian and Ubuntu.

Operating System :

  • Debian, Ubuntu with sudo access (tested with Ubuntu 20)

  • Open HTTP or HTTPS port (default: 8888)

  • An internet connection to download the project from Gitlab, its Python dependencies, and the AI models.

Software:

  • Python 3.10

For developments and tests, we have been using Ubuntu 20.04 LTS, with Python 3.10.9

 

Installation using the script as standalone installer

There are several ways to install the AI Agent. The first one implies to download (or copy) the installation script from Gitlab, and upload it on to your server.

  1. Download the install.sh file (or create a copy of it).

You can download the script from Gitlab (master):

wget https://gitlab.datafari.com/sandboxespublic/datafari-ai-agent/-/raw/master/bin/install.sh

  1. Add execution rights to the file

sudo chmod +x install.sh

  1. Run the script with the proper arguments. In that case:

  • --full is essential. It will force the script to download the project from Gitlab

  • --gpu should be used if and only if the Large Language Model you plan has GPU support

  • --start is optional. You can use it to start the server right after the installation.

bash install.sh --full --gpu

If the operation is successful, the project and the dependencies should have been downloaded.

More information about the install.sh script arguments here: AI Agent - Technical documentation | install.sh

  1. Start the AI Agent (if you haven’t already)

cd datafari-ai-agent bash bin/start.sh

  1. Make sure the services are working by trying the /models URL.

Execution logs can be found in ./logs/aiagent-*.log.

Installation with manual project cloning

You can install the AI Agent without downloading the script first. This process implies manually cloning the project from Gitlab, and run the installation script from the bin folder.

  1. cd to the folder where you want to install the AI Agent. In our example, we want /opt/datafari-ai-agent to be the root of the projet. Make sure you have write, read and execute rights in that folder.

cd /opt

  1. Clone the Git repository.

Install git (only if not already installed):

sudo apt install git

Download the project:

sudo git clone https://gitlab.datafari.com/sandboxespublic/datafari-ai-agent.git

  1. Add proper rights to project and to the bin folder.

You can use datafari user, or any other non-root user. Please avoid using the root user for installation, this could cause errors.

Sudo as a specific user (if required):

sudo su datafari

Set the proper rights:

sudo chown $USER -R datafari-ai-agent cd datafari-ai-agent chmod +x bin/*

  1. Run the installation script with the proper arguments. In that case:

  • --gpu should be used if (and only if) the Large Language Model you plan has GPU support

  • --start is optional. You can use it to start the server right after the installation.

  • --logrotate (optional): Enable the automatic log rotation feature (using logrotate), and skip the prompt asking you to enable or disable it.

  • --nologrotate (optional): Disable the automatic log rotation feature (using logrotate), and skip the prompt asking you to enable or disable it.

bash bin/install.sh --gpu

If the operation is successful, the project and the dependencies should have been downloaded.

More information about the install.sh script arguments here: AI Agent - Technical documentation | install.sh

  1. Start the AI Agent (if you haven’t already)

bash bin/start.sh

  1. Make sure the services are working by trying the /models URL.

Execution logs can be found in ./logs/aiagent-*.log.

Manual installation

If, for any reason, you can’t or don’t want to use an installation script (for debugging purpose, for example), you may process a manuel installation.

  1. cd to the folder where you want to install the AI Agent. In our example, we want /opt/datafari-ai-agent to be the root of the projet. Make sure you have write, read and execute rights in that folder.

cd /opt

 

  1. Clone the Git repository. You will probably need to use sudo. If so, make sure your Linux user is the owner of the project.

sudo git clone https://gitlab.datafari.com/sandboxespublic/datafari-ai-agent.git sudo chown $USER -R datafari-ai-agent sudo chmod 775 -R datafari-ai-agent

 

  1. Add execution rights to the bin folder.

cd datafari-ai-agent chmod +x bin/*

 

  1. Install dependencies with:

Do NOT use sudo to install Python dependencies.

pip install -r requirements.txt

 

  1. Install llama-cpp-python:

If your environment SUPPORTS GPU, use:

pip install llama-cpp-python==0.2.77 --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu122 --force-reinstall --no-dep

If your environment DOES NOT SUPPORT GPU, use:

pip install llama-cpp-python==0.2.77 --no-dep

 

  1. Start the AI Agent

bash bin/start.sh

 

  1. Make sure the services are working by trying the /models URL or by checking the logs.

Execution logs can be found in ./logs/aiagent-*.log.

Start and stop the services

Once the AI Agent fully installed and configured, you can start and stop the services using the script available in the ./bin directory.

Use the following commands from the project root.

To start the services:

bash bin/start.sh

To stop the services:

bash bin/stop.sh

Execution logs can be found in ./logs/aiagent-*.log.

To read logs in real time, you can use the command (replace {logfilename} by the actual log file name):

tail -f ./logs/{logfilename}

If you are using models from Hugging Face, the first request processing might take a while since the Agent needs to download the model. However, the model will be stored in cache for the next requests.

Manually add new GGUF models

By default, our solution is configured to download models from Hugging Face. Check AI Agent - API documentation to see how to dynamically download models.

However, if you are hosting the AI Agent on a machine that does not have an internet connexion, or if you want to prevent the web services users from downloading new models, you can manually download GGUF models.

First, you need to make sure that the model you want to used it is supported by llama-cpp. Llama-cpp only supports the GGUF model format.

If conditions are met, you can manually download your model (from Hugging Face, for example). Then, upload your model on the server, directly into the ./model directory. Then, restart the AI Agent.

The model should now be detected, and can be called by properly setting the model parameter in the HTTP request.

If you need to change the folder in which models are stored, you can edit the LOCAL_DIR parameter in the .env file.

Troubleshooting

Having an issue installing or using the AI Agent? Have a look at this section, where we describe problems we met, and how to solve them.

Strange or empty response from the LLM

The services seems to working, but the output is empty or contains unexpected characters ? The issue might come from the LLM. Consider using another model, or different prompt.

Out Of Memory

Large Language Models consume a lot of RAM. That is why we recommend aiming for a 64 GB of RAM. In that case,the problem comes from the GPU VRAM

image-20241105-150204.png

In that situation, try editing the .env file, and set the value N_GPU_LAYERS to 30. Each optimal configuration depends on the model, so if you need more information about the configuration options in en .env file, check the llama-cpp-python documentation, or our technical documentation.

Error on start-up

Meeting this error when starting the AI Agent with enabled GPU ?

Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/llama_cpp/llama_cpp.py", line 75, in _load_shared_library return ctypes.CDLL(str(_lib_path), **cdll_args) # type: ignore File "/opt/conda/lib/python3.10/ctypes/init.py", line 374, in init self._handle = _dlopen(self._name, mode) OSError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /opt/conda/lib/python3.10/site-packages/llama_cpp/lib/libllama.so) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/ubuntu/datafari-ai-agent/main.py", line 8, in <module> from llama_cpp import Llama, CreateChatCompletionResponse File "/opt/conda/lib/python3.10/site-packages/llama_cpp/init.py", line 1, in <module> from .llama_cpp import * File "/opt/conda/lib/python3.10/site-packages/llama_cpp/llama_cpp.py", line 88, in <module> _lib = _load_shared_library(_lib_base_name) File "/opt/conda/lib/python3.10/site-packages/llama_cpp/llama_cpp.py", line 77, in _load_shared_library raise RuntimeError(f"Failed to load shared library '{_lib_path}': {e}") RuntimeError: Failed to load shared library '/opt/conda/lib/python3.10/site-packages/llama_cpp/lib/libllama.so': /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /opt/conda/lib/python3.10/site-packages/llama_cpp/lib/libllama.so)

This one seems to be related to the llapa-cpp-python version and the GLIBC installed on your machine.

Try reinstalling it using the following command from the project root:

pip install llama-cpp-python==0.2.77 --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu122 --force-reinstall

Alternatively, you can use this command to check the GLIBC version installed on your machine:

ldd --version

Solutions to install a specific version of GLIBC are suggested here (askubuntu.com).

Error on installation

An error about numpy version may occur during the installation.

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. langchain 0.1.12 requires numpy<2,>=1, but you have numpy 2.1.3 which is incompatible. langchain-community 0.0.28 requires numpy<2,>=1, but you have numpy 2.1.3 which is incompatible.

This is a known issue, that should not prevent the AI Agent from working. It might be fixed when langchain and langchain-community upgrade their numpy dependency version.