Python Scripts

Russian Text

5 files in python-scripts folder are related to Russian documents' pre-processing. The procedure includes:

• Converting pdf to docx: cleanpdf.py, process_directory function
• Creating large files to be uploaded on DeepL website: cleanpdf.py, merge_docx_files
• Recreating the original docx files: docxtopdf.py
• Converting docx files to pdfs: docxtopdf.sh, pandoc_latex_template.yaml

RAG Pipeline

6 files in the python-scripts folder are related to the RAG pipeline.

• Loading parameters: parameters.py, config.json
• Loading the embedding model: get_embedding_function.py
• Loading the LLM model: get_llm_function.py
• Creating the RAG chain: get_rag_chain.py
• Class to create the final Conversational RAG chain: ConversationalRagChain.py

parameters.py

This script is responsible for loading and managing configuration parameters, including API keys, paths, and model settings. It reads configuration from a config.json file and sets up global variables like DATA_PATH, CHROMA_ROOT_PATH, EMBEDDING_MODEL, and LLM_MODEL. It also includes functions to load API keys from a .env file and to print the current configuration.

Key Functions:

• load_api_keys(): Loads API keys from a .env file and sets them as environment variables.
• load_config(config_name, show_config): Loads configuration from config.json based on the given configuration name.
• print_config(): Prints the current configuration settings.

---

config.json

This file contains configuration settings for different environments or use cases. Each configuration specifies paths to data, the embedding model, and the language model to be used.

Example Structure:

• data_path: Path to the data folder.
• chroma_root_path: Path to the folder where the Chroma database will be stored.
• embedding_model: Name of the model used for embeddings.
• llm_model: Name of the language model used.

---

get_embedding_function.py

This script provides a function to load different embedding models based on the specified model name. It supports various models like openai, and voyage models.

Key Function:

• get_embedding_function(model_name): Returns the embedding function corresponding to the given model name.

---

get_llm_function.py

This script is responsible for loading language models (LLMs) based on the specified model name. It supports models like Mistral, Llama, and GPT-3.5-turbo.

Key Function:

• get_llm_function(model_name): Returns the language model function corresponding to the given model name.

---

get_rag_chain.py

This script creates a Retrieval-Augmented Generation (RAG) chain for question-answering tasks. The RAG chain combines document retrieval with language model generation to provide relevant answers based on retrieved context.

Key Function:

• get_rag_chain(params): Creates and returns a RAG chain configured with the specified parameters. It supports different search types like similarity, similarity with score threshold, and MMR (Maximal Marginal Relevance).

---

ConversationalRagChain.py

This script defines a class ConversationalRagChain that encapsulates a RAG application to enable natural conversations. It manages chat history, formats outputs, and integrates with the RAG chain created by get_rag_chain.py.

Key Class:

• ConversationalRagChain: A class that extends the Chain class to handle conversational interactions using RAG. It maintains chat history and formats responses appropriately.

Database

4 files in python-scripts folder are related to the Database.

• Loading parameters : parameters.py, config.json
• Loading the embedding model : get_embedding_function.py
• Function to create, reset, clear the database : populate_database.py

Main Function Documentation

The main() function in the script serves as the entry point for managing the application's database. It provides command-line interface options to clear, reset, and populate the database using a specified configuration.

Arguments

• --config: (Optional)
  Type: str
  Description: Specifies the name of the configuration used to populate the database. Configurations need to be filled in config.json.

• --reset: (Optional)
  Action: store_true
  Description: If this flag is set, the function will reset the database before populating it with the specified configuration. Resetting involves clearing the current data in the database (using the clear_database function) and then loading the new data.

• --clear: (Optional)
  Action: store_true
  Description: If this flag is set, the function will clear the database entirely. If the --config argument is also provided, the database will be cleared based on the specific subfolder related to the configuration.

Functionality

1. Populating the Database:

   • When only the --config option is provided without the --reset or --clear flags, the function populates the database directly based on the configuration specified.
   • The load_documents function is used to retrieve the data, which is then processed by split_documents into smaller chunks.
   • Finally, these chunks are added to the database using the add_to_chroma function.

2. Resetting the Database (--reset flag) with a Configuration:

   • If the --reset flag is combined with the --config option, the function will first clear the database subfolder specified by the configuration.
   • After clearing, the function proceeds to load the documents with load_documents, split them into chunks with split_documents, and then add them to the database using the add_to_chroma function.

3. Clearing the Database (--clear flag):

   • When the --clear flag is used, the function clears the database.
   • If a configuration file is specified using the --config option, the database is cleared for the subfolder that matches the EMBEDDING_MODEL defined in the config.
   • The clear_database function is invoked to handle the clearing process.

Example Usage

• Populate the Database:

  python populate_database.py --config CONFIG_NAME
  

• Reset the Database:

  python populate_database.py --config CONFIG_NAME --reset
  

• Clear the Database:

  python populate_database.py --config CONFIG_NAME --clear