Highlights

APPROACH

The key steps of the project include:

1. Document Loading and Preprocessing:

   • The system utilized a custom document loader to handle various file formats, including .txt, .pdf, and .docx.

   • Extracted text was then split into smaller chunks using recursive chunking with an overlap to maintain context across segments. This step is crucial for efficient LLM processing.

2.  Embedding and Storage:

   • Each text chunk was converted into a vector representation using OpenAI's embedding model.

   • These embeddings were stored in a vector database (Qdrant) for efficient retrieval during the question-answering process.

3. Retrieval Strategy:

   • To retrieve relevant contract documents for a given user question, a Maximum Marginal Relevance (MMR) retrieval approach was implemented within Qdrant.

   • MMR prioritizes diversity in retrieved documents, ensuring a broader range of perspectives and information is presented in the response.

4. Response Generation:

• Langchain's ConversationalRetrievalChain was employed to generate the final response.

• This chain leverages the OpenAI LLM for response creation, while incorporating the retrieved documents and a memory component.

• The memory component allows the system to retain context from previous interactions, fostering a more coherent and natural conversation flow.

This project aims to develop a novel Redash chat add-on that allows users to extract insights from data using  natural language. 

Outcome:

• Users can ask questions about existing Redash dashboards in natural language.

• Generate SQL queries based on user questions using Large Language Models (LLMs).

• Automatically generate visualizations from SQL queries or existing visualizations.

• Create new Redash dashboards based on user queries.

Technical Skills and Knowledge

• 1. Data Analysis & Visualization

  2. Programming & Development

• Proficiency in Python and Javascript (React), Experience with SQL, Understanding complex codebases, Prompt engineering, OpenAI API usage, Add-on/plugin development experience

• 3. SQL & Database Management

• Experience with Celery for concurrent tasks, Docker and Docker Compose for deployment

• 4. Natural Language Processing (NLP)

  5. Vector databases

  6. Machine Learning & AI

Approach

1. Create Database Schema

• • Design a schema to efficiently store and query YouTube data (channel performance, user base, video expenses).

• Building Redash Chat Add-on

  • Frontend (React): Design and implement a user-friendly dashboard interface for the add-on.

  • Backend (Quart - lightweight async Flask): Build a system for data storage and processing based on the designed schema.

• LLM Integration

  • Integrate LLMs (OpenAI or LangChain) to understand and process natural language queries for data insights.

  • Develop functionality within the add-on to translate natural language queries into actionable SQL queries compatible with Redash.

• Automatic Dashboard Generation

  • Implement functionalities for:

    • Automatic visualization generation based on user queries, generated SQL queries, or existing visualization context.

    • Creating new Redash dashboards using a collection of visualizations.