DRDO

AutoGen Search: My Project at DRDO

Video demo of AutoGen Search:

This blog details my work as a Software Development Intern at the Defence Scientific Information & Documentation Centre (DESIDOC), a lab of the prestigious Defence Research and Development Organization (DRDO), Government of India. Here, I developed AutoGen Search, an AI-powered search engine for the DRDO eLibrary.

(For the detailed version with code files, kindly proceed to GitHub: https://github.com/Anchor27/AutoGen-Search)

Problem Statement & Solution

1. About DESIDOC, DRDO

DESIDOC, DRDO manages over 300,000 books, technical reports, and scientific publications of the Defence Science Library (DSL), Digital Library, and various e-Services for defence purposes.

2. Problem Statements and Proposed Solutions

Despite the extensive collection of valuable resources, the DRDO eLibrary faced significant challenges in its search functionality:

Auto-Complete Suggestions:

Problem: Absence of an autocomplete-suggestions in search bar required users to manually type full queries, increasing errors and making searches inefficient.
Solution: An 'Auto-complete suggestions' integrated in search-bar implemented via Elasticsearch that provides the user with real time search suggestions based on the database of available books, journals and other e-resources of the eLibrary.

Search Summary:

Problem: Users had to open individual books or journals to determine their relevance, leading to time-consuming and inefficient searches.
Solution: Providing AI-powered search summary to user, powering from Large Language Models (GPT 3.5 Turbo in this case).

Architecture

User search-prompt input

User opens the search page having a search-bar, a search-button and a dedicated area for displaying the LLM powered search results.
User starts typing the search-query in the search bar.

Auto-Complete suggestions

The incomplete user query is sent to Elasticsearch.
Elasticsearch has an indexed database of the titles of books, journals and other scientific materials. Elasticsearch performs a search against this database.
Elasticsearch returns the list of sentences that match the user-search query to the frontend.
Frontend displays the list of Auto-complete suggestions in the form of a drop-down.
User can either select any of these Auto-complete suggestions or complete the query himself.
User presses the search-button which sends the query to the backend.

LLM powered search-results

Backend processes the query and feeds it into the GPT 3.5 Turbo LLM.
GPT generates the search results using a custom prompt specialized for generating search-engine like results.
The generated results are then sent to the frontend which is then displayed on the Webpage.

Components

The 4 main components of AutoGen Search are: User, Frontend, Backend, Database and Search

1. User

The user can be a web-based user using any of the modern web-browsers.

2. Frontend

The Frontend of AutoGen Search is developed using HTML, CSS, and JavaScript (jQuery) to facilitate the communication with the backend.

Search Bar: Search bar connected to the autocomplete feature powered by Elasticsearch.
Autocomplete Dropdown: When a user begins typing, frontend makes an AJAX call to retrieve potential query completions from Elasticsearch.
Results Display: After the user submits a query, the search results—generated by the LLM—are presented in a styled results section.

3. Backend

It's responsible for managing communication between the Frontend, Elasticsearch, and the Large Language Model (LLM).

The key functions of the backend include:

Routing: The /autocomplete route handles autocomplete requests & /process_query route manages LLM-powered query processing.
Elasticsearch Integration: It sends partial user queries to Elasticsearch for fetching autocomplete suggestions in real-time.
LLM Integration: The backend communicates with OpenAI’s GPT-3.5 Turbo model via the OpenAI API to generate summarized search results.
JSON-based API: The backend responds to both Elasticsearch and LLM requests in JSON format, ensuring seamless data exchange.

4. Elasticsearch

In the AutoGen Search system, Elasticsearch is used to handle autocomplete suggestions based on the user’s partially typed query.

Indexing Queries: A database of titles of various books, journals and other scientific publications of DRDO eLibrary is indexed into Elasticsearch, forming the database.
Real-Time Search: When a user types in the search bar, Elasticsearch receives the partial query and searches the indexed database in real-time. It then returns matching suggestions that appear as dropdown.

Tech Stack

1. Frontend

CSS + HTML: For the layout and styling of the webpage.
AJAX: For asynchronous requests to ensure that users experience real-time interaction with the search bar.
jQuery: To manipulate DOM elements dynamically.

2. Backend

Python: Programming Language
Flask (Python): Used for creating the web server and handling API requests.
OpenAI API: For interfacing with GPT-3.5 Turbo to generate search summaries.
REST API: Communication protocol between the frontend and backend.

3. Elasticsearch (Database and Search)

Elasticsearch Engine: For indexing and querying data in real-time.
Elasticsearch Python Client and API: Facilitates the communication between the Flask backend and Elasticsearch.

Google Sites

Report abuse