Project Description: 

Vector databases and approximate nearest neighbor (ANN) search systems are built on an implicit assumption: that high-dimensional embeddings (e.g., 768- or 1536-dimensional vectors from language or vision models) actually live on a much lower-dimensional structure. This "intrinsic dimensionality" (ID)---as opposed to the ambient dimensionality of the vector itself---is believed to govern how hard a dataset is to index and search efficiently, yet ID is difficult to estimate reliably, and different estimators can disagree substantially, especially under noise, curvature, or non-uniform density.

This project asks: can we reliably estimate the intrinsic dimensionality of real embedding datasets, and does that estimate tell us anything useful about how those datasets behave in nearest-neighbor search?

Learning Objectives:

* Understand what intrinsic dimensionality means (as distinct from ambient dimensionality) and why it matters for the curse of dimensionality in nearest-neighbor search.

* Understand the mechanics and assumptions behind several classical ID estimation methods.

* Understand how ANN indexes (e.g., HNSW, IVF) work at a conceptual level, and why search cost and recall depend on data geometry.

* Designing and running controlled synthetic experiments with known ground truth.

* Implementing and validating statistical estimators in Python.

* Working with real-world embedding datasets: loading, preprocessing, and computing k-NN structures at scale.

* Using ANN libraries (FAISS/hnswlib) and interpreting recall/latency benchmarks.

* Collaborative software engineering: shared repo, code review, reproducible experiment scripts.

Skills needed: 

Python, basic statistics, Git/Github, familiarity with embeddings, ANN libraries