Edge-AI for Audio & Music

What it does

• Recognises songs locally from a short clip or mic input using audio fingerprints.

• Understands style with CLAP embeddings for “sounds-like” retrieval

• Respects musicality by considering mood, tempo, key, and energy.

• Builds playlists in two modes: smooth continuity or planned contrast.

• Explains choices with short LLM blurbs via Ollama (optional, still offline).

Why I made it

Streaming “Flow” features can swing genres and moods randomly or get stuck. I wanted full control: my files, my tags, predictable transitions, and instant results without the cloud.

How it works (pipeline)

1. Fingerprint local tracks to identify a playing clip.

2. Extract features with librosa: tempo, key, energy.

3. Embed each track with CLAP (HTSAT-tiny) → 512-D vectors.

4. Index embeddings with FAISS for fast nearest-neighbor search.

5. Sequence tracks with a cost function that balances style distance, mood, tempo, key, energy.

6. Explain transitions with a small local LLM via Ollama (optional).
   
   

Highlights

• 100% offline: no external APIs or accounts.

• Windows-friendly: avoids Essentia on Windows, uses pure-Python fallbacks.

• One-command ingest: incremental, only processes new files.

• Unified CLI – python main.py ingest then python main.py playlist...

• Autoinstall guardrails – missing deps are installed at runtime when possible.

What I solved

• Dependency hell on Windows: replaced hard builds with pure-Python paths, optionalised fragile libs, added runtime installers and clear fallbacks.

• Repeat work: incremental ingest stores features and embeddings to avoid reprocessing.

• Chaotic transitions: sequencing engine that preserves mood while allowing planned style jumps.

What’s next

• Lightweight web UI for drag-and-drop clips and playlist export.

• GPU-accelerated FAISS on RTX for huge libraries.

Edge-AI Music Intelligence: “Local Music DSP + LLMs”

Local AI Music Identification & Recommendation System

Summary
I designed and implemented a fully offline music identification and recommendation engine that runs entirely on local hardware without relying on cloud APIs. The goal was to reproduce and extend “Shazam-like” song recognition while solving key issues I’ve observed in mainstream services such as Deezer Flow, namely abrupt mood/style shifts, lack of pattern continuity, and limited personalization logic.

Key Features & Pipeline

• Audio Fingerprinting:
  Used a local implementation (PyDejavu) to fingerprint my library into a SQLite database, enabling sub-second recognition from an 8-second microphone or file snippet.

• High-Level Audio Features:
  Extracted mood, energy, tempo, key, and timbral descriptors with Essentia(But I couldn't make this option work... So...); tagged genre, instruments, and mood labels with MusicNN.

• Audio Embeddings for Similarity:
  Computed 512-dimensional CLAP (HTSAT-Tiny) embeddings for “sounds-like” retrieval across genres. Stored in a FAISS index for instant nearest-neighbor search.

• Sequencing Engine (Custom anti-Flow logic):
  Implemented weighted cost functions to control transitions:

  • Smooth mode: maintain style & mood continuity.

  • Contrast mode: deliberately switch styles while avoiding mood/energy whiplash.

• Local LLM Explanations:
  Integrated Qwen 2.5 VL 3B (via Ollama) to produce short, natural-language justifications for each playlist transition—entirely offline.

• Resource Efficiency:
  Designed for controlled VRAM usage (≤ 8 GB) and predictable latency (< 0.5 s end-to-end).
  
  

Workflow

1. Ingest: Fingerprint, extract features, compute embeddings, build FAISS index (one-off).

2. Runtime:

   • Recognize currently playing track from snippet.

   • Retrieve candidates (smooth or contrast policy).

   • Sequence playlist using cost-based constraints.

   • Call local LLM for explanations.

3. Output: 20-track playlist with mood-aware sequencing and concise reasoning.
   
   

Impact & Uniqueness

• Fully autonomous, no internet required; ideal for privacy-sensitive or offline contexts.

• Balances musical surprise with emotional continuity, solving a common problem in auto-generated playlists.

• Modular design: fingerprinting, embedding, sequencing, and explanation components are independent and reusable.
  
  

Tech Stack

• Languages: Python

• Audio Processing: PyDejavu, Essentia, MusicNN, CLAP

• Vector Search: FAISS

• LLM Integration: Ollama (Qwen 2.5 VL 3B, DeepSeek R1 8B)

• Data Formats: SQLite, NumPy, JSON, YAML

• Hardware Optimization: CUDA-enabled PyTorch, quantized LLMs, float16 embeddings