Spaces:
Running
Running
| title: AURA Neural Search | |
| emoji: 🔍 | |
| colorFrom: indigo | |
| colorTo: purple | |
| sdk: docker | |
| app_port: 7860 | |
| pinned: false | |
| # AURA // Neural Visual Search Engine | |
| A high-performance, fully local multi-modal visual search engine matching Pinterest's visual search and Google Lens. Built with **FastAPI**, **CLIP** (Contrastive Language-Image Pre-training), and **NumPy**. | |
| The search engine features a glassmorphism frontend and works with a high-resolution catalog of 100 fashion products streamed directly from Kaggle's fashion dataset mirror on Hugging Face. | |
| --- | |
| ## Setup & Running Guide | |
| ### 1. Prerequisites | |
| * Python 3.9 or higher | |
| * Recommended: [uv](https://github.com/astral-sh/uv) (fast Python package installer and manager) | |
| ### 2. Setup Virtual Environment & Install Dependencies | |
| #### Using standard `pip`: | |
| ```powershell | |
| # Create virtual environment | |
| python -m venv .venv | |
| # Activate virtual environment | |
| .venv\Scripts\activate | |
| # Install requirements | |
| pip install -r requirements.txt | |
| ``` | |
| #### Using `uv` (Recommended / Ultra Fast): | |
| ```powershell | |
| # Create virtual environment | |
| uv venv | |
| # Install requirements | |
| uv pip install -r requirements.txt | |
| ``` | |
| ### 3. Initialize the Catalog & Precompute Embeddings | |
| Run the catalog generator script. This will download 100 high-resolution fashion products (apparel, footwear, accessories, bags) and precompute their CLIP embeddings into `catalog.json` and `embeddings.npy`: | |
| ```powershell | |
| .venv\Scripts\python.exe generate_catalog.py | |
| ``` | |
| ### 4. Run the Web Server | |
| Launch the FastAPI development server using Uvicorn: | |
| ```powershell | |
| .venv\Scripts\python.exe -m uvicorn app:app --host 127.0.0.1 --port 8000 --reload | |
| ``` | |
| ### 5. Open the Web Application | |
| Open your browser and navigate to: | |
| **[http://127.0.0.1:8000/](http://127.0.0.1:8000/)** | |
| --- | |
| ## Production Deployment (Render & Vercel) | |
| This application is configured to run in a split deployment configuration: | |
| - **Backend**: Deployed on **Render** (as a Docker Web Service). | |
| - **Frontend**: Deployed on **Vercel** (hosting the static client assets). | |
| ### 1. Backend Deployment on Render | |
| Render will automatically build and run the backend using the provided `Dockerfile` (optimized for CPU to minimize build size and time). | |
| 1. Go to your **Render Dashboard** and click **New > Web Service**. | |
| 2. Connect your GitHub repository (`jashparmaraiml-20/Visual-Search`). | |
| 3. Select **Docker** as the Runtime (Render will automatically detect the `Dockerfile` at the root). | |
| 4. Set the Instance Type to **Free** (or higher). | |
| 5. Render will automatically inject the `$PORT` environment variable. The Dockerfile's launch command is configured to bind to `0.0.0.0:$PORT` automatically. | |
| 6. Once deployed, note down your Render Web Service URL (e.g., `https://visual-search-backend.onrender.com`). | |
| ### 2. Frontend Configuration & Vercel Deployment | |
| Before deploying the frontend, update `static/app.js` with your backend endpoint. | |
| 1. Open `static/app.js` and set the `API_BASE` variable at the top of the file to your Render backend URL: | |
| ```javascript | |
| const API_BASE = "https://your-backend.onrender.com"; // Replace with your Render URL | |
| ``` | |
| 2. Commit and push this change to your repository. | |
| 3. Go to **Vercel** and import your project from GitHub. | |
| 4. Set the **Root Directory** of the project to `static` (so Vercel only serves the static client assets: `index.html`, `app.js`, `style.css`, and `images/`). | |
| 5. Click **Deploy**. Vercel will host the frontend on its fast edge CDN. | |
| --- | |
| ## System Design & Core Concepts | |
| ### 1. Vector Embeddings with CLIP | |
| At the heart of the system is the CLIP model (`openai/clip-vit-base-patch32`), which aligns images and text in the same semantic space of 512 dimensions. | |
| * **Text Embedding**: A text query (e.g., "red sneakers") is tokenized and encoded by the text transformer into a 512-dimensional vector. | |
| * **Image Embedding**: An image is resized, normalized, and encoded by the vision transformer into a 512-dimensional vector. | |
| * **Cosine Similarity**: The cosine similarity between any two vectors $u$ and $v$ is calculated as: | |
| $$\text{Similarity}(u, v) = \frac{u \cdot v}{\|u\| \|v\|}$$ | |
| Since we normalize vectors to unit length ($\|u\| = 1$), the cosine similarity is simplified to the dot product: | |
| $$\text{Similarity}(u, v) = u \cdot v$$ | |
| This allows lightning-fast search using simple matrix multiplication. | |
| ### 2. High-Level Architecture | |
| The system consists of three main components: | |
| 1. **Premium Frontend UI**: Single Page Application implementing a glassmorphism theme, drag-and-drop image upload, instant search feedback, and instant visual similarity queries ("Find Similar" hover feature). | |
| 2. **FastAPI Web Server**: Serves API routes, runs the local CLIP inference engine, manages the product catalog, and serves static files (product images, HTML, CSS, JS). | |
| 3. **Local Vector Store**: Memory-resident vector database using NumPy matrix calculations. | |
| ```mermaid | |
| graph TD | |
| A[Client Browser UI] -->|Text Query / Image Upload| B[FastAPI Backend] | |
| B -->|Preprocess Input| C[CLIP Model Inference] | |
| C -->|Generate Embedding Vector| D[Vector Matcher] | |
| D -->|Cosine Similarity Dot Product| E[(Local Vector Store / Catalog)] | |
| E -->|Sorted Match Results| B | |
| B -->|JSON Response| A | |
| ``` | |
| --- | |
| ## Detailed Data Flows & Userflows | |
| ### Userflow 1: Text-to-Image Search (e.g., searching "red sneakers") | |
| ```mermaid | |
| sequenceDiagram | |
| autonumber | |
| actor User | |
| participant UI as Frontend (JS/CSS) | |
| participant API as FastAPI Server | |
| participant CLIP as CLIP Inference Engine | |
| participant DB as Local Vector Store (NumPy) | |
| User->>UI: Types "red sneakers" and clicks Search/presses Enter | |
| UI->>API: POST /api/search (Form Data: q="red sneakers") | |
| API->>CLIP: Generate text embedding (shape: [1, 512]) | |
| CLIP-->>API: Return embedding vector | |
| API->>DB: Query similarity (dot product against catalog embeddings matrix [N, 512]) | |
| DB-->>API: Sorted index and similarity scores | |
| API-->>UI: JSON response: [{product_id, title, score, image_url}, ...] | |
| UI->>User: Render matches with smooth fade-in animations | |
| ``` | |
| ### Userflow 2: Drag-and-Drop Image Visual Search | |
| ```mermaid | |
| sequenceDiagram | |
| autonumber | |
| actor User | |
| participant UI as Frontend (JS/CSS) | |
| participant API as FastAPI Server | |
| participant CLIP as CLIP Inference Engine | |
| participant DB as Local Vector Store (NumPy) | |
| User->>UI: Drags & drops or uploads an image (e.g., of a leather jacket) | |
| UI->>API: POST /api/search (Multipart Form Data: file) | |
| API->>CLIP: Generate image embedding (shape: [1, 512]) | |
| CLIP-->>API: Return embedding vector | |
| API->>DB: Query similarity (dot product against catalog embeddings matrix [N, 512]) | |
| DB-->>API: Sorted index and similarity scores | |
| API-->>UI: JSON response: [{product_id, title, score, image_url}, ...] | |
| UI->>User: Update results showing visually matching items | |
| ``` | |
| ### Userflow 3: "Find Similar" (Pinterest style) | |
| ```mermaid | |
| sequenceDiagram | |
| autonumber | |
| actor User | |
| participant UI as Frontend (JS/CSS) | |
| participant API as FastAPI Server | |
| participant DB as Local Vector Store (NumPy) | |
| User->>UI: Hovers product card, clicks "Find Similar" button | |
| UI->>API: POST /api/search/similar (JSON: {product_id: "prod-4"}) | |
| API->>DB: Retrieve cached embedding for "prod-4" (no CLIP run needed!) | |
| API->>DB: Dot product target embedding against all other catalog vectors | |
| DB-->>API: Sorted similarity rankings (excluding self) | |
| API-->>UI: JSON response: visually similar product catalog list | |
| UI->>User: Re-render results grid with similar products | |
| ``` | |
| --- | |
| ## Component Details | |
| ### 1. Vector Store (`vector_store.py`) | |
| Encapsulates vector insertion and query logic. | |
| * **Data Structure**: Standard Python `dict` mapping product IDs to product metadata, and a NumPy array storing normalized vector embeddings. | |
| * **Search Function**: Calculates cosine similarity using raw matrix-vector multiplication (`np.dot(embeddings, query_vector)`). | |
| * **Scaling / 1-Line Swap**: | |
| ```python | |
| # Simple switch in vector_store.py | |
| # self.provider = InMemoryVectorStore() | |
| # self.provider = QdrantVectorStore() # <-- SWAP HERE to scale to millions of items | |
| ``` | |
| ### 2. CLIP Model Wrapper (`models.py`) | |
| Handles downloading, caching, and inference of the CLIP model using `transformers` and `torch`. | |
| * **Model**: `openai/clip-vit-base-patch32` (balanced size, fast CPU execution, high-accuracy multi-modal embeddings). | |
| * **Device**: Auto-selects CUDA GPU if available; otherwise defaults to CPU. | |
| * **Methods**: | |
| * `get_text_embedding(text: str) -> np.ndarray` | |
| * `get_image_embedding(image: PIL.Image) -> np.ndarray` | |
| ### 3. Catalog Generator (`generate_catalog.py`) | |
| Initializes the system by downloading fashion product images and computing their embeddings. | |
| * Streams high-res (384x512) product data from Hugging Face. | |
| * Precomputes product visual embeddings and saves them to enabling instant server startup. | |
| ### 4. FastAPI Server (`app.py`) | |
| Defines the HTTP interface: | |
| * `GET /`: Serves UI. | |
| * `GET /api/products`: Fetches all catalog products. | |
| * `POST /api/search`: Multi-modal search. Handles either text query or uploaded image file. | |
| * `POST /api/search/similar`: Look up cached embedding for a given product ID and return matching items. | |
| * `POST /api/index`: Adds a new product to the catalog, downloads image, computes embedding, and saves to vector store. | |