To manage ballooning database space and improve similarity search, we have fundamentally refactored how browsing session data and embeddings are stored. Heavy raw payloads like DOM content and user interactions are now compressed and offloaded to AWS S3, keeping the Postgres tables lean by only storing object keys and metadata. In parallel, text embeddings were migrated to the native vector(768) type via the pgvector extension, leveraging an HNSW index to replace slower application-side cosine distance calculations. This architectural shift significantly cuts our database footprint while delivering incredibly fast and scalable semantic search capabilities. Database migration meme

Migrated storage architecture to S3 and introduced native pgvector queries to optimize database performance. - subh05sus/cosmyc-ai