The Problem: Why Relational Databases Can't Handle AI Memory
I spent three months last year debugging why a utility's AI assistant kept forgetting context between sessions. The culprit wasn't the LLM — it was PostgreSQL trying to do similarity search across 400,000 maintenance records. Query times exceeded 30 seconds. The answer was obvious once I stopped treating embeddings like regular data: we needed a vector database.
Vector databases solve a specific problem that traditional databases can't handle efficiently: finding semantically similar information in high-dimensional space. When your LLM converts "turbine vibration anomaly" into a 1,536-dimension embedding vector, you need infrastructure that can search millions of those vectors in milliseconds, not minutes. This isn't theoretical — it's the difference between a RAG system that works and one that times out.
At EthosPower, we've deployed Qdrant, Weaviate, Milvus, and ChromaDB across different energy sector environments. Each has distinct trade-offs that matter when you're operating under NERC CIP constraints or running air-gapped systems in substations. If you're evaluating these platforms, ask our AI assistant for a personalised comparison based on your specific infrastructure.
Architecture Reality: How Vector Databases Actually Work
Vector databases use approximate nearest neighbor (ANN) algorithms to search high-dimensional space efficiently. The key word is "approximate" — you're trading perfect accuracy for speed. In practice, this means HNSW (Hierarchical Navigable Small World) graphs or IVF (Inverted File) indexes that get you 95-99% recall in 2-5ms instead of 100% recall in 2,000ms.
Here's what that architecture looks like in production:
- Ingestion pipeline: Documents → chunking → embedding model → vector insertion with metadata
- Index structure: ANN algorithm builds searchable graph or tree structure
- Query path: Query text → embedding → ANN search → filtered results → reranking
- Storage layer: Vectors in optimized format (often memory-mapped), metadata in separate store
The difference between platforms comes down to implementation details. Qdrant uses Rust with memory-mapped files and can run entirely in RAM for sub-millisecond queries. Weaviate is Go-based with native GraphQL APIs and built-in vectorization. Milvus relies on separate components (MinIO, etcd, Pulsar) for distributed deployments. ChromaDB started as an embedded database and still excels at single-node scenarios.
Component Deep-Dive: What You're Actually Deploying
Qdrant: The Rust Pragmatist
Qdrant is what I deploy when data sovereignty matters and air-gapped operation is non-negotiable. Written in Rust, it's a single binary that runs on a Raspberry Pi or scales to billions of vectors. I've run it on OT networks with 4GB RAM serving 50,000 equipment manuals with 3ms p99 latency.
Key architectural decisions:
- Immutable segments with copy-on-write updates minimize lock contention
- Payload filtering happens during ANN search, not after (10x faster for metadata queries)
- HNSW index parameters (M=16, ef_construct=100) provide good defaults, tune ef at query time
- Memory-mapped storage means vectors stay on disk until accessed, critical for large datasets
Configuration that matters: Set storage_type: Mmap for datasets larger than RAM, use on_disk_payload: true to keep metadata searchable without loading everything into memory. For NERC CIP environments, Qdrant's lack of external dependencies means simpler security boundaries.
Weaviate: The Hybrid Operator
Weaviate's distinguishing feature is hybrid search — combining vector similarity with BM25 keyword search in a single query. This matters for energy documentation where exact model numbers matter as much as semantic meaning. Searching for "SEL-351 relay" needs to match the exact model, not just "protective relays in general."
Architecture notes:
- Built-in vectorization modules (text2vec-transformers, etc.) simplify deployments
- GraphQL schema enforcement prevents metadata chaos
- Sharding by class allows different retention policies per data type
- Backup/restore via filesystem snapshots, not logical dumps
I deployed Weaviate 1.23 at a renewable energy company handling 2M+ solar panel inspection reports. Hybrid search eliminated false positives from pure semantic search ("cracked panel" vs. "panel crack pattern analysis" are different failure modes). Query performance: 8ms p99 with alpha=0.7 favoring vector search.
Milvus: The Distributed Beast
Milvus is overengineered for most energy sector deployments, but when you need trillion-scale search across global operations, it's the only open-source option that works. The architecture mirrors Snowflake: separate compute and storage, Pulsar for message queuing, MinIO for object storage, etcd for coordination.
Deployment reality:
- Minimum viable cluster: 8 nodes (2 query, 2 data, 2 index, coordinator, proxy)
- Resource requirements: 64GB RAM minimum per node, NVMe storage for indexes
- Operational complexity: You're managing a distributed system, not a database
- Kubernetes-native, Helm charts assume you know what you're doing
I only recommend Milvus when you have dedicated platform engineering and datasets exceeding 100M vectors. A large oil & gas company uses it for seismic data similarity search (500M+ waveform embeddings). Query latency: 12ms p95, but that's across petabytes of underlying data.
ChromaDB: The Development Darling
ChromaDB started as an embedded database for local development and that's still its strength. It's SQLite for vectors — one pip install and you're running. For prototyping RAG applications or departmental AI tools that don't need enterprise scale, it's perfect.
Practical limits I've observed:
- Sweet spot: 100K-1M vectors, single server
- Beyond 5M vectors, query latency degrades noticeably
- No built-in sharding or replication (client-server mode is relatively new)
- Excellent for air-gapped laptops running local LLMs
I use ChromaDB for isolated OT environments where a field engineer needs AI assistance on a ruggedized laptop with no network connectivity. Load 50K equipment manuals, run Ollama locally, query in 15ms. Simple, reliable, no cloud dependencies.
Operational Reality: What Breaks in Production
Memory Management
Vector databases are memory-intensive. A 1M document collection with 1536-dim embeddings consumes 6GB just for vectors (1M 1536 4 bytes). Add HNSW index overhead (3-4x) and metadata, you're at 30GB minimum. Plan capacity assuming 5x raw vector size.
Qdrant handles this best with memory-mapped storage. Weaviate requires more RAM for equivalent performance. Milvus distributes the problem but adds operational complexity. ChromaDB works until you exceed available RAM, then performance falls off a cliff.
Index Rebuild Times
Changing index parameters requires rebuilding. For 10M vectors, expect:
- Qdrant: 15-20 minutes (single-threaded, Rust efficient)
- Weaviate: 25-30 minutes (Go, more CPU cores help)
- Milvus: Distributed rebuild, 10-15 minutes with proper cluster sizing
- ChromaDB: 30-45 minutes (Python overhead visible)
This matters for NERC CIP compliance where you can't have long maintenance windows during grid operations. I schedule index updates during low-load periods and maintain read replicas.
Backup and Recovery
Vector databases don't back up like PostgreSQL. You're dealing with binary indexes and memory-mapped files. Qdrant supports snapshots (filesystem-level, works well). Weaviate has backup modules (backup to S3/GCS/filesystem). Milvus uses MinIO snapshots (complex but works at scale). ChromaDB is just files, copy them.
For NERC CIP compliance, I maintain full snapshots plus incremental metadata backups. Recovery time objective: 15 minutes for Qdrant/ChromaDB, 45 minutes for Weaviate, 2+ hours for Milvus (distributed system complexity).
Integration Patterns: RAG and Beyond
Most energy sector deployments use vector databases for Retrieval-Augmented Generation. The pattern:
- User query → embedding model → query vector
- Vector DB retrieves top-k similar documents (k=5-20)
- Rerank results by metadata (date, source, classification)
- Inject top-3 into LLM context
- LLM generates response grounded in retrieved data
But I've also deployed vector databases for:
- Equipment failure pattern matching: 500K historical failure reports, find similar failure modes in 5ms
- Regulatory compliance search: NERC standards, IEEE guides, vendor documentation — semantic search across heterogeneous sources
- AI memory systems: Combine vector search with Neo4j knowledge graphs for persistent agent memory
- Anomaly detection: SCADA timeseries → embeddings → detect abnormal operational patterns
The Neo4j integration is particularly powerful. Store entity relationships in Neo4j (substations, transformers, protection schemes), store document embeddings in Qdrant, query both for "show me all similar incidents at substations with the same transformer configuration." We've built this pattern at three utilities now. For understanding how these architectural decisions impact your project timeline and costs, try the AI Implementation Cost Calculator.
Deployment Models: Cloud, On-Prem, Air-Gapped
Cloud Deployments
Qdrant Cloud and Weaviate Cloud exist but I rarely use them for energy sector work. Data sovereignty concerns mean most deployments are self-hosted. If you do use cloud:
- Run in your own VPC (Qdrant supports this, Weaviate too)
- Encrypt at rest and in transit (non-negotiable for CIP-002+)
- Understand data residency (which AWS region, what jurisdiction)
Milvus on cloud requires significant Kubernetes expertise. ChromaDB cloud offering is too new for production energy deployments.
On-Premises
This is where 80% of our deployments live. Qdrant on Ubuntu 22.04 LTS, 64GB RAM, NVMe storage, behind the corporate firewall. Typical configuration:
```
qdrant:
storage:
storage_path: /mnt/nvme/qdrant
service:
max_request_size_mb: 64
grpc_port: 6334
hnsw_index:
m: 16
ef_construct: 100
```
Weaviate deployments use Docker Compose for smaller sites, Kubernetes for enterprise. Milvus is Kubernetes-only in practice (Helm charts are the supported path). ChromaDB runs as a systemd service for single-server deployments.
Air-Gapped OT Networks
This is where platform choice really matters. Qdrant and ChromaDB work flawlessly — single binary or Python package, no external dependencies, no license server callbacks. Weaviate requires more planning (vectorization modules need to be bundled). Milvus is impractical (too many distributed components, assumes internet access for container pulls).
I've deployed Qdrant on Siemens ruggedized industrial PCs running Ubuntu Core in substations. 16GB RAM, 500K vectors, 4ms query latency, zero network dependencies. That's the gold standard for OT AI infrastructure.
The Verdict
After five years deploying vector databases in energy operations, here's what I actually recommend:
Choose Qdrant if data sovereignty, air-gapped operation, or NERC CIP compliance drives your architecture. It's fast, operationally simple, and works in resource-constrained environments. I've never had a Qdrant deployment fail due to the database itself.
Choose Weaviate if hybrid search matters and you need built-in vectorization. The GraphQL API simplifies integration for teams comfortable with modern web stacks. Good fit for enterprise IT environments with existing Kubernetes.
Choose Milvus only if you have platform engineering resources and truly need trillion-scale search. It's powerful but operationally complex. Most energy companies don't need this level of scale.
Choose ChromaDB for prototyping, isolated deployments, or field equipment running local AI. It's not enterprise infrastructure, but it's excellent at what it does.
My default recommendation: Start with Qdrant, run it on-premises, scale to millions of vectors before considering anything else. If you need specific deployment guidance for your infrastructure, try EthosAI Chat to get recommendations tailored to your operational environment.