Skip to content

barrel_vectordb Features

Additional features and advanced functionality for barrel_vectordb.

DiskANN Index

DiskANN is an SSD-optimized vector index based on the Vamana graph algorithm. It provides efficient billion-scale vector search with sub-linear memory usage.

Key Features

  • Two-pass Vamana construction - Builds a high-quality graph with alpha-RNG pruning
  • FreshVamana streaming updates - Insert and delete without full rebuild
  • Hot layer - Sub-millisecond writes absorbed in memory, compacted to disk
  • Lazy graph loading - O(1) startup time with on-demand node loading
  • PQ compression - Product quantization for reduced memory footprint
  • 4KB sector-aligned I/O - Optimized for SSD performance

Usage

%% Create a new DiskANN index
{ok, Index} = barrel_vectordb_diskann:new(#{
    dimension => 768,
    r => 64,              %% Max out-degree
    l_build => 100,       %% Build search width
    l_search => 100,      %% Query search width
    alpha => 1.2,         %% Pruning factor (>1 for long-range edges)
    distance_fn => cosine %% cosine or euclidean
}).

%% Build from a list of vectors
Vectors = [{<<"id1">>, Vector1}, {<<"id2">>, Vector2}, ...],
{ok, Index1} = barrel_vectordb_diskann:build(Options, Vectors).

%% Insert a single vector
{ok, Index2} = barrel_vectordb_diskann:insert(Index1, <<"id3">>, Vector3).

%% Batch insert (more efficient)
{ok, Index3} = barrel_vectordb_diskann:insert_batch(Index2, [
    {<<"id4">>, Vector4},
    {<<"id5">>, Vector5}
], #{}).

%% Search for K nearest neighbors
Results = barrel_vectordb_diskann:search(Index3, QueryVector, 10).
%% => [{<<"id1">>, 0.95}, {<<"id2">>, 0.89}, ...]

%% Search with options
Results = barrel_vectordb_diskann:search(Index3, QueryVector, 10, #{
    l_search => 200  %% Higher = better recall, slower
}).

%% Delete a vector (lazy delete)
{ok, Index4} = barrel_vectordb_diskann:delete(Index3, <<"id1">>).

%% Check if consolidation needed (deleted > 10% of active)
case barrel_vectordb_diskann:needs_consolidation(Index4) of
    true -> {ok, Index5} = barrel_vectordb_diskann:consolidate_deletes(Index4);
    false -> Index5 = Index4
end.

%% Get index info
Info = barrel_vectordb_diskann:info(Index5).
%% => #{size => 4, dimension => 768, r => 64, ...}

Disk Mode

For large indexes, use disk mode for persistence and lazy loading:

%% Create with disk storage
{ok, Index} = barrel_vectordb_diskann:new(#{
    dimension => 768,
    storage_mode => disk,
    base_path => <<"/path/to/index">>
}).

%% Build and persist
{ok, Index1} = barrel_vectordb_diskann:build(Options, Vectors),
ok = barrel_vectordb_diskann:sync(Index1).

%% Open existing index (O(1) startup)
{ok, Index2} = barrel_vectordb_diskann:open(<<"/path/to/index">>).

%% Close when done
ok = barrel_vectordb_diskann:close(Index2).

Hot Layer

Enable the hot layer for sub-millisecond write latency:

{ok, Index} = barrel_vectordb_diskann:new(#{
    dimension => 768,
    storage_mode => disk,
    base_path => <<"/path/to/index">>,
    hot_enabled => true,
    hot_max_size => 10000,           %% Max vectors in hot layer
    hot_compaction_threshold => 0.8  %% Compact at 80% capacity
}).

%% Inserts go to hot layer first (sub-ms latency)
{ok, Index1} = barrel_vectordb_diskann:insert(Index, <<"id">>, Vector).

%% Search combines hot layer + disk results
Results = barrel_vectordb_diskann:search(Index1, Query, 10).

%% Manual compaction (hot layer -> disk)
{ok, Index2} = barrel_vectordb_diskann:compact(Index1).

Cluster Mode

DiskANN integrates transparently into barrel_vectordb's cluster architecture. The cluster layer is backend-agnostic - it works identically with HNSW, DiskANN, or FAISS.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                     Cluster Layer                           │
│  ┌─────────────┐  ┌────────────┐  ┌─────────────────────┐  │
│  │ shard_router│→ │ scatter.erl│→ │ parallel shard calls│  │
│  └─────────────┘  └────────────┘  └─────────────────────┘  │
└──────────────────────────┬──────────────────────────────────┘
                           │ (backend-agnostic API)
┌──────────────────────────▼──────────────────────────────────┐
│                    Per-Shard DiskANN                        │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐    │
│  │ Shard 0  │  │ Shard 1  │  │ Shard 2  │  │ Shard N  │    │
│  │ DiskANN  │  │ DiskANN  │  │ DiskANN  │  │ DiskANN  │    │
│  │ on disk  │  │ on disk  │  │ on disk  │  │ on disk  │    │
│  └──────────┘  └──────────┘  └──────────┘  └──────────┘    │
└─────────────────────────────────────────────────────────────┘

Creating a DiskANN Collection

%% Create a collection with DiskANN backend
barrel_vectordb_mesh:create_collection(<<"my_collection">>, #{
    dimension => 768,
    num_shards => 4,
    replication_factor => 2,
    backend => diskann,
    diskann => #{
        storage_mode => disk,
        r => 64,
        l_build => 100,
        hot_enabled => true,
        hot_max_size => 10000
    }
}).

Each shard automatically gets its own isolated DiskANN index with:

  • Own vector files on disk (StorePath/diskann/)
  • Own graph structure (disk + ETS cache)
  • Own hot layer for fast writes
  • Own RocksDB ID mapping

Search Flow (Scatter-Gather)

  1. Query arrives at any cluster node
  2. shard_router forwards to scatter:search_vector
  3. Parallel calls to all shards (each queries its local DiskANN)
  4. Each shard returns top K×2 candidates
  5. Results gathered, deduplicated, merged by score
  6. Final top K returned to client
%% Search across all shards
{ok, Results} = barrel_vectordb_mesh:search_vector(
    <<"my_collection">>,
    QueryVector,
    #{k => 10, l_search => 200}
).

Write Flow (Consistent Hash)

  1. Document ID → consistent hash → single shard
  2. Shard's DiskANN hot layer absorbs write (sub-ms latency)
  3. Background compaction merges hot → cold (disk)
%% Write routed to single shard via consistent hash
ok = barrel_vectordb_mesh:add_vector(
    <<"my_collection">>,
    <<"doc_id">>,
    <<"text content">>,
    #{metadata => #{type => <<"article">>}},
    Vector
).

Memory Efficiency Per Shard

Component HNSW DiskANN (disk)
Vectors In RAM On SSD
Per vector ~4 KB ~530 B
1M vectors/shard ~4 GB ~530 MB

With 4 shards and DiskANN disk mode, a 4M vector collection uses ~2 GB RAM total vs ~16 GB with HNSW.

Backend Independence

The barrel_vectordb_diskann.erl module has zero cluster-specific code. The same module runs identically in standalone or clustered mode. This means you can:

  • Swap backends without changing cluster configuration
  • Test with HNSW locally, deploy with DiskANN in production
  • Mix backends across different collections in the same cluster

Parameters

Parameter Default Description
dimension required Vector dimension
r 64 Max out-degree (connections per node)
l_build 100 Search width during construction
l_search 100 Default search width for queries
alpha 1.2 Pruning factor (>1 keeps long-range edges)
distance_fn cosine Distance function: cosine or euclidean
storage_mode memory memory or disk
use_pq false Enable product quantization

File Layout (Disk Mode)

/path/to/index/
  diskann.meta     # Erlang term metadata
  diskann.graph    # Vamana graph (4KB aligned nodes)
  diskann.vectors  # Full float32 vectors (4KB aligned)
  diskann.pq       # PQ codebooks + codes
  diskann.idmap    # ID to offset mapping

Benchmarks

barrel_vectordb includes a comprehensive benchmark suite for measuring performance.

Quick Start

# Run all benchmarks
./scripts/run_benchmarks.sh

# Quick run (fewer iterations)
./scripts/run_benchmarks.sh --quick

# Full benchmark suite
./scripts/run_benchmarks.sh --full

# Export results
./scripts/run_benchmarks.sh --json
./scripts/run_benchmarks.sh --csv
./scripts/run_benchmarks.sh --all-formats

# Compare with baseline
./scripts/run_benchmarks.sh --compare baseline.json

Programmatic Usage

%% Run all benchmarks
rebar3 as bench shell
barrel_vectordb_bench:run_all().

%% Run with options
barrel_vectordb_bench:run_all(#{
    iterations => 100,
    warmup_iterations => 10,
    dimension => 128,
    output_format => json,  %% console | json | csv | all
    output_file => "bench_results"
}).

%% Run a specific benchmark
{ok, Result} = barrel_vectordb_bench:run(insert_single).
{ok, Result} = barrel_vectordb_bench:run(search_k10, #{iterations => 500}).

Available Benchmarks

Benchmark Description
insert_single Single vector insert
insert_batch_10 Batch insert 10 vectors
insert_batch_100 Batch insert 100 vectors
insert_batch_1000 Batch insert 1000 vectors
search_k1 Search for top 1 result
search_k10 Search for top 10 results
search_k50 Search for top 50 results
search_filtered Search with metadata filter
index_build_1k Build index with 1K vectors
index_build_10k Build index with 10K vectors
get_single Get single document by ID
delete_single Delete single document
concurrent_writers Concurrent write operations

Backend Comparison

Compare HNSW, FAISS, and DiskANN performance:

# Quick comparison
./scripts/run_backend_bench.sh --quick

# Default comparison
./scripts/run_backend_bench.sh

# Full benchmark suite
./scripts/run_backend_bench.sh --full

Or programmatically:

rebar3 as bench_faiss shell
barrel_vectordb_backend_bench:run_all().

Backend Overview

Backend Memory Usage Build Time Search Latency Best For
HNSW High (~4KB/vector) Fast ~0.25ms Small-medium datasets, low latency
FAISS Medium Fast Very fast GPU acceleration, batch queries
DiskANN Low (~530B/vector in disk mode) Slower ~1ms Large-scale datasets, memory-constrained

DiskANN Backend

DiskANN is ideal for billion-scale vector search where memory is limited:

  • Memory mode: Vectors in RAM, graph + PQ codes in memory
  • Disk mode: Vectors on SSD with LRU cache, ~7.5x memory reduction vs HNSW

See the DiskANN Index section above for detailed API documentation.

Benchmark Output

Results include:

  • ops_per_sec - Operations per second (throughput)
  • mean_ms - Mean latency in milliseconds
  • p50_ms - Median latency
  • p95_ms - 95th percentile latency
  • p99_ms - 99th percentile latency
  • min_ms / max_ms - Min/max latency

Example JSON output:

{
  "benchmark": "search_k10",
  "ops_per_sec": 12500,
  "mean_ms": 0.08,
  "p50_ms": 0.07,
  "p95_ms": 0.12,
  "p99_ms": 0.18,
  "iterations": 100
}

Comparing Results

# Run baseline
./scripts/run_benchmarks.sh --json
mv bench_results.json baseline.json

# Make changes, then compare
./scripts/run_benchmarks.sh --compare baseline.json

The comparison shows percentage differences for each metric, highlighting regressions and improvements.