7 posts tagged with "search"

Search functionality and implementations

View All Tags

Spice v2.2.0 (Aug 25, 2026)

August 25, 2026 · 104 min read

Viktor Yershov

Senior Software Engineer at Spice AI

Spice v2.2.0 is now available! 🚀

Spice v2.2.0 focuses on real-time data, performance, and stability. Cloud Connect links self-hosted runtimes to Spice.ai Cloud for management and observability. MySQL datasets and PostgreSQL catalogs can now stream source changes into accelerated datasets. Debezium sources can send CDC events directly without Kafka. Warm in-memory indexes improve vector and full-text search performance.

Highlights in v2.2.0 include:

Cloud Connect — link self-hosted runtimes for BYOC (Bring Your Own Cloud) to Spice.ai Cloud for management and observability
MySQL CDC — MySQL tables now stay in sync in real time using MySQL binlog replication, with no Kafka or Debezium infrastructure and no scheduled refreshes
PostgreSQL Catalog CDC — accelerate an entire PostgreSQL database in real time with a few lines of configuration and no per-table setup
Faster Search — vector and full-text search can now serve from an in-memory index by default, with no configuration change

What's New in v2.2.0

Cloud Connect

Spice Cloud Connect is a feature of Spice.ai Cloud.

Spice Cloud Connect links self-hosted runtimes for BYOC (Bring Your Own Cloud) to Spice.ai Cloud for management and observability. Run spice cloud link to enroll a standalone runtime.

Try it with the Cloud Connect on a Development Machine recipe.

MySQL Change Data Capture

The MySQL Data Connector now supports refresh_mode: changes. The runtime loads an initial snapshot of the table. It then applies inserts, updates, and deletes from the MySQL binlog as they commit. The acceleration stays current without scheduled refreshes.

datasets:
  - from: mysql:orders
    name: orders
    params:
      mysql_host: localhost
      mysql_db: mydb
      mysql_user: replicator
      mysql_pass: ${secrets:mysql_pass}
    acceleration:
      engine: cayenne
      refresh_mode: changes

Failover-safe GTID positions: When the source server has GTID enabled, the runtime tracks the replication position as a GTID set. A GTID position stays valid across a replica promotion, so replication survives a failover to a new primary without a rebuild.

PostgreSQL Catalog CDC

The PostgreSQL Catalog Connector is now Beta with new Catalog CDC acceleration support at Alpha.

A PostgreSQL catalog can now use refresh_mode: changes. One configuration replicates every table that the include patterns match. Queries then read fresh PostgreSQL data with no per-table setup.

catalogs:
  - from: pg
    name: pg
    include:
      - 'public.*'
    params:
      pg_host: localhost
      pg_db: mydb
      pg_user: postgres
      pg_pass: ${secrets:pg_pass}
    acceleration:
      refresh_mode: changes

Known limitations while catalog CDC acceleration is Alpha: configuration may change; a table dropped and recreated in the source can serve rows captured before it was recreated (#12110); and a durable catalog acceleration can come back empty after a restart (#12729). Feedback is welcome in #11850.

Debezium CDC Without Kafka

Any Debezium source plugin can now stream change events directly into Spice, with no Kafka bus in between. A dataset with from: cdc:… accepts change events at POST /v1/datasets/{name}/cdc, in JSON or Avro. The existing Kafka path (from: debezium:…) is unchanged.

Faster Search with Warm In-Memory Indexes

Search adds a warm in-memory tier for vector and full-text indexes. The runtime writes each change to the warm tier and to the durable store together. Queries read the warm tier first and fall back to the durable store. The warm tier covers vector indexes, full-text indexes, .vectors datasets, views, and chunked Elasticsearch vector columns.

More search improvements:

cosine_distance uses SIMD instructions through the simsimd library.
Full-text search applies stemming by default.
Full-text search pushes SQL filters down into the tantivy index.
A delete now removes the document from full-text and vector indexes. BM25 statistics no longer count superseded documents.
The runtime loads local rerankers from text-embeddings-inference models.
The runtime validates vector search parameters before it runs the SQL query.

Operations & Observability

Query timeout: The new runtime.query.timeout setting bounds the duration of every query. An expired query fails with HTTP 504 or gRPC DEADLINE_EXCEEDED.

runtime:
  query:
    timeout: 30s

CPU sizing for Kubernetes pods without a CPU limit: A Spice runtime pod with a CPU request and no CPU limit now sizes itself to twice the request instead of every core on the node. Set SPICE_CPU_CORES=all (or runtime.cpu.cores: all) to burst to all cores, for example when a 0.5-core request runs on a 24-core node.
Trace IDs in logs: Every log record from a query carries the query's trace ID. Filter the logs by one ID to see everything that query did.
Improved dashboards: The Grafana and Datadog dashboards add new and improved panels, and support multi-replica and Kubernetes deployments.
Helm: The Helm chart supports a custom Deployment strategy and StatefulSet updateStrategy.
Dataset status: A non-accelerated dataset now shows the Error state when its source is unavailable, instead of appearing healthy while queries fail.

Other Notable Improvements and Bug Fixes

Drasi (Alpha): The runtime forwards CDC changes and runtime tables to a Drasi source.
Microsoft SQL Server: Kerberos integrated authentication now works on Unix.
HTTP connector: OAuth2 client-credentials authentication.
Oracle DATE: Values lost their time of day. They now map to a timestamp.
Catalog exclude patterns: The Glue and Cayenne catalogs ignored exclude patterns. Both pattern lists now apply.
Spice Cayenne schema changes: Partitioned accelerations could report a source schema change as applied while partitions kept the old schema. This mismatch caused lossy casts or append failures. Cayenne now rejects in-place evolution for partitioned accelerations. The configured schema-change policy then handles the change.
Snowflake NUMBER values: The connector could remove fractional digits during schema discovery. It treated numeric metadata as absent and defaulted the scale to zero. It now preserves the source precision and scale.

See the Changelog for the full list of fixes.

SDK Updates

Updated SDKs release alongside v2.2.0:

spice.js v3.2.0 — adds query cancellation (listActiveQueries/cancelActiveQuery), mTLS client certificates, HTTP fallback when Flight is unavailable, and typed parameter binding via Flight SQL prepared statements. Also fixes nsql() and search() response handling and named parameters over HTTP.
spicepy v4.0.0 — adds streaming of large results, natural-language queries (nsql), vector and hybrid search, and query cancellation. Spice.ai Cloud users need this update: the legacy hostnames are retired in favor of region-specific endpoints.
spice-rs v3.2.0 — adds search, NSQL, async queries, query cancellation, and mTLS.

Dependency Updates

Dependency / Component	Version
DataFusion	v54.1
iceberg-rust	v0.10.0
Turso	v0.7.2
Rust toolchain	v1.96.1

New Contributors

@boittega made their first contribution in #11509
@lesbass made their first contribution in #12253

Contributors

Breaking Changes

CPU sizing for Kubernetes pods without a CPU limit: A Spice runtime pod with a CPU request and no CPU limit now sizes itself to twice the request instead of every core on the node. Set SPICE_CPU_CORES=all (or runtime.cpu.cores: all) to keep the previous behavior, or set runtime.cpu.cores to a specific core count.
Partitioned DuckDB accelerations are removed: The DuckDB accelerator now rejects partition_by. Use the Cayenne or Arrow accelerator for partitioned datasets.
Adaptive Cayenne tuning requires an explicit opt-in: An unset cayenne_tuning now resolves to auto. Set cayenne_tuning: adaptive to enable the closed-loop controller.
ONNX ML inference is removed: The runtime no longer loads ONNX models, and it no longer serves the /v1/predict endpoints. Use an LLM model provider instead.
One process-wide Vortex segment cache: Cayenne tables now share one segment cache instead of one cache per table. Set cayenne_segment_cache_mb under runtime.params to size it. When unset, the cache takes 1/64 of the memory entitlement, clamped to 256 MiB - 2 GiB.
The Pingora cache engine is now a Spice.ai Enterprise feature: An OSS build with engine: pingora degrades to the Moka engine and logs the substitution.
Deprecated settings: pg_replication_temporary_slot is deprecated. cayenne_segment_cache_mb at the table level is deprecated — set it under runtime.params instead. This release renames the MongoDB num_docs_to_infer_schema setting to schema_infer_max_records. The old name still works and warns.

Cookbook Updates

Added a new recipe for MySQL CDC (Binlog Replication): stream real-time changes from a MySQL table into Spice with refresh_mode: changes.
Added a new recipe for Aurora MySQL CDC (Binlog Replication): the same CDC setup for an AWS-hosted Amazon Aurora MySQL cluster.
Added a new recipe for PostgreSQL Catalog CDC: replicate every table of a PostgreSQL database with one catalog configuration.
Added a new recipe for Cloud Connect on a Development Machine: connect a local Spice instance to Spice.ai Cloud, deploy live changes, and deliver a secret.

The Spice Cookbook includes more than 104 recipes to help you get started with Spice quickly and easily.

Upgrading

To upgrade to v2.2.0, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.2.0 image:

docker pull spiceai/spiceai:2.2.0

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.2.0

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed

Changelog

feat(testoperator): --prepare-only / --skip-prepare for htap source reuse by @bjchambers in #11486
test(cluster): de-flake scheduler_failover (drop unreliable scheduler_node assertion) by @phillipleblanc in #11518
feat(cayenne): wire orphaned-DV cleanup knob to spicepod params + doc sync by @bjchambers in #11523
fix(cayenne): warn (not panic) on benign manifest/listing race; re-enable now-passing ignored tests by @lukekim in #11511
fix(datafusion): accurate projected scan byte size so hash joins build the smaller side by @sgrebnov in #11503
fix(cayenne): seed persisted num_rows for hash-join sizing by @sgrebnov in #11515
fix(cayenne): user-visible DELETE WHERE pk IN (...) reports the real row count by @lukekim in #11514
test(cayenne): CDC convergence/resurrection fuzz harness by @bjchambers in #11502
fix(cayenne): correctness & memory_limit fixes from perf audit (2 P0, 3 P1) by @lukekim in #11516
perf(cayenne): compaction IO-hygiene follow-ups — input-page evict + O_DIRECT writer + bench by @lukekim in #11498
Update end_game.md by @bjchambers in #11535
chore: Vendor ttf-parser, lopdf, pdf-extract by @peasee in #11521
fix: Update tpch benchmark snapshots for federated/mysql[catalog].yaml by @app/github-actions in #11531
Fix s3 vectors API by @krinart in #11536
Update from spiceai/datafusion#177 by @Jeadie in #11541
perf(cayenne): off-write_lock N>1 mem-tier checkpoint + adaptive query-admission throttle by @lukekim in #11538
fix(flightsql): propagate bearer token to per-endpoint clients by @boittega in #11509
fix: Placeholder table initialization lock swap by @peasee in #11540
chore(cluster): bump ballista pin for the null-aware anti-join fix by @phillipleblanc in #11544
fix(runtime-tools): fix memory table identifier validation rejecting valid names by @Jeadie in #11546
test(chbench): enable 4-way mem-tier PK sharding on SF1000 memory-durability configurations by @sgrebnov in #11553
fix(reorder): reorder inner-join islands inside EXISTS / NOT EXISTS by @sgrebnov in #11552
feat(cayenne): enable orphaned-DV cleanup by default (per-table threshold 20) by @bjchambers in #11533
fix(postgres): delete old key on CDC primary-key updates by @phillipleblanc in #11551
Properly exclude vendored crates from make lint-rust-fix by @krinart in #11560
fix: arrow IndexedMemTable serves stale results after DML/retention/sync (fixes #11262) by @claudespice in #11532
perf(cayenne): concurrent per-shard mem-tier checkpoint encode (drain parallelism) by @lukekim in #11558
fix(physical-plan): estimate col=literal selectivity from NDV, not the flat 20% default (Datafusion) by @sgrebnov in #11562
fix: Read Postgres DB source error when available by @peasee in #11566
feat(cayenne): cold object-store tier with read-optimized clustering by @lukekim in #11543
Generic CDC replication-lag metric by @krinart in #11554
fix: ignore transitive quick-xml advisories (RUSTSEC-2026-0194/0195) in cargo-deny by @krinart in #11571
test(chbench): add cold-object-store-tier SF-1000 HTAP run; cap SF-1000 mem-tier at 1 GiB by @lukekim in #11573
fix: surface silent Turso read-path conversion failures (fixes #11276) by @claudespice in #11570
feat(testoperator): emit P99 + Max replication lag and staleness telemetry for HTAP by @sgrebnov in #11572
feat(runtime): cost-based eager aggregation, enabled by default (datafusion spiceai-54) by @bjchambers in #11446
Bump datafusion to include spiceai/datafusion#181 by @Jeadie in #11563
fix(cayenne): drain in-flight maintenance before in-process reopen (mutation_property flake) by @bjchambers in #11578
Unified JSON nesting by @krinart in #11555
feat(testoperator): capture under-load EXPLAIN plans as HTAP artifacts by @sgrebnov in #11576
feat(cayenne): freshness-SLO-driven adaptive mem-tier shrink (+ freshness diagnostics) by @lukekim in #11574
fix(cayenne): fold un-checkpointed mem-tier keys into the cold keyset rebuild by @lukekim in #11592
perf(cayenne): reuse the PK RowConverter across the sharded CDC apply by @lukekim in #11590
Add labels to spiceai-dev by @krinart in #11591
Rename mongo num_docs_to_infer_schema (deprecated) -> schema_infer_max_records by @krinart in #11593
fix(postgres): don't ACK replication slot past uncommitted envelopes on empty transactions by @bjchambers in #11582
refactor(connectors): extract abfs, adbc, cosmosdb, ducklake, gcs, git, github, glue, kafka, snowflake-native, spiceai into data-connectors crates by @Jeadie in #11407
Revert "Generic CDC replication-lag metric" by @lukekim in #11600
refactor(cayenne): remove orphaned-DV cleanup spicepod param, make it always-on by @bjchambers in #11575
fix(mongodb): substitute delete when UpdateLookup finds no document by @bjchambers in #11583
ci(chbench): SF1000 tuned+adaptive every 3h, other configs once daily by @lukekim in #11601
feat(cayenne): add integer support to maintained AVG by @bjchambers in #11564
Refresh spidapter Spice Cloud token from client credentials by @krinart in #11594
fix: cluster filtered COUNT(*) on append-only tables returns unfiltered total (fixes #11599) by @claudespice in #11603
fix(scylladb): error on lossy CQL decimal conversion instead of silent truncation/NULL (fixes #11267) by @claudespice in #11604
ci(chbench): drop postgres-arrow and tuned (non-memory) from scheduled HTAP runs by @lukekim in #11606
ci(chbench): lower tuned SF1000 mem-tier max age to 15s for scheduled HTAP runs by @lukekim in #11618
fix(cayenne): fold the un-checkpointed mem-tier into the persisted-bloom PK-index rebuild (SF-100 over-count) by @lukekim in #11609
Potential fix for 1 code quality finding by @lukekim in #11614
fix(postgres): shared replication slot drops WAL for partitioned tables (fixes #11290) by @claudespice in #11607
bench(chbench): add 4-slot CDC variant pod for SF-1000 memory (separate from default) by @bjchambers in #11605
fix(cayenne): don't serve drifted memory-CDC row count as Exact to COUNT(*) by @bjchambers in #11602
chore(deps): vendor pgwire-replication 0.3.2 by @bjchambers in #11617
Add SQL warehouse ID to integration workflow by @Jeadie in #11561
fix(search): cosine_distance returns NULL not NaN for zero-magnitude vectors by @claudespice in #11621
feat(spidapter): tuned + adaptive MongoDB CDC spicepods with scenario selection by @bjchambers in #11579
Potential fixes for 3 code quality findings by @lukekim in #11613
Potential fixes for 2 code quality findings by @lukekim in #11611
Potential fixes for 2 code quality findings by @lukekim in #11615
Potential fix for 1 code quality finding by @lukekim in #11612
feat(cayenne): split mem-tier into ingestion + immutable pieces; advance the replication slot on a cheap periodic seal (default 2s) by @lukekim in #11622
fix(mcp): encode catalog-qualified tool names without '/' (fixes #10894) by @claudespice in #11629
fix(refresh): apply refresh_sql override on manual refresh when source unchanged (fixes #11353) by @claudespice in #11626
fix: report empty accelerated schema instead of misleading missing-column error (fixes #10920) by @claudespice in #11625
fix(imap): store message date as millisecond Timestamp, not seconds in Date64 (fixes #11547) by @claudespice in #11624
test(bucket): guard frozen partition-hash output against silent ahash drift (addresses #11277) by @claudespice in #11623
Bump datafusion (spiceai/datafusion#182) and datafusion-table-providers (#27): fix q16 CollectLeft planning error and SQLite q6 wrong revenue by @Jeadie in #11598
fix(cluster): prevent executor flaps and distributed query stalls by @phillipleblanc in #11565
fix(cli): allow spice trace on dynamic MCP/custom tool_use tasks (fixes #10995) by @claudespice in #11630
docs: add clang/lld to arm64 Linux build prerequisites (fixes #11120) by @claudespice in #11632
perf(cayenne): mem-tier visible-batch memo + maintained MIN/MAX IVM by @lukekim in #11631
docs: consolidate agent instructions and fix stale references by @lukekim in #11634
build(deps): bump the github-actions-dependencies group across 2 directories with 7 updates by @app/dependabot in #11627
perf(flight): offload query-result IPC encoding off the IO runtime by @lukekim in #11608
feat(runtime): reservation-aware Cayenne CDC query-memory default (75%→70%) by @lukekim in #11641
Revert "ci(chbench): lower tuned SF1000 mem-tier max age to 15s for scheduled HTAP runs" by @lukekim in #11648
fix(cayenne): don't bump mem-tier version on seal (QPH regression from #11622) by @lukekim in #11649
feat(cayenne): vendor versioned row_converter module (byte-compatible with arrow-row 58.3.0) by @phillipleblanc in #11654
fix(flightsql): advertise analyzed query schemas by @phillipleblanc in #11656
feat(cayenne): bounded append freshness — segmented stream publishing + publish-triggered executor stats broadcast by @phillipleblanc in #11620
fix(postgres): decouple replication keepalive/feedback from apply backpressure by @bjchambers in #11653
fix: bump DataFusion for eager decimal SUM schema by @phillipleblanc in #11657
fix(cayenne): stop mem-tier snapshot storms under memory pressure and byte-range splitting of small snapshot scans by @sgrebnov in #11645
fix(ducklake): add opt-in automatic_migration to attach older catalogs (fixes #10899) by @claudespice in #11635
ci(e2e): tolerate empty-type DuckDB variant of the parquet-rename race by @claudespice in #11636
build(deps): bump the aws-sdk group with 3 updates by @app/dependabot in #11628
ci(chbench): FIFO-queue HTAP runs via concurrency queue: max; simplify dispatcher by @bjchambers in #11659
fix(bucket): guard against ahash AES-path re-bucketing of partition data (fixes #11277) by @claudespice in #11665
fix(cayenne): gate metastore reinsert_sequence schema behind a user_version (fixes #11291) by @claudespice in #11651
fix(cayenne): durable CDC delete keeps the pk IN (...) fast path (fixes #11633) by @lukekim in #11642
ci(chbench): make scheduled HTAP runs adaptive-only (retire tuned, convert cold tier to adaptive) by @lukekim in #11677
fix(imap): decode text body into content instead of storing raw MIME (fixes #11549) by @claudespice in #11667
feat(cayenne): end-to-end integrity checksums for WAL records and Vortex data files (fixes #11639) by @claudespice in #11646
feat(spidapter): date-prefix per-run MongoDB CDC databases by @bjchambers in #11658
test(cayenne): fuzz mem-tier CDC + seq-prefix bake; harden durable-path count exactness by @bjchambers in #11643
deps: bump spiceai/datafusion to pick up ORDER BY alias unparser fix by @Jeadie in #11655
fix(postgres_replication): size CDC change-batch builders to the transaction row count by @sgrebnov in #11675
Release notes for v2.1.0 by @Jeadie in #11429
perf(pgwire-replication): incremental zero-copy framing (FrameReader) + bounded message size by @lukekim in #11668
fix(cdc): memory-mode changes stream no longer fatally stops on a transient deferred-commit queue (fixes #11644) by @bjchambers in #11678
test: update snapshots from job 84685029883 by @Jeadie in #11569
fix: Distinguish no filters vs lit(false) for RefreshSql distributed accelerations by @peasee in #11596
Fix cargo-deny crossbeam-epoch advisory by @Jeadie in #11681
fix: Update benchmark snapshots for trunk run by @app/github-actions in #11666
chore(deps): upgrade Vortex to 0.76.0 by @lukekim in #11682
refactor: extract runtime-metrics crate, move ComponentType to runtime-api-types by @Jeadie in #11524
feat(cloud-connect): standalone instance adoption by @lukekim in #11060
docs: keep the feature set constant across a session (incremental-build hygiene) by @bjchambers in #11679
feat(mysql): binlog replication for refresh_mode: changes by @lukekim in #11672
feat(cdc): cap keys per durable delete plan + absorb fall-through metrics (#11673) by @bjchambers in #11680
ci(chbench): run scheduled SF1000-adaptive HTAP bench every 6h instead of 3h by @lukekim in #11698
fix: MySQL connector fails to convert unsigned integer types (fixes #8364) by @claudespice in #11691
observability(cdc): localize CDC back-pressure across the ingest pipeline by @bjchambers in #11610
fix(cache): cache empty (zero-row) SQL result sets by @bjchambers in #11699
Update spicepod.schema.json by @Jeadie in #11706
Add support for MysSQL in Ch-Bench by @krinart in #11700
observability(chbench): limit CH-BenCHmark logs to info and above by @bjchambers in #11711
perf(cdc): cache per-dataset metric labels to avoid per-event string copies by @bjchambers in #11712
feat(postgres): pgoutput binary-format CDC decoding + zero-copy decoder by @bjchambers in #11702
Update integration_models.yml by @Jeadie in #11715
fix(cayenne): zero-row append refresh fails with "No such file or directory" by @sgrebnov in #11710
perf(cdc): cut shared-slot Postgres replication pump per-event overhead by @bjchambers in #11709
fix(spicepod): reject unknown fields on columns[] config (fixes #10972) by @claudespice in #11719
feat(cayenne): tier-gated O_DIRECT compaction writer (EBS/NAS network storage) by @lukekim in #11696
fix(cayenne): harden Vortex compaction writes by @phillipleblanc in #11692
perf(vortex): push boolean NOT filters into the Vortex scan by @lukekim in #11721
chore(chbench): cap scheduled HTAP runs at 250K tpmC (rate 9250) by @sgrebnov in #11723
MySQL CH-benCHmark HTAP: SF1 + SF1000 adaptive configs by @krinart in #11722
feat(testoperator): TPC-DS SF100 federated cluster-bench spicepod + default spicehq nodegroup by @phillipleblanc in #11716
refactor(runtime): move AccelerationSource and Acceleration into runtime-acceleration by @Jeadie in #11530
remove(ml): delete model_components crate and ONNX/Tract ML inference by @Jeadie in #11684
ci: dedupe testoperator query_overrides run steps by @Jeadie in #11690
Update search snapshots by @Jeadie in #11714
refactor: use inherent is::<T>() on dyn TableProvider/dyn ExecutionPlan for type checks by @Jeadie in #11585
chore: remove unused BlueOak license allowance by @phillipleblanc in #11762
perf(vortex): remove per-file and per-segment allocations on the scan path by @lukekim in #11733
fix(ci): use GraphQL API to check isLatest in spiced_docker workflow by @Jeadie in #11735
fix(release): use release notes heading as GitHub release title by @Jeadie in #11744
feat(chbench): headline run summary + adaptive slot-topology variants by @bjchambers in #11734
fix(ci): update e2e CLI tests for Windows-no-runtime and chat error wording by @Jeadie in #11717
build(rust): upgrade toolchain to 1.96.1 by @lukekim in #11743
fix(deps): enable reqwest "query" feature for isolated crate builds by @claudespice in #11760
build: define clippy lints in [workspace.lints], opt in every crate by @bjchambers in #11676
refactor(runtime): parameterize table-provider unwrapping by inner-fn set by @Jeadie in #11759
fix(cayenne): retain source deletion index in scan memos to close ABA hazard (fixes #11303) by @claudespice in #11739
fix(mysql): refuse binlog resume across source layout drift by @phillipleblanc in #11761
perf: avoid unnecessary allocations on search hot paths by @phillipleblanc in #11767
feat(spidapter): support cluster_name for private nodegroup targeting by @phillipleblanc in #11740
feat(cayenne): finalize datalake (cold) tier v1 UX, credentials, and e2e integration test by @sgrebnov in #11731
perf(vortex): push CAST(CASE ... END) filters into the Vortex scan by @lukekim in #11729
docs: lead README with real-time analytics-node + CDC messaging (Spice 2.0) by @lukekim in #11774
chore(chbench): extend scheduled HTAP runs to 15m by @lukekim in #11773
perf(cayenne): lazy NDV — compute on file spill, off the inline hot loop by @bjchambers in #11741
ci(rust): install 1.96.1 in workflows that pin the toolchain explicitly by @lukekim in #11769
perf(cdc): deferred parsing — move Postgres CDC decode + build off the shared pump by @bjchambers in #11732
docs: correct sort-kernel SIMD note (AVX2, not AVX-512) by @lukekim in #11771
fix(cayenne): roll cold-tier files at cayenne_cold_target_file_size_mb by @sgrebnov in #11746
docs(cayenne): move the Cayenne technical reference into the repo by @bjchambers in #11775
perf(cayenne): unify upsert PK-key hashing on prehashed XXH3-128 digests by @bjchambers in #11736
perf: avoid unnecessary allocations on query hot paths by @phillipleblanc in #11768
Extended schema inference for mysql by @krinart in #11742
chore(hash-index): add SBBF probe/insert baseline benchmark by @lukekim in #11770
feat(cayenne): incremental datalake promotion, per-file PK blooms, physical GC by @sgrebnov in #11745
test(cayenne): limit property-test concurrency by @phillipleblanc in #11784
perf(cdc): lock-free per-member ack slots + coalescing for shared Postgres replication by @bjchambers in #11779
perf(chbench): speed up MySQL and Postgres seed loading by @bjchambers in #11782
fix(ci): download MinIO client from public-data mirror by @phillipleblanc in #11790
ci: developer sign-off attestation gating the merge queue by @lukekim in #11776
fix(tests): synchronize chat expect prompts correctly by @phillipleblanc in #11802
perf(chbench): default adaptive pod to 4-slot grouping + deep prefetch by @bjchambers in #11799
refactor(cli): replace open crate with in-tree system-open by @phillipleblanc in #11791
refactor(runtime): drop direct rustls-pemfile dependency by @phillipleblanc in #11792
Cache ListingTable file statistics to avoid per-query footer re-parse by @phillipleblanc in #11793
ci: inherit sign-off across clean base merges by @phillipleblanc in #11804
perf(cayenne): hash PK once and skip OwnedRow clone on present path by @bjchambers in #11805
fix(postgres catalog): register partitioned parent, not child partitions by @bjchambers in #11798
fix(postgres catalog): quote foreign-key target identifiers by @bjchambers in #11796
refactor(cayenne): rename remaining cold_* params to cayenne_datalake_* by @sgrebnov in #11795
build: add release-profiling Cargo profile for CPU profiling by @bjchambers in #11807
fix(ci): make CH-benCH template restore robust to leftover replication slots by @sgrebnov in #11811
perf(cayenne): remove eager-NDV escape hatch, speed up per-value NDV hashing by @bjchambers in #11806
Auto-refresh the Attestation check after signoff by @bjchambers in #11815
refactor(duckdb): remove partitioned DuckDB accelerator modes by @lukekim in #11808
feat(testoperator): add option to skip the HTAP analytic gate by @bjchambers in #11810
fix: prevent executor S3 region poisoning before object-store bind by @phillipleblanc in #11766
perf(cayenne): reuse per-batch scratch allocations in KeyBasedDeletionFilterStream by @bjchambers in #11817
Update openapi.json by @app/github-actions in #11816
build(deps): bump the github-actions-dependencies group across 2 directories with 5 updates by @app/dependabot in #11819
fix(task_history): capture ExplainAnalyze metrics from the executed plan by @phillipleblanc in #11794
feat: generic spiced env-var passthrough for HTAP testoperator runs by @bjchambers in #11824
feat(cayenne): datalake tier hardening + row-capped PK-bloom-backed promotion by @sgrebnov in #11812
chore: release finalization for v2.2.0 by @Jeadie in #11738
feat(cayenne): add memory mode (mode: memory) — fully in-RAM accelerator by @lukekim in #11720
Fix secrets in .github/workflows/integration_models.yml. by @Jeadie in #11781
feat(runtime): add runtime.query.timeout parameter by @sgrebnov in #11822
feat(cloud-connect): align proto to canonical, fix enrollment ordering, add TLS e2e by @lukekim in #11829
docs(criteria): mark PostgreSQL Catalog Connector as Alpha by @bjchambers in #11785
test(cayenne): add regression tests for the mid-pass overwrite guard by @sgrebnov in #11844
perf(cayenne): light delta encoding + higher CDC coalescing defaults by @bjchambers in #11826
ci: run Cayenne doc PDF build only on trunk merges by @bjchambers in #11860
fix(cayenne): drain staged Stage-B publishes before cold promotion; add cold-tier fuzz coverage by @sgrebnov in #11847
feat(cayenne): atomic cross-partition append + delete/on-conflict atomicity by @lukekim in #11803
feat(cayenne): min/max maintained aggregates + N>1 CDC retract by @lukekim in #11862
perf(chbench): CSV-based seed loading for MySQL and Postgres by @bjchambers in #11843
fix(tests): re-enable prop_concurrent_cold_sqlite cayenne cold-tier fuzz test by @sgrebnov in #11871
fix(deps): replace yanked spin releases by @phillipleblanc in #11878
fix(cayenne): correct stale mem_checkpoint_lock comment by @bjchambers in #11872
Add run links to testoperator_dispatch.yml by @Jeadie in #11747
Cayenne serializable transactions: gated writes, per-key OCC, multi-table, FlightSQL, durable write-back by @phillipleblanc in #11870
fix(cayenne): partitioned datasets deadlock against the global encode budget and never become ready by @Jeadie in #11825
test(runtime): re-runnable turso file cleanup + bump turso to 0.7.0 by @lukekim in #11783
Use liteparse for PDF document parsing by @Jeadie in #11522
ci: add remote signoff workflow by @Jeadie in #11864
fix(cli): populate org in spice cloud apps --output json (fixes #11041) by @claudespice in #11867
fix(embeddings): restore params broken by #10853 by @Jeadie in #11788
Move cargo advisory checks to scheduled workflow by @Jeadie in #11879
perf(cayenne): bound cold-promotion Z-order sort into streaming byte-capped runs by @sgrebnov in #11890
fix(cayenne): write-back transaction atomicity — stage (not publish) and read mem-tier rows by @phillipleblanc in #11889
fix(runtime-table-partition): restore sound modulo inequality partition pruning by @Jeadie in #11891
Replace dotenvy with in-repo dotenv crate by @phillipleblanc in #11894
Remove unnecessary allocations from results-cache and hot conversion paths by @phillipleblanc in #11895
Increase ready_wait timeout for chbench sf1000 cold-tier configs by @sgrebnov in #11900
fix(mysql): retry binlog checkpoint upsert on transient accelerator write lock by @krinart in #11876
fix(cayenne): gate freshness mem-tier shrink on apply backlog by @lukekim in #11893
fix(postgres catalog): honor unsupported_type_action for catalog-discovered tables by @bjchambers in #11875
fix(postgres): make TPC-H/TPC-DS benchmark CI actually run at SF1/SF10/SF100 by @bjchambers in #11883
feat(search): writethrough compound SearchIndex/VectorIndex with optional fallback by @Jeadie in #11892
Revert "perf(cayenne): light delta encoding + higher CDC coalescing defaults" by @bjchambers in #11910
perf(vss): SIMD-accelerate cosine_distance via simsimd by @Jeadie in #11748
Add run-name to signoff.yml by @Jeadie in #11912
fix(cayenne): serialize cold-tier mem-tier checkpoint by @sgrebnov in #11907
fix(postgres catalog): discover materialized views and foreign tables by @bjchambers in #11874
test(chbench): stronger content fingerprint + MySQL sidecar template caching by @krinart in #11901
test(chbench): align money columns to canonical DECIMAL schema (CMU BenchBase) by @sgrebnov in #11921
fix(cayenne): close per-key OCC missed-conflict holes and fused-txn IVM staleness by @lukekim in #11916
ci(attestation): fast-track pure reverts past developer sign-off by @bjchambers in #11913
fix(cloud-connect): accept the portal's 5-char adoption-code segments by @phillipleblanc in #11926
docs(layering): codify crate tiers + workspace layering guard by @bjchambers in #11919
refactor(cayenne): datalake tiering UX — param rename + tracing by @sgrebnov in #11920
feat(params): typed component params via #[derive(TypedParams)] — embeddings pilot by @Jeadie in #11809
Fix SQlite round type error. by @Jeadie in #11927
fix(signoff): let remote sign-off refresh the Attestation check in CI by @bjchambers in #11929
feat(testoperator): print rows around CH-benCH analytical-gate mismatches by @bjchambers in #11923
fix(postgres catalog): don't abort the whole catalog on one schema's discovery failure by @bjchambers in #11873
ci(signoff): target-lint changed crates before full local/remote gate by @lukekim in #11909
feat(runtime)!: always-on schema inference; remove schema_inference config by @lukekim in #11880
refactor(layering): extract ClickHouse into connector-clickhouse; add restricted_deps guard by @bjchambers in #11931
perf: remove unnecessary allocations on hot query, search, and metrics paths by @phillipleblanc in #11918
Release notes for v2.1.1 by @Jeadie in #11906
refactor(layering): prepare DynamoDB for extraction from runtime by @bjchambers in #11935
Coalesce Null primary keys in RRF. by @Jeadie in #11519
ci(e2e): strip ANSI before duckdb_append graceful-shutdown log whitelist by @phillipleblanc in #11937
feat(catalog): PostgreSQL catalog-level CDC acceleration (changes mode) by @bjchambers in #11897
chbench(mysql): parallel analytical gate, decimal-comparison fix, and reseed/CI hardening by @krinart in #11946
fix: array_any_value panic when output is hash-repartitioned (empty list elements) by @bjchambers in #11952
feat(layering): checkpoint sidecar as per-engine crates + DI the dynamodb connector by @bjchambers in #11938
chore(cluster): pin Ballista to upstream 54 merge tip by @phillipleblanc in #11941
feat(vortex): upgrade fork pins to Vortex 0.79.0 by @lukekim in #11950
chbench(htap): run mysql configs at the shared 9250 rate; 2x converge wait by @sgrebnov in #11956
fix(testoperator): HTAP drain gate false-fails on stale probe observations — final snapshot decides convergence (fixes #11953) by @claudespice in #11966
feat(catalog): replica-identity-aware eligibility for PostgreSQL catalog CDC by @bjchambers in #11951
refactor(layering): extract DynamoDB into connector-dynamodb by @bjchambers in #11960
Add MemoryVectorIndex: in-memory external-store VectorIndex with brute-force exact k-NN by @Jeadie in #11908
ci(signoff): prefer lab SSH for remote sign-off; skip Rust when no .rs changes by @lukekim in #11977
Reapply #11826: light delta encoding + CDC coalescing (SF1000 convergence lever) — gated on #11943 + SF1000 fingerprint run by @lukekim in #11944
docs: align component statuses with signed release criteria by @lukekim in #11970
fix: Spidapter flight auth passthrough by @peasee in #11976
Unified CDC config by @krinart in #11777
feat(cdc): Debezium plugin push ingest to Spice without Kafka by @lukekim in #11955
refactor(connectors): unify registration on the linkme slice; restore schema coverage by @bjchambers in #11972
GTID-based MySQL CDC by @krinart in #11813
test(mysql): ignore mysql_binlog_replication_end_to_end_cayenne by @sgrebnov in #11986
fix(cayenne): support Decimal128 in maintained SUM/AVG aggregates (fixes #11933) by @claudespice in #11979
feat(params): typed params for VectorStore and FtsStore engines by @Jeadie in #11954
chore(layering): remove orphaned dead code from connector extractions by @bjchambers in #11988
feat(cayenne): default-on adaptive cold layout from observed filters (F4) by @lukekim in #11973
fix: Fairly allocate partitions from scheduler assignment cycle only by @peasee in #11853
fix(cli): avoid escaped ANSI version notification by @ewgenius in #11996
MySQL Shared binlog connection by @krinart in #11814
feat(catalog): fail-loud + metrics + by-kind reporting for PostgreSQL catalog CDC (#11850) by @bjchambers in #11983
fix(cayenne): P1 audit — subset compact, SMJ 2.5× HT, mem-tier pool account by @lukekim in #11991
Add OAuth2 client-credentials grant and configurable auth header to HTTP connector by @krinart in #11981
fix(telemetry): validate runtime.telemetry.metric_prefix against OTel name syntax by @ewgenius in #12002
fix(cayenne): purge CDC mem-tier on delete-all/TRUNCATE by @sgrebnov in #12009
Update signoff.yml. by @Jeadie in #11998
feat(dev): add scripts/signoff mine — attestation status across your open PRs by @Jeadie in #11978
refactor(layering): extract MongoDB provider into connector-mongodb by @bjchambers in #11982
perf: remove three unnecessary allocations on the query hot path by @phillipleblanc in #12029
refactor(layering): evacuate cosmosdb/graphql/github/sharepoint providers into their connector crates by @bjchambers in #12024
fix(mysql-cdc): detect source reset on GTID resume by @sgrebnov in #12023
fix(smb): exclude final SESSION_SETUP response from preauth integrity hash (fixes #11148) by @claudespice in #12003
fix(postgres): validate pg_replication_slot names before CDC refresh by @ewgenius in #12001
fix(postgres): honor pg_connection_string for CDC (closes #11994) by @ewgenius in #12000
feat: Support OTLP Histogram ingest, unix nanos time by @peasee in #11992
feat(search): write-through warm + fallback compound index for FTS (#11886) by @Jeadie in #11971
docs: PR-description and code-comment conventions by @bjchambers in #12032
refactor(postgres cdc): consolidate every dataset onto the shared pump by @bjchambers in #12028
feat(cloud-connect): Standalone instance adoption connection by @peasee in #11980
fix(cdc): don't force the durable path on zero-row readiness heartbeats (fixes #12007) by @claudespice in #12030
Upgrade DataFusion to 54.1.0 by @krinart in #11974
perf(github): parallelize serial fetches (commits + workflow logs) by @lukekim in #12017
refactor(layering): evacuate odbc/scylladb/imap/git providers into their connector crates by @bjchambers in #11993
refactor(layering): carve runtime-component out of runtime, break the ArcRuntime cycle by @bjchambers in #12031
feat(mssql): enable tiberius integrated-auth-gssapi for Kerberos integrated auth on Unix by @v1gnesh in #11386
build(deps): bump the aws-sdk group across 1 directory with 5 updates by @app/dependabot in #11821
fix(cayenne): clear a committed CDC upsert's staging WAL at finalize (#12027) by @bjchambers in #12034
feat(catalog): deterministic instance-independent replication slot + fail-loud + restart recovery (#11850) by @bjchambers in #12026
Auto-cap DuckDB accelerator memory to prevent startup over-commit by @lukekim in #11985
perf(refresh): speed up dataset refresh at startup by @lukekim in #12015
fix(cayenne): don't create stray file: directory in memory mode (fixes #11922) by @claudespice in #11940
fix(aws): accept aws_session_token for temporary credentials (fixes #10932) by @claudespice in #12041
fix(mysql): verify an adopted layout against the event's own column types (fixes #11764) by @claudespice in #12048
fix(mysql): take the binlog rotate target from the ROTATE event, not its header offset (fixes #12042) by @claudespice in #12044
fix(search): check dataset readiness before embedding the query (fixes #10956) by @claudespice in #12043
fix(databricks): pin the authentication mode with databricks_auth_mode so an auto-loaded client secret can't switch U2M to M2M (fixes #11508) by @claudespice in #12040
fix(imap): narrow scans with IMAP SEARCH instead of refetching the mailbox (fixes #11548) by @claudespice in #12039
fix(cayenne): rank inference-derived sort columns below observed filter columns by @lukekim in #12049
perf(cayenne): warm-subset compaction + mem-tier memory_limit honesty by @lukekim in #12035
fix(cayenne): purge the mem-tier on delete-all for a table without a primary key (fixes #12072) by @claudespice in #12073
fix(accelerator): let an index declare a finalize failure fatal so a stale index can't report a successful refresh (fixes #12038) by @claudespice in #12050
fix(http): restore a join's embedded projection in HTTP subquery pushdown (fixes #11009) by @claudespice in #12054
fix(e2e): report a REPL that exits mid-script instead of passing the step (fixes #12057) by @claudespice in #12059
fix(imap): fetch the raw message only when a scan reads content (fixes #12045) by @claudespice in #12060
chore: Update Turso crate to 0.7.1 by @claudespice in #12062
feat(chbench): in-memory _bench_ts watermarks for the MySQL HTAP staleness probe by @sgrebnov in #12055
test(chbench): default MySQL adaptive spicepods to one shared binlog dump by @sgrebnov in #12076
feat(search): warm in-memory writethrough/fallback index for .vectors datasets and views by @Jeadie in #11914
fix(search): expunge superseded documents from full-text BM25 statistics (fixes #12053) by @claudespice in #12056
fix(write-back): gate durable write-back on a safe source delivery primitive (fixes #11915) by @claudespice in #12051
fix(cayenne): keep protected snapshots a position-delete rewrite never folded in (fixes #11477) by @claudespice in #12052
ci(signoff): fast-track Attestation for Dependabot and non-Rust PRs, and cut the local gate by @lukekim in #12081
fix(search): reach a compound's inner full-text tier when a change stream attaches (fixes #12061) by @claudespice in #12063
fix(accelerator): report PK equality pushdown Inexact so a point lookup can't return the whole table (fixes #12070) by @claudespice in #12071
fix(github): paginate issues and pull_requests on an immutable sort key so a row touched mid-scan isn't dropped (fixes #12067) by @claudespice in #12069
fix(otel): scope reserved metric column names to the data-point shape (fixes #12064) by @claudespice in #12065
fix(cayenne): sample the subset compaction append fence before the listing it guards (fixes #12074) by @claudespice in #12075
fix(search): drop a search hit whose row is not in the base table (fixes #12089) by @claudespice in #12094
feat(datasets): mark non-accelerated datasets Error when their source is unavailable by @krinart in #12079
fix(models): restore spice.ai/spiceai as a model source by @lukekim in #12092
fix(cayenne): bound the cold PK-index rebuild; build shard views in one pass by @sgrebnov in #12078
fix: Update Search integration test snapshots by @Jeadie in #12077
feat(catalog): view warnings, docs-linked messages, and clearer startup summary for PostgreSQL catalog CDC (#11850) by @bjchambers in #12022
fix(postgres): log the cumulative delivery wait in the sink-stall warning by @sgrebnov in #12093
refactor(cloud-connect): single-source the sealed-secret wire crypto in a shared crate by @peasee in #12124
build(deps): bump quinn-proto from 0.11.14 to 0.11.16 by @app/dependabot in #12037
perf(cayenne): demand-driven scan-view cache with access-based freshness (alternative to #11948) by @bjchambers in #12005
build: exclude libnfs from the nextest workspace run by @phillipleblanc in #12131
build(deps): bump the aws-sdk group across 1 directory with 3 updates by @app/dependabot in #12068
fix(search): keep CDC changes streams working when embeddings wrap a CDC source by @Jeadie in #12086
fix(cli): report a truncated release download as a failed download (fixes #12120) by @claudespice in #12121
fix(postgres): honor inline PEM pg_sslrootcert on the replication path by @phillipleblanc in #12128
refactor(layering): extract RuntimeStatus into a runtime-status crate by @bjchambers in #12115
fix(libnfs): model AUTH as an opaque type so its layout assertion holds by @phillipleblanc in #12132
feat(duckdb): on_full_refresh: replace_file — full refresh into a new database file, atomically replaced by @lukekim in #12135
Add v2.1.2 release notes by @sgrebnov in #12146
feat(duckdb): 'on_full_refresh: checkpoint_file' to bound acceleration file growth by @sgrebnov in #12139
fix(lint): remove unused DuckDBTableWriter import in duckdb accelerator tests by @sgrebnov in #12158
fix(duckdb): drop a file-replacement test assertion that can never hold by @bjchambers in #12178
fix(cayenne): Remove redundant EmptyExec within UnionExec used by CayenneTableProvider::scan by @Jeadie in #12126
build(deps): bump async-openai for aggregated rate-limit logging by @phillipleblanc in #12149
perf(mysql): fast row-image decoder for CDC change builds by @sgrebnov in #12122
feat(catalog): pre-flight replication-slot-capacity check for PostgreSQL catalog CDC (#11850) by @bjchambers in #12114
fix(oracle): map DATE to a timestamp so the time-of-day is not silently truncated (fixes #12096) by @claudespice in #12097
fix(search): skip the warm vector index when nothing hydrates it (fixes #12101) by @claudespice in #12103
refactor(layering): extract the DataAccelerator contract into data-accelerator-api by @bjchambers in #12099
fix(cli): surface the registry error behind a failed spice add instead of blaming the archive (fixes #12116) by @claudespice in #12119
fix(ci): gate Rust checks on the lint config the gate itself reads (fixes #12111) by @claudespice in #12112
fix: Ensure executors report table statistics for distributed plan ordering by @peasee in #11854
fix: EMBED_UDF_NAME without models feature by @krinart in #12136
fix(openai): openai_responses_tools: web_search uses web_search, not web_search_preview by @Jeadie in #12142
fix(ci): cancel workflow runs left behind by superseded merge-queue batches (fixes #12170) by @claudespice in #12177
fix(runtime): return from a readiness wait when the runtime shuts down (fixes #12125) by @claudespice in #12165
perf(postgres): coalesce shared-slot CDC envelopes so a slow sink stops throttling the walsender by @bjchambers in #12147
fix(ci): make cargo_deny_advisories.yml parseable so the Rust advisory scan can run (fixes #12181) by @claudespice in #12182
fix(cluster): run GetTaskHistory under read-only SQL validation by @lukekim in #12174
fix(duckdb): count accelerated views in the coordinated memory budget (fixes #12123) by @claudespice in #12161
fix(ci): run the PR hygiene check on a GitHub-hosted runner by @claudespice in #12163
fix(cayenne): warm PK existence caches at first write; bound keyset memory during bulk load by @sgrebnov in #12133
ci: remote sign-off dispatches GitHub Actions only, never SSH to lab hosts by @lukekim in #12205
Standalone instances: spice connect enroll-only split + connect surface by @krinart in #12143
fix(search): Fix ordering to correct NDCG@k by @Jeadie in #12191
refactor(layering): carry component configuration into connectors and extract data-connector-api by @bjchambers in #12157
fix(logging): mute the candle embedding backend's GeLU notice below -vv by @phillipleblanc in #12194
test: redact connection context in federated explain snapshots by @bjchambers in #11800
Enable deletes for Index, SearchIndex, and VectorIndex by @Jeadie in #11961
feat(testoperator): trace probe latency breaches and percentiles under HTAP load by @sgrebnov in #12209
Add v2.1.1 and 2.1.2 as supported in SECURITY.md by @sgrebnov in #12190
fix(ci): seed the TPC-H Postgres benches from the fleet's dataset and schedule the catalog schema tests by @bjchambers in #12214
fix(cluster): restrict ExpandSecret to spicepod-referenced keys by @lukekim in #12156
test: redact endpoint-URL compute contexts in federated explain snapshots by @bjchambers in #12215
fix(testoperator): a fully-censored table floors worst-P99 at the discard cap by @lukekim in #12196
ci(e2e): tolerate any garbage type token in the duckdb_append parquet-rename race by @bjchambers in #12207
refactor(cloud-connect): update the proto contract for evolvability and versioning by @peasee in #12153
chore(cloud-connect): use port 443 for gateway addresses in the doc example and test fixtures by @phillipleblanc in #12252
fix(ci): authenticate the management API integration suite with secrets that exist (fixes #12184) by @claudespice in #12185
feat(search): add Recall@k, MRR@k, and Precision@k retrieval metrics by @Jeadie in #12193
fix(search): enable stemming by default for full-text search by @Jeadie in #12220
fix: Report query metrics when task history is disabled by @sgrebnov in #12227
fix(runtime): count accelerated views when gating the Cayenne compaction pool (fixes #12164) by @claudespice in #12171
fix(secrets): snapshot the store registry instead of holding its lock across awaits (fixes #12127) by @claudespice in #12166
fix(search): address Elasticsearch deletes by document _id (fixes #12267) by @claudespice in #12273
fix(postgres): deprecate pg_replication_temporary_slot, which could never work (fixes #12213) by @claudespice in #12265
fix(cayenne): always materialize a snapshot directory the catalog may reference (fixes #12208) by @claudespice in #12262
fix(duckdb): size the connection pool for accelerated views too (fixes #12160) by @claudespice in #12169
fix(ci): link brew formulas that are installed but unlinked in setup-cc by @claudespice in #12286
fix(runtime): size memory budgets from the process's own cgroup limit by @lukekim in #12263
Harden dataset availability checks against idle connection resets by @krinart in #12180
test(mysql): harden binlog CDC with dump-reconnect and full type-matrix coverage by @sgrebnov in #12264
fix: send enc_pubkey_pem on /renew for standalone runtime by @phillipleblanc in #12312
Connection-scale client modes for testoperator throughput and load tests by @lukekim in #12280
fix(search): remove all chunk documents when a chunked Elasticsearch index deletes a row (fixes #12088) by @claudespice in #12268
fix(duckdb): run spice_sys DuckDB sidecar writes on the blocking pool (fixes #12175) by @claudespice in #12204
fix(cayenne): the write-concurrency raise must respect the memory brake by @lukekim in #12317
feat(runtime): expose the memory numbers that explain an OOM as gauges by @lukekim in #12195
fix(flightsql): mark stamped statistics inexact when a LIMIT is pushed to the remote scan (fixes #12292) by @claudespice in #12293
fix(deps): bump wasmtime to 47.0.3 to clear RUSTSEC-2026-0222 (fixes #12295) by @claudespice in #12298
fix(telemetry): read the cgroup CPU quota along the whole cgroup path (fixes #12299) by @claudespice in #12300
refactor(layering): move the DataFusion helpers below runtime and derive the federation deny-list from a function registry by @bjchambers in #12210
fix(search): reject a persisted full-text index whose schema no longer matches the configuration (fixes #12274) by @claudespice in #12275
fix(spiced): keep dependency logging and chat progress alive when task history is off (fixes #12279) by @claudespice in #12281
fix(runtime): count a mid-stream response failure as a failure in the HTTP metrics (fixes #12284) by @claudespice in #12291
fix(cluster): mark a coordinator leaf scan's cached executor statistics inexact (fixes #12303) by @claudespice in #12304
fix(runtime): report a memory-pool refusal as ResourcesExhausted and answer it with 503 (fixes #12282) by @claudespice in #12289
fix(cayenne): enforce the sharded PK keyset byte budget; correct per-entry accounting by @lukekim in #12192
fix(runtime): carve the Cayenne compaction memory pool only when a dataset can compact into it (fixes #12320) by @claudespice in #12326
feat(spiced): report fatal signals before exit by @sgrebnov in #12334
bug: Filters on _match fail during planning by @Jeadie in #12247
bug: S3 metadata-filter conversion silently drops unconvertible AND/OR operands by @Jeadie in #12248
fix(ci): reject a failed sign-off on the head commit (fixes #12357) by @claudespice in #12362
ci: let signoff.yml sign off fork PRs by @Jeadie in #12025
fix(search): resolve an append stream through a vector scan (fixes #12313) by @claudespice in #12314
docs: Spice v2.1.3 release notes by @sgrebnov in #12378
bug: MemoryVectorIndex leaves external-store entries on delete. by @Jeadie in #12246
fix(cluster): keep the executor's cluster-service channel alive while idle (fixes #12301) by @claudespice in #12302
fix(https): reject OAuth2 params on structured HTTP file datasets (fixes #12315) by @claudespice in #12321
fix(scheduler): drive the interval-timing tests on a virtual clock (fixes #12323) by @claudespice in #12329
fix(ci): do not report a shutdown-cancelled sidecar task as a failure (fixes #12322) by @claudespice in #12331
fix(logging): let log colour follow the output sink (fixes #12327) by @claudespice in #12335
fix(cayenne): report the auto-tuned config once per resolution (fixes #12330) by @claudespice in #12341
chore: Update Turso crate to 0.7.2 by @claudespice in #12344
fix(runtime): stop retrying a dataset configuration failure that no retry can clear (fixes #12339) by @claudespice in #12345
build(deps): bump the github-actions-dependencies group across 3 directories with 9 updates by @app/dependabot in #12359
feat(llms): GLM 5.2 across 3+ nodes, context_length + paged_attention params, MXFP4 for DeepSeek-V4 by @lukekim in #11990
feat(runtime): size every CPU-derived pool from the CPU entitlement by @bjchambers in #12276
chore(ci): bump the spiceio setup action to v0.5.9 by @lukekim in #12391
feat(testoperator): serve /health and /v1/ready for the HTAP run by @lukekim in #12373
feat(catalog): durable storage modes for PostgreSQL catalog CDC, and let the catalog own its accelerated tables by @bjchambers in #12222
fix(runtime): correctly classify a memory-pool refusal as ResourcesExhausted (fixes #12380) by @sgrebnov in #12382
fix(ci): budget the sign-off run below the pool's job wall (fixes #12340) by @claudespice in #12343
fix(bench): install make before the seeded-database check; surface catalog error causes by @bjchambers in #12372
Clean up SQLite sidecar files in cold bloom catalog test by @sgrebnov in #12404
fix(ci): name an incoherent Cargo.lock before every cargo job fails on it (fixes #12375) by @grokspice in #12423
test(search): exercise the S3 Vectors warm-tier fallback path by @Jeadie in #12197
fix(ci): stop the spiced child E2E cleanup was leaking, scoped to this runner (fixes #12058) by @grokspice in #12431
perf(cayenne): give refresh_mode full its own write profile by @lukekim in #12338
fix(ci): skip the Attestation refresh when the sign-off SHA is no longer the PR head (fixes #12360) by @grokspice in #12433
fix(monitoring): correct p99 queries in the Datadog dashboard by @sgrebnov in #12444
fix(ci): tell an out-of-disk sign-off apart from a failing branch (fixes #12412) by @grokspice in #12426
feat(search): warm in-memory tier for chunked Elasticsearch vector columns by @Jeadie in #12082
Update spicepod.schema.json by @app/github-actions in #12451
fix(search): preserve field names with capitals and '.' by @Jeadie in #12389
fix(ci): size the cayenne property-test timeout ceiling to its measured runtime (fixes #12336) by @grokspice in #12438
fix(models): pass the pinned revision to a HuggingFace embedding model (fixes #12430) by @grokspice in #12446
fix(ci): make the out-of-disk sign-off verdict survive being out of disk (fixes #12427) by @grokspice in #12461
fix(ci): stop Enforce Pulls with Spice racing itself, and let a Dependabot PR pass it (fixes #12377) by @grokspice in #12466
docs: update v2.1.3 release notes for the Iceberg timestamptz fix (trunk sync) by @phillipleblanc in #12459
fix(ci): install protoc before building the spice CLI in the E2E CLI workflow by @phillipleblanc in #12449
fix(ci): reject an if: naming a context GitHub does not provide there (refs #12396) by @grokspice in #12468
fix(monitoring): align Datadog dashboard with OTel service instance identity by @ewgenius in #12454
chore(ci): bump the spiceio setup action to v0.5.10 by @lukekim in #12490
test(cayenne): query result correctness vs standalone engines and Spice accelerators by @lukekim in #12098
fix(elasticsearch): keep the response body out of errors on the row-data request paths (fixes #12409) by @grokspice in #12456
fix(ci): key the sign-off concurrency group on the commit, not the dispatch input (fixes #12472) by @grokspice in #12474
perf(ci): run the workspace unit-test gate in one cargo invocation by @bjchambers in #12437
feat(monitoring): Runtime Resources observability on Grafana dashboard by @ewgenius in #12441
fix(ci): report cache health on the build failures it explains (refs #12420) by @grokspice in #12471
fix: resolve fixed-offset timezones when writing Vortex files by @phillipleblanc in #12463
fix(search): stop re-parsing a dotted index key as relation.column (fixes #12462) by @grokspice in #12464
fix(ci): build the ADBC BigQuery driver with the Go version its go.mod requires by @grokspice in #12470
fix(llms): keep the Hugging Face cache token and harden the model E2E job by @grokspice in #12422
fix(ci): stop a cancelled Remote Sign-off from claiming the checks failed (fixes #12424) by @grokspice in #12425
fix(runtime): fail the write when an index cannot prepare its write window (fixes #12421) by @grokspice in #12448
fix(runtime): stop retrying a permanent configuration failure on the catalog and dataset load paths (fixes #12417) by @grokspice in #12443
chore(ci): bump spiceio setup action to v0.6.0 by @lukekim in #12522
fix(ci): say what was unready when a spiced readiness wait times out (refs #12473) by @grokspice in #12484
feat(cayenne): drive incremental vacuum from the maintenance tick by @lukekim in #12435
feat(ci): allow additional cargo features for spiceai-dev Docker builds by @ewgenius in #12198
fix(cli): authenticate the SQL REPL's nql line like the session's SQL (fixes #12491) by @grokspice in #12496
fix(search): keep the row's primary key out of Elasticsearch bulk-index failures (fixes #12370) by @claudespice in #12410
fix(cpu-budget): state a low CPU request plainly, and only below half the cores by @bjchambers in #12440
fix(models): reject a pinned Model2Vec revision instead of 401ing on it (fixes #12445) by @grokspice in #12475
fix(cli): keep a login credential on the origin it was minted for (fixes #12505) by @grokspice in #12508
bug: S3 vector indexing succeeds when the embedding column is absent by @Jeadie in #12249
bug: S3 Vectors advertises unsupported metadata predicates as exact pushdown (fixes #12243) by @Jeadie in #12250
fix(cli): treat a blank --api-key as no key at all (fixes #12498) by @grokspice in #12501
task: Add #[derive(TypedParams)] for secret stores. by @Jeadie in #12154
fix(runtime): report a deferred dataset that cannot build its connector (fixes #12414) by @grokspice in #12483
fix(mysql): pre-flight CDC replication privileges with an actionable error (fixes #11967) by @grokspice in #12485
fix(testoperator): reject a client fleet the HTTP executors cannot bound (fixes #12348) by @claudespice in #12349
fix(ci): commit the catalog schema snapshot baselines and stop the daily PR churn by @phillipleblanc in #12525
refactor(runtime): describe every table-provider wrapper layer once in a layer table by @phillipleblanc in #12200
Add testoperator dispatch support for search benchmarks and replace benchmarks_search.yml by @Jeadie in #12224
bug: RRF discards a candidate stream after an empty first batch (fixes #12239) by @Jeadie in #12384
fix(cli): stop the SQL REPL asking a runtime its queries never went to (fixes #12493) by @grokspice in #12494
test(runtime): run the metrics tests that are built but never run by @sgrebnov in #12510
fix(tests): assert the search cache status instead of wall-clock timings (fixes #12487) by @grokspice in #12488
fix(search): resolve a chunked index's entries from the authoritative store, not the read listing (fixes #12266) by @claudespice in #12411
fix(search): Improvements for search harness by @Jeadie in #12226
test(cayenne): cover the selective PK join and the inline/file tier boundary by @lukekim in #12517
fix(runtime): surface the unknown-connector suggestion, and report it once (fixes #12415) by @grokspice in #12469
fix(ci): publish no sign-off verdict when the run was signalled (fixes #12518) by @grokspice in #12544
refactor(layering): move the modules the accelerated table shares below runtime by @bjchambers in #12219
chore(ci): upgrade spiceio setup action to v0.7.0 by @lukekim in #12611
fix(runtime): keep the leading slash on an inferred local Iceberg warehouse root (fixes #12533) by @grokspice in #12540
fix(ci): let a cancelled sign-off correct only the failure it posted (fixes #12428) by @grokspice in #12432
fix(runtime): compare an append high-water mark inclusively on a day-granular time column (fixes #12492) by @grokspice in #12500
fix(runtime): report a dataset connector load failure once, not once per site (fixes #12365) by @grokspice in #12526
fix(workers): give the weighted-router test a noise margin it can survive (fixes #12537) by @grokspice in #12538
fix(runtime): reject a Hadoop table URL too short to name a namespace (fixes #12539) by @grokspice in #12542
fix(ci): probe a pinned Hugging Face revision at a URL the Hub has (fixes #12553) by @grokspice in #12554
feat(testoperator): pin the load phase to a target query rate by @lukekim in #12519
fix(cli): follow only same-origin redirects, so the API key cannot leave the origin by @grokspice in #12503
fix(ci): use spiceio and sccache for E2E macOS aarch64 builds by @lukekim in #12590
perf(cdc): build a drained CDC burst with one blocking-pool handoff by @sgrebnov in #12514
feat(cloud-connect): complete the spice connect surface — install, codeless connect, remove by @peasee in #12159
fix(scripts): distinguish merge-queue PRs in signoff mine by @Jeadie in #12383
bug: HTTP RRF truncates each candidate leg before fusion by @Jeadie in #12386
fix(cayenne): pair the cold manifest with the warm snapshot a scan captured by @phillipleblanc in #12577
fix(deps): bump the Vortex pin and guard the task-cancellation regression by @phillipleblanc in #12548
chore(deps): bump datafusion and table-providers for the DuckDB timezone and retention fixes by @phillipleblanc in #12545
fix(cayenne): surface the real error for partitioned Cayenne writes by @Jeadie in #12535
fix(postgres): re-snapshot in-memory Cayenne on slot resume, and match the slot's lifetime to the accelerator's by @bjchambers in #12221
fix(cli): bound the Spice.ai login poll instead of retrying forever (fixes #12506) by @grokspice in #12523
fix(runtime): name a UTC timestamp column's zone in a spelling DuckDB knows (fixes #12528) by @grokspice in #12534
fix(mysql): raise net_write_timeout on the shared binlog dump session (fixes #12527) by @claudespice in #12586
fix(google-genai): read a Gemini SSE stream as bytes, so a split character survives (fixes #12597) by @claudespice in #12601
fix(mssql): follow an availability group's read-only routing redirect (fixes #11453) by @claudespice in #12607
test(cayenne): measure small-file fan-out against the seeded appends, not a listed count (fixes #12602) by @claudespice in #12613
fix(ci): keep the build cache's S3 endpoint out of the runtime under test (fixes #12624) by @claudespice in #12626
perf(ci): verify the CLI binary instead of building it again by @bjchambers in #12486
fix(postgres): publish a member's held CDC envelope before a bulk transaction (fixes #12311) by @claudespice in #12408
fix(cayenne): consolidate the two partition creators, fixing the accelerator's missing directory sync (fixes #12212) by @claudespice in #12619
fix(cayenne): run a backend-parameterized test on a stack its plan fits in (fixes #12436) by @grokspice in #12561
fix(ci): let only the merge queue pass Attestation without a sign-off (fixes #12679) by @grokspice in #12681
fix(ci): take the trusted sign-off helpers from the workflow's own commit (fixes #12657) by @claudespice in #12662
fix(ci): name the E2E jobs that did not succeed (fixes #12643) by @grokspice in #12649
fix(ci): give the openai model job a ceiling above its own setup (fixes #12644) by @grokspice in #12650
fix(ci): recognise the Flight bind wording so a foreign ready cannot pass (fixes #12642) by @grokspice in #12648
fix(ci): call an unreachable compiler cache infrastructure, not a failing branch (fixes #12556) by @grokspice in #12557
fix(search): clear a replacing index instead of keeping rows the source dropped (fixes #12066) by @grokspice in #12564
fix(ci): retry the sign-off status post, and say so when it never lands (fixes #12701) by @grokspice in #12704
build(deps): bump aws-smithy-runtime from 1.12.0 to 1.12.1 in the aws-sdk group across 1 directory by @app/dependabot in #12358
task: Add #[derive(TypedParams)] for LLMs. (fixes #12150) by @Jeadie in #12155
fix(ci): let a nightly's later test suites survive an earlier one's failure (fixes #12625) by @claudespice in #12627
fix(tests): wait for a partition to persist before snapshotting its plan (fixes #12645) by @grokspice in #12652
fix(object-store): bound an FTP and SFTP connection attempt end to end (fixes #12647) by @grokspice in #12655
test(iceberg): decide the Hadoop catalog test's backends by environment (fixes #12646) by @grokspice in #12665
fix(ci): give each E2E model job its own spiced ports (fixes #12419) by @grokspice in #12685
fix(cache): evict the Pingora cache down to max_size instead of only recording it (fixes #12688) by @grokspice in #12694
fix(cayenne): bound protected-snapshot compaction by the compaction memory pool by @sgrebnov in #12541
fix(ci): leave an incomplete sign-off pending, not failed (fixes #12741) by @grokspice in #12742
fix(telemetry): resolve duration histograms below a millisecond (fixes #12693) by @grokspice in #12699
fix(ci): bound every E2E job, so a wedged build cannot hold the queue (fixes #12717) by @grokspice in #12719
feat(testoperator): add new search benchmark datasets by @Jeadie in #12237
test(cayenne): name the small-file compaction tests after the path they take (fixes #12612) by @grokspice in #12740
fix(cli): bound an inference call by silence, not by total duration (fixes #12583) by @claudespice in #12589
fix(turso): store and read back the same set of types (fixes #12628) by @claudespice in #12633
fix(catalogs): keep a Glue database whose tables the include patterns select (fixes #12630) by @claudespice in #12638
fix(catalogs): apply a catalog's exclude patterns, not just its include (fixes #12636) by @claudespice in #12641
fix(cayenne): build a carry-forward rewrite from the classified manifest rows (fixes #12708) by @claudespice in #12711
fix(runtime): subtract an append dedup window as a multiset, not a set (fixes #12499) by @grokspice in #12513
bug: FTS exec drops absent columns and violates its declared schema (fixes #12228) by @Jeadie in #12245
bug: rrf() linear recency decay divides the document age by the decay window twice (fixes #12232) by @Jeadie in #12385
chore(deps): bump datafusion rev to spiceai-54 with outer-join unparser fix by @Jeadie in #12683
fix(ci): recognise sccache's startup-timeout wording as an unusable cache (fixes #12622) by @claudespice in #12788
fix(spiced): open the log with the startup banner by @bjchambers in #12615
fix(metrics): resolve S3 Vectors latency below a hundred milliseconds (fixes #12698) by @grokspice in #12702
fix(google-genai): bound one unterminated SSE event so a stalled endpoint cannot grow spiced (fixes #12600) by @grokspice in #12690
feat(cpu-budget): size a burstable pod from its CPU request, with an all-cores opt-out by @bjchambers in #12581
fix(ci): refuse a sign-off the branch's Makefile cannot run (fixes #12813) by @claudespice in #12815
test(postgres): say which readiness wait timed out and what it saw (fixes #12730) by @grokspice in #12731
fix(mssql): bound one connection attempt, so a stalled peer cannot pin a pool slot (fixes #12606) by @grokspice in #12733
fix(ci): ask whether the disk watcher runs before make writes to it (fixes #12734) by @grokspice in #12736
fix: delete three source files that no crate root declares (fixes #12735) by @grokspice in #12738
fix(telemetry): keep a startup-recorded gauge on the operator meter, so it reaches /metrics (fixes #12667) by @grokspice in #12754
fix(cloud-connect): pin rustls where the release call presents its identity (fixes #12760) by @grokspice in #12764
test(cloud-client): assert the redirect policy by behaviour against a live server (fixes #12509) by @grokspice in #12767
fix(search): decline a warm vector tier the accelerator cannot refill (fixes #12102) by @grokspice in #12768
fix(ci): build without the compiler cache when it cannot be reached (fixes #12770) by @grokspice in #12771
fix(json): end an array element where serde did, so bare scalars read (fixes #12782) by @grokspice in #12785
fix(cache): count an invalidation as an eviction, and export the counters before one fires (fixes #12687) by @grokspice in #12791
fix(ci): ask whether the runner can hold the build before starting one (fixes #12798) by @grokspice in #12799
fix(ci): size the cayenne property-test ceiling above the pool, not at it (fixes #12811) by @grokspice in #12814
fix(ci): bound every merge-queue-required job with timeout-minutes by @grokspice in #12817
deps: bump datafusion-table-providers to 894e279 (fixes #12585) by @bjchambers in #12663
fix(ci): take stale build output off the runners, not just report a full one (fixes #12800) by @grokspice in #12801
fix(cayenne): give the Turso metastore sidecars one pool over cayenne.db (fixes #12727) by @grokspice in #12804
fix(deps): drop the unreferenced ctor dev-dependency from data_components (fixes #12664) by @grokspice in #12819
fix(ci): decline a verdict when the sign-off run itself was signalled (fixes #12710) by @grokspice in #12724
fix(postgres): hold a resuming shared slot's ack floor for tables with no member by @bjchambers in #12676
refactor(layering): split the table-provider layers out of runtime by @bjchambers in #12661
build(deps): bump the github-actions-dependencies group across 3 directories with 7 updates by @app/dependabot in #12842
fix(spiced): make the crash handler more robust by @sgrebnov in #12834
fix(catalogs): apply the Cayenne catalog's include and exclude patterns (fixes #12766) by @grokspice in #12833
fix(search): decide a compound index's write fatality per half, not by OR (fixes #12826) by @grokspice in #12828
feat(cloud): organization context for spice cloud commands by @lukekim in #12515
perf(postgres catalog): skip metadata queries for schemas no include pattern can reach by @bjchambers in #12651
bug: S3 vector metadata conversion silently drops filterable metadata (fixes #12240) by @Jeadie in #12387
build(deps): bump nvidia/cuda from 13.3.0-cudnn-runtime-ubuntu24.04 to 13.3.1-cudnn-runtime-ubuntu24.04 in the docker-dependencies group by @app/dependabot in #12356
fix(runtime): keep serving when the working directory cannot be walked (fixes #6301) by @claudespice in #12803
fix(connectors): report an object-store timeout as a timeout, not as bad credentials (fixes #12793) by @claudespice in #12797
test(runtime): say which components were not ready when the wait times out (refs #12396) by @grokspice in #12827
fix(cayenne): measure tuner memory pressure as unreclaimable demand (fixes #12531) by @sgrebnov in #12623
Include v2.1.3 and 2.1.4 in SECURITY.md by @sgrebnov in #12497
fix(search): leave the stored rows in place when a delete cannot be applied (fixes #12822) by @grokspice in #12825
fix(google-genai): resume the SSE scan where it stopped, not at the buffer start (fixes #12689) by @grokspice in #12713
fix(json): report a malformed JSON array element instead of dropping the rest of the file (fixes #12755) by @claudespice in #12777
fix(runtime): honour a declared type's precision instead of narrowing it (fixes #12756) by @claudespice in #12774
fix(cayenne): drop the partition-value guard that key encoding made unreachable (fixes #12616) by @claudespice in #12753
fix(runtime): pin the catalog-acceleration contract in both feature configurations (fixes #12743) by @claudespice in #12752
fix(cli): report an answer the model stopped early, instead of printing it as whole (fixes #12596) by @grokspice in #12715
fix(runtime): scope query jobs and active queries to the principal that submitted them by @phillipleblanc in #12841
Compute all metrics@k for all 0 < k <= n during search quality metrics. by @Jeadie in #12521
task: Add a truncate to huggingface and file embedding providers by @Jeadie in #12620
fix(cache): never serve a result whose tables changed after it read them by @bjchambers in #12703
test(cayenne): cover datalake-tier statistics, pruning, and promotion triggers by @sgrebnov in #12847
docs(ci): warn that re-running enforce-pull-with-spice replays a stale payload (fixes #12809) by @grokspice in #12810
fix(cayenne): bound the maintained-aggregate index by memory budget, and let a stale registry recover by @lukekim in #12573
fix(runtime): only initialize accelerators for the datasets a reload applies by @phillipleblanc in #12872
feat: promote BYOC and Cloud Connect changes to trunk by @phillipleblanc in #12852
fix(flight): resolve per-request settings from the live app by @phillipleblanc in #12873
fix(cli): keep polling when the token exchange has not seen the auth code yet by @phillipleblanc in #12871
feat(cayenne): inline a small whole-table refresh into the metastore by @lukekim in #12367
ci: upgrade spiceio setup action to v0.8.0 by @lukekim in #12885
fix(runtime): measure the CDC prefetch backlog, and stop its byte counter wrapping by @lukekim in #12675
fix(turso): keep a reloaded file-accelerated dataset queryable by @phillipleblanc in #12882
fix(query): preserve expression ordering across partition wrappers by @Jeadie in #12854
fix(runtime): report every start-time-only runtime.* section a reload changes by @phillipleblanc in #12874
fix(search): handle null embeddings in vector writes and JIT search by @Jeadie in #12855
test(embeddings): truncate over-length inputs in the MiniLM embed spicepods by @Jeadie in #12856
fix(search): Improve search benchmarking DX and fix some vector index names by @Jeadie in #12892
test(search): dispatch all MTEB search variants by @Jeadie in #12850
fix(runtime): stop a superseded scheduler incarnation from claiming the heartbeat key by @phillipleblanc in #12851
fix: Ensure OTel ingest materializes missing/null dimension columns by @peasee in #12087
feat(cloud-connect): apply a component-only Spicepod deployment in place by @phillipleblanc in #12895
feat(cli): typed login sessions and explicit connect organization selection by @phillipleblanc in #12912
refactor: make each table wrapper a typed layer that answers where a walk goes by @bjchambers in #12891
feat(runtime): put every query's trace id on its log records by @lukekim in #12610
feat(cayenne): bound the SUM of the per-table PK keyset caches by @lukekim in #12802
fix(cache): read and expire a Pingora entry under one hold of its shard (fixes #12832) by @grokspice in #12839
fix: Update Search integration test snapshots by @Jeadie in #12707
fix(runtime): invalidate localpod children's cached results on parent refresh by @krinart in #12897
fix(cayenne): use valid snapshot IDs for deferred appends by @Jeadie in #12853
fix(ci): New branch for each push-snap-changes (only affects integration_search.yml) by @Jeadie in #12684
fix(search): make chunked embedding and offset columns nullable (#12778) by @Jeadie in #12783
feat(cayenne): charge scan materialization to the query memory pool by @lukekim in #12759
fix(cache): run the Pingora invalidation scan off the calling runtime worker (fixes #12806) by @grokspice in #12808
fix(cache): count the removals the Pingora engine performs itself (fixes #12792) by @grokspice in #12830
fix(cayenne): reserve append sequence for partitioned upsert appends (#12779) by @Jeadie in #12784
test(cayenne): prove the keyset degrade releases each shard as it converts by @lukekim in #12790
perf(postgres catalog): resolve a schema's table schemas in one query (#12106) by @bjchambers in #12886
fix(vortex): expose segment cache metrics on scrape by @phillipleblanc in #12944
refactor: point connector imports at the crates that actually own them by @bjchambers in #12925
fix(postgres): rebuild the acceleration when a replication slot's history is gone by @bjchambers in #12922
fix(cayenne)!: only an explicit cayenne_tuning: adaptive enables the closed loop by @lukekim in #12949
fix(dev-tools): show merge-queue and conflict state as labels in scripts/signoff mine by @Jeadie in #12924
test: cover warm-index delete paths across backends (#11964) by @Jeadie in #12635
fix(cayenne): fractional proptest scaling + batch timestamp-partition test inserts by @Jeadie in #12796
Helpful tool to track what users/bots have got done lately by @Jeadie in #12603
Release notes housekeeping by @krinart in #12977
build(deps): bump serde_with from 3.20.0 to 3.21.0 by @app/dependabot in #11969
refactor: make runtime's re-export shims crate-visible by @bjchambers in #12969
fix(runtime): surface OTel metric tables with duplicate columns, and count unsupported metric types as rejected by @peasee in #12923
fix: Update Search integration test snapshots by @app/github-actions in #12979
Make signoff.yml more searchable. by @Jeadie in #12392
bug: Support search UDTFs in SQL function components by @Jeadie in #12919
docs: document the stacked-PR workflow, with a tested restack helper by @bjchambers in #12951
fix(llms): fall back to pytorch_model.bin for embeddings without safetensors by @Jeadie in #12971
feat(monitoring): improve the Datadog dashboard for multi-replica and Kubernetes deployments by @sgrebnov in #12972
Charge the query memory pool for concurrent Vortex split decodes by @lukekim in #12940
docs(release-notes): write release notes in Simplified Technical English by @lukekim in #12947
refactor: narrow the DataConnector trait off the runtime's types by @bjchambers in #12993
fix(cayenne): compact a DDL-created partition on an interval, not only on write (fixes #12617) by @grokspice in #12757
fix(cayenne): hand out a pooled Turso connection in autocommit (refs #12820) by @grokspice in #12821
fix: fail the lint gate on a source file no crate root declares (fixes #12737) by @grokspice in #12744
fix(search): stage a full refresh of the memory vector tier, so dropped rows leave it (refs #12413) by @grokspice in #12823
fix(graphql): keep the configured schema when a page has no rows (fixes #13004) by @claudespice in #13016
fix(runtime): register task_history before embeddings/rerankers load by @Jeadie in #12978
test(postgres): converge the catalog on source DDL, and warn when it selects nothing by @bjchambers in #12988
fix(cayenne): budget for catalogs in the Cayenne memory classification by @bjchambers in #13027
fix(cayenne): hold primary keys committed while the existence index is checked out by @lukekim in #13019
feat(search): push SQL filters down into tantivy full-text search by @Jeadie in #12812
fix(cayenne): serve the maintained count Inexact while its delta is queued (fixes #12824) by @grokspice in #12829
fix: Update Search integration test snapshots by @app/github-actions in #13060
fix(testoperator benchmarks): benchmark fixes by @krinart in #13012
fix(runtime): stop query_active_count drifting up when queries share a request (fixes #12883) by @claudespice in #13034
chore(deps): bump model2vec-rs for fast WordPiece tokenizer by @Jeadie in #12986
ci(cayenne-doc): render diagrams locally with mermaid-cli, and fix the diagram that has been failing the build for a month by @bjchambers in #13063
feat(monitoring): improve results cache panels on the Datadog dashboard by @sgrebnov in #13066
feat(cayenne)!: one process-wide Vortex segment cache instead of a cache per table by @bjchambers in #12983
fix(connectors): keep the projected schema when a query returns no rows (fixes #13015) by @claudespice in #13078
fix(deps): bump the datafusion pin past the merged unparser fixes (fixes #12406) by @claudespice in #13083
fix(cache): compact a sliced result before caching it (fixes #12921) by @claudespice in #13043
refactor(cdc): give the sidecar checkpoint stores a below-runtime interface by @bjchambers in #13045
fix(cayenne): re-baseline the maintained row count when a promotion folds a delete tombstone (fixes #12846) by @claudespice in #12998
fix(acceleration): stop an engine-required type rewrite reading as a stale acceleration schema (fixes #13014) by @claudespice in #13074
fix(udfs): return NULL for a vector distance that is not defined (refs #11263) by @claudespice in #13091
fix(cayenne): refuse a data directory that contains the metastore (fixes #13055) by @claudespice in #13101
fix(json): report content after a JSON array instead of dropping it (fixes #12786) by @claudespice in #13097
fix(runtime): bound the spicepod apply so an unloadable dataset cannot wedge it (fixes #12862) by @claudespice in #13095
fix(catalogs): apply the Glue catalog's exclude patterns (fixes #12634) by @claudespice in #13103
fix(metrics): make component metrics and tests more robust by @sgrebnov in #13094
feat(cache)!: ship the Pingora cache engine as an Enterprise-only feature by @lukekim in #13000
fix: Update benchmark snapshots by @app/github-actions in #13056
fix(postgres): compare CDC positions against what the slot can stream, not what it retains by @bjchambers in #12990
docs(cayenne): compress the changelog, and give it a rule for what earns a row by @bjchambers in #13067
test: remove a test's container when it is done with it, including on failure by @bjchambers in #13123
fix(flight): record one metric sample per Flight RPC (fixes #12844) by @claudespice in #13112
feat(drasi): forward CDC changes and runtime tables to a Drasi source (Alpha) by @lukekim in #12653
fix(bench): time the engine instead of the results cache, and validate ClickBench by @lukekim in #13150
refactor(connector): retype the DataConnector surface to DatasetSpec by @bjchambers in #13096
bug: SQL-tier function body does not support named arguments (=>) for search UDTFs (fixes #12898) by @Jeadie in #12920
fix(cache): compact the sliced and wide results that compaction previously skipped by @sgrebnov in #13176
docs(stacked-prs): correct when a restack forfeits the sign-off by @bjchambers in #13130
fix(postgres): load a provably empty CDC acceleration through its snapshot bootstrap by @bjchambers in #13175
fix(cayenne): keep Arrow timestamp units (Vortex supports ns) by @lukekim in #13180
Cloud Connect: enrollment and identity foundation by @phillipleblanc in #13181
refactor(model): give the model-provider contracts a home below llms by @bjchambers in #13201
Fix runtime lint with zero features by @krinart in #13208
fix(benchmarks): BigQuery + Snowflake fixes + updated snapshots by @krinart in #13136
Revert "fix(search): decline a warm vector tier the accelerator cannot refill (fixes #12102)" (#12768) by @Jeadie in #13041
fix(snapshots): don't bootstrap the snapshot file_create just discarded by @lukekim in #13179
Declare the CPU entitlement to Vortex at startup by @sgrebnov in #13173
fix(mysql): fix net_write_timeout session syntax and extend transient replication errors by @ewgenius in #13178
fix(postgres): classify EOF and server termination errors as transient in CDC replication by @ewgenius in #13163
feat(testoperator): support customer-supplied datasets in run search by @Jeadie in #12982
Add PDF page splitter and FinanceBench staging job (#12858) by @Jeadie in #12984
fix(cayenne): frame the persisted PK bloom with the probe function that filled it (fixes #13137) by @claudespice in #13153
refactor(layering): move the connectors below the runtime by @bjchambers in #13166
feat(cli): manage the Cloud Connect service with launchd on macOS by @phillipleblanc in #13203
Support local rerankers from text-embeddings-inference by @Jeadie in #12929
fix(catalog): make the PostgreSQL catalog's errors and warnings actionable by @bjchambers in #13199
fix(cloud-connect): look again when a lock create reports the entry missing by @phillipleblanc in #13231
Skip DynamoDB in scheduled benchmarks by @ewgenius in #13222
Populate the file metadata cache with Vortex footers at write time by @sgrebnov in #13228
perf(cayenne): give the PK filter a cache-line layout, and the bit count it asks for by @bjchambers in #13219
fix(mysql): rebuild a purged-position acceleration atomically instead of emptying it (fixes #12967) by @claudespice in #13023
fix(cloud-connect): heartbeat restart-required, attachment by app id, and this crate's integration tests in the gate by @phillipleblanc in #13234
fix(search): join all of a dataset's vector indexes on one VectorScanTableProvider by @krinart in #13209
Validate vector search parameters before SQL query by @lesbass in #12253
feat(helm): support customizing Deployment strategy and StatefulSet updateStrategy by @sgrebnov in #13249
fix(postgres): recover from a replication slot lost while streaming, and state what each slot costs by @bjchambers in #13221
fix(cayenne): stop a partitioned acceleration reporting a schema change it never applied (fixes #12999) by @claudespice in #13057
feat(cloud-connect): name the local spicepod a cloud-managed instance serves or ignores by @phillipleblanc in #13262
fix(snowflake): keep NUMBER precision and scale during schema discovery by @phillipleblanc in #13272
fix(secrets): check secret references where the components resolve them by @peasee in #13265
docs(criteria): sign off the PostgreSQL Catalog Connector at Beta by @bjchambers in #13235
fix(cloud-connect): resolve locked service state through the retained directory descriptor (fixes #13204) by @claudespice in #13292
chore(deps): upgrade iceberg-rust to v0.10.0 by @krinart in #13189
fix(cloud-connect): fix Windows build of identity state-file helpers by @sgrebnov in #13308
fix(cayenne): normalize append de-duplication types by @ewgenius in #13200
feat(cli): unify Spice Cloud enrollment and service workflows by @phillipleblanc in #13326
feat(postgres): native upsert delivery for durable write-back by @Jeadie in #13323
feat(cli): create an unattached Cloud Connect project by @phillipleblanc in #13333
fix(cayenne): exclude protected snapshots above the delete fence from subset compaction by @krinart in #13343
fix(cli): resolve a Cloud Connect project's data-plane region from its config by @krinart in #13370
fix(cli): keep a granted Spice Cloud login when the identity endpoint is silent by @bjchambers in #13376
fix(duckdb): fix concurrent-query failures after a replace_file swap with multiple DuckDB files by @Jeadie in #13383

*Full Changelog: https://github.com/spiceai/spiceai/compare/v2.1.0...v2.2.0

Spice v2.1.0 (Jul 9, 2026)

July 9, 2026 · 36 min read

Jack Eadie

Token Plumber at Spice AI

Spice v2.1.0 is now available! 🎉

Spice v2.1.0 is the next minor release of Spice, headlined by high-throughput Cayenne CDC, scaling and resilience improvements to PostgreSQL logical replication, expanded distributed query with Iceberg catalog scans and broadcast joins, and the upgrade to DataFusion v54 (including v53), Arrow v58.3, and Vortex v0.74. The release also adds experimental adaptive self-tuning for the Cayenne accelerator, distributed GLM inference, and a range of security, search, and connector improvements.

Highlights in v2.1.0 include:

High-Throughput Cayenne CDC — in-memory CDC tier, dedicated compaction runtime, and write-path optimizations that cut replication lag on high-volume CDC workloads
PostgreSQL Replication at Scale — multiple changes-mode datasets share a single replication slot, unchanged-TOAST recovery, and resilient reconnects across rolling deploys
Distributed Query — distributed Ballista scans of Iceberg catalog tables, broadcast joins for small dimension tables, and shared scheduler job state with failover
DataFusion v54 — upgrade to DataFusion v54 (folding in v53), Arrow v58.3, and Vortex v0.74, bringing faster joins, scans, and planning
Adaptive Self-Tuning (Experimental) — opt-in closed-loop tuning and maintained aggregates that adapt Cayenne to hardware, schema, and live workload

What's New in v2.1.0

High-Throughput Cayenne CDC

A major focus of v2.1 is Spice Cayenne write-path throughput for change-data-capture (HTAP) workloads:

In-Memory CDC Tier: A new in-memory CDC tier and follow-ups cut replication lag on hot upsert tables, with bounded mem-tier checkpointing and O(1) per-scan deletion views, plus a two-phase off-fence checkpoint on the ingest path.
Dedicated Compaction Runtime: A dedicated compaction runtime with CDC pipelining and protected snapshots isolates compaction from query and ingest paths, with parallelized deletion-vector writes, per-batch directory-barrier coalescing, and size-aware parallel encode for protected-snapshot compaction.
Incremental Protected Snapshot Compaction: Incremental compaction of protected snapshots (used in Cayenne's merge-on-read deletion index) reduces disk usage and improves query performance.
Smaller WAL & Metadata-Only Publish: cayenne_insert_record table IDs are stored as 16-byte raw-UUID BLOBs, cutting CDC WAL volume ~34%; upsert commits publish metadata-only, dropping per-key insert records; transient staged CDC deltas are light-encoded.
Delta-Write Encoding Levels: A new cayenne_delta_encoding setting (default auto) selects delta-write encoding, and cayenne_compression_strategy: zstd is now fully wired.
In-Memory CDC Sharding: PK-hash intra-apply sharding parallelizes in-memory CDC apply.
Scan Safety Under Write: In-flight scans are ref-counted so snapshot GC can't delete Vortex files mid-read; in-RAM scan parallelism, query admission control, and sound scan output ordering improve read behavior under sustained CDC.

Delta-write encoding effort and Vortex compression are tunable per accelerator. cayenne_delta_encoding: auto (the default) size-gates fresh CDC/append writes — small deltas use a light scheme and are re-encoded during compaction — or pin an explicit level 0..10 (7 is the full default cascade); cayenne_compression_strategy selects the Vortex compression:

acceleration:
  engine: cayenne
  refresh_mode: changes
  params:
    cayenne_delta_encoding: auto # 'auto' (default), or pin a level 0..10 (7 = full cascade)
    cayenne_compression_strategy: zstd # 'btrblocks' (default) or 'zstd'

Change Data Capture & HTAP

PostgreSQL logical replication (CDC, refresh_mode: changes, introduced in v2.0) gets significant scaling and resilience work in v2.1:

Shared Replication Slot: Multiple refresh_mode: changes PostgreSQL datasets on the same connection can name the same pg_replication_slot to share a single replication slot, walsender decoder, and publication, with decoded changes multiplexed by (schema, table) to each dataset. This collapses the slot count from one-per-dataset to one — staying well under Postgres's default max_replication_slots = 10.

datasets:
  - from: postgres:public.orders
    name: orders
    params:
      pg_db: mydb
      pg_replication_slot: spice_cdc # shared slot name
    acceleration:
      refresh_mode: changes
  - from: postgres:public.customers
    name: customers
    params:
      pg_db: mydb
      pg_replication_slot: spice_cdc # same name -> one slot, walsender & publication
    acceleration:
      refresh_mode: changes

Unchanged-TOAST Recovery: Under REPLICA IDENTITY FULL, when an UPDATE leaves a large TOASTed column unchanged, pgoutput sends an "unchanged" marker; Spice now fills that value from the old tuple — its old value is its current value — so updates no longer error or drop columns. Without an old tuple, the error persists with a hint to enable REPLICA IDENTITY FULL.
Transient Walsender Contention: Slot-contention errors during rolling deploys — SQLSTATE 55006 ("replication slot is active for PID") and 53300 ("requested standby connections exceeds max_wal_senders") — are now classified as transient and retried with backoff instead of fatally terminating the stream. Replication connections are also released at shutdown start (not process exit), freeing walsender seats for replacement instances.
Strict CDC Param Validation: PostgreSQL CDC parameters are strictly validated rather than silently defaulted.
Debezium Schema Evolution: Fixes for Debezium schema-evolution support, including tombstone-message handling and sign-extension of minimal-width base64 decimals.

Distributed Query

Distributed Query gains:

Distributed Iceberg Catalog Scans: Ballista distributes scans of Iceberg catalog tables across executors.
Broadcast Joins: Small dimension tables are broadcast to executors for distributed joins.
Shared Scheduler Job State with Failover: Ballista job state is shared so the scheduler can fail over without losing in-flight work.

Performance & Query Engine

Apache DataFusion is upgraded to v54, folding in v53, alongside Arrow v58.3 and Vortex v0.74 (with a pin bump adding intra-file decode split and a per-execution kernel cache). Two DataFusion releases land in this upgrade:

DataFusion v54 (release notes): adds LATERAL joins, SQL lambda functions (x -> expr with array_transform/array_filter/array_any_match), spilling nested-loop joins, and a faster arrow-avro reader. Performance work includes morsel-driven Parquet scans (up to ~2x faster for skewed scans), 20-50x faster sort-merge semi/anti/mark joins, redundant-sort-key pruning, NDV-based cardinality estimation, and inner_product/cosine_distance functions.
DataFusion v53 (release notes): adds LIMIT-aware Parquet row-group pruning, broader filter pushdown through joins and UNION, nested-field pushdown (get_field into the scan), faster query planning (some plans dropping from ~4-5ms to ~100us), and 42 faster built-in functions.

Federation deny-list enforcement and catalog DDL are restored after the DataFusion upgrades, and a cost-based left-deep join reordering rule is added for Cayenne acceleration.

AI & LLM

Native GLM Support with Distributed Inference: Native GLM model support with surfaced reasoning_content, including tensor-parallel GLM inference. Load a GLM model with model_type: glm4 (glm4moe and glm4moelite are also supported):

models:
  - name: glm
    from: huggingface:huggingface.co/THUDM/glm-4-9b-chat
    params:
      model_type: glm4

For large models, GLM inference can be distributed across nodes (tensor parallelism) via the mistral.rs pure-TCP ring all-reduce backend — no NCCL/system dependency. This is a Spice.ai Enterprise feature requiring the distributed build. Run the same model on each node, changing only node_rank:

models:
  - name: glm
    from: huggingface:huggingface.co/THUDM/glm-4-9b-chat
    params:
      model_type: glm4
      distributed_backend: ring
      nodes: 10.0.4.21,10.0.4.22 # ordered host/IP per rank; the ring backend currently requires exactly 2
      node_rank: 0 # rank of THIS node in [0, world_size); rank 0 serves the API. Set node_rank: 1 on 10.0.4.22

NSQL Context Endpoint: A new GET /v1/nsql/context endpoint returns the SQL dialect, dataset schemas (with optional sample rows), and registered functions that Spice injects into natural-language-to-SQL (POST /v1/nsql) requests — useful for inspecting or caching exactly what the model sees:

# Inspect the context injected into /v1/nsql requests (examples_limit default 3, max 100)
curl "http://localhost:8090/v1/nsql/context?include_examples=true&examples_limit=3"

Returns the dialect, per-dataset schema (keys, indexes, searchable columns), the registered function inventory, and sample rows (abbreviated):

{
  "context": "# Spice.ai NSQL Context",
  "instructions": [
    "Write SQL for the Spice runtime, which uses Apache DataFusion with the SQL parser configured for the PostgreSQL dialect.",
    "Use table and column descriptions, primary keys, foreign keys, unique constraints, and indexes when choosing joins and filters."
  ],
  "sql": {
    "engine": "Apache DataFusion",
    "version": "54.0.0",
    "dialect": "PostgreSQL",
    "parser": "DataFusion SQL parser configured with PostgreSQL dialect"
  },
  "datasets": [
    {
      "name": "sales.orders",
      "table": "orders",
      "description": "Customer orders",
      "columns": [
        { "name": "order_id", "data_type": "Int64", "nullable": false, "primary_key": true, "indexed": true },
        { "name": "customer_id", "data_type": "Utf8", "nullable": false, "vector_search": true, "full_text_search": true }
      ],
      "primary_key": ["order_id"],
      "foreign_keys": [
        { "columns": ["customer_id"], "foreign_table": "spice.sales.customers", "foreign_columns": ["id"] }
      ]
    }
  ],
  "functions": {
    "summary": "Spice SQL runs on Apache DataFusion ... Run SELECT * FROM list_udfs() to inspect the full registered function inventory",
    "search": [
      { "name": "vector_search", "syntax": "vector_search(dataset, 'query text'[, column])" },
      { "name": "text_search", "syntax": "text_search(dataset, 'query text'[, column])" }
    ]
  },
  "samples": [
    { "title": "Example rows for `sales.orders`", "content": "| order_id | customer_id |\n| --- | --- |\n| 42 | CUST-1 |" }
  ]
}

Search & Vectors

S3 Vectors Pagination: QueryVectors paginates for top-K up to 10,000.
Elasticsearch kNN Candidate Pool: The default kNN candidate pool is raised from 10 to 1000 for better recall.

SQL & Query Engine

FlightSQL Substrait Plans: CommandStatementSubstraitPlan support.
Large Result Streaming: Flight streaming is optimized for large result sets.
Write Authorization: The SQL tool allows writes for ReadWrite API keys.
Schema Evolution Policies: on_schema_change supports widening-only evolution and a drop_and_recreate policy.

Security & Connectors

Kafka mTLS: Mutual TLS configuration is surfaced in the Kafka data connector.
Secret Resolution at Startup: Secret references are checked and reported at startup.
DuckDB HNSW: Upgrade to DuckDB v1.5.3 with the statically linked VSS (HNSW) vector extension.

Adaptive Self-Tuning (Experimental)

The Spice Cayenne accelerator gains experimental opt-in self-tuning. cayenne_tuning: auto derives configuration from the detected hardware and inferred schema, while adaptive additionally runs a per-table closed-feedback controller that adapts flush caps, the in-memory CDC tier, compaction cadence, and write concurrency toward operator SLOs (replication lag, freshness, query latency, queries-per-hour). Cayenne can also maintain aggregates incrementally — with predicate-aware delta serving and incremental retraction — and fold whole-table SUM/AVG/COUNT/MIN/MAX from statistics. These features are experimental and disabled by default.

datasets:
  - from: postgres:public.orders
    name: orders
    acceleration:
      engine: cayenne
      refresh_mode: changes
      params:
        cayenne_tuning: adaptive # 'auto' (static, env- + schema-derived) or 'adaptive' (closed-loop)

Observability

Per-Dataset Query Attribution: The query_executions metric gains a datasets dimension.
HTAP Diagnostics: Improved HTAP replication diagnostics on non-convergence.
Cayenne Write Observability: Write-phase observability for the in-memory CDC tier.

Notable Bug Fixes

Cayenne Utf8View: The Utf8View read schema avoids a hash-join offset overflow.
Cayenne metastore: cayenne_metastore: turso is honored for partitioned tables and the dataset checkpoint.
Dual-write detection: Dual-write accelerated tables are detected behind the metadata-enrichment wrapper.
digest_many collisions: Values are length-prefixed so column boundaries can't collide.
Turso WAL checkpoint: WAL checkpoints route through the native Turso connection.
TLS status probe: The status check probes the metrics endpoint over HTTPS when TLS is enabled.
Search snippet offsets: Character chunk offsets persist so search snippets aren't shifted or garbled.
Async query chunk offsets: /v1/queries chunk row_offset uses the cumulative offset rather than chunk_index * chunk_size.

Dependency Updates

Dependency / Component	Version
DataFusion	v54
Arrow (arrow-rs)	v58.3
Vortex	v0.74
iceberg-rust	v0.9.1
DuckDB	v1.5.3
Rust toolchain	v1.95.0

Contributors

Breaking Changes

No breaking changes.

Cookbook Updates

No new cookbook recipes.

The Spice Cookbook includes more than 100 recipes to help you get started with Spice quickly and easily.

Upgrading

To upgrade to v2.1.0, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.1.0 image:

docker pull spiceai/spiceai:2.1.0

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.1.0

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed

Changelog

Make DuckDB schema cast logic more robust by @sgrebnov in #10991
perf(cayenne): reduce allocation overheads in hot paths by @lukekim in #10950
improve error message for 'params.spiceai_region' by @Jeadie in #10954
fix(acceleration): rename WriteMode variants to fix #10960 by @phillipleblanc in #10974
add Scylla, bigquery and turso to throughput benchmarking by @Jeadie in #11006
deps(ballista): pull in shuffle-on-object-store correctness fixes (datafusion-ballista PRs #42 + #43) by @phillipleblanc in #10919
fix(kafka): seek to sidecar offsets via post_rebalance callback on restart by @ewgenius in #11007
Propagate source comments into schema metadata by @lukekim in #10944
feat(chbench-driver): better alignment to BenchBase by @sgrebnov in #11012
fix: handle EXISTS/NOT EXISTS subqueries in federation analyzer by @sgrebnov in #10996
Enable filter pushdown in spicepod defined UDTFs by @Jeadie in #11004
Refactor spice dataset configuration command by @Jeadie in #10999
fix: ensure HashJoinExec partition counts match when join input is statically empty by @phillipleblanc in #11025
feat(chbench): Improve OLTP throughput and reduce PostgreSQL CDC overhead by @sgrebnov in #11018
feat(cluster): add distributed query observability metrics by @phillipleblanc in #10990
fix: delegate truncate in PolyTableProvider to inner write provider by @claudespice in #11036
Remove default runtime features - enable explicitly in spiced by @phillipleblanc in #11037
fix: preserve field and schema metadata in Vortex physical schema calculation by @claudespice in #11013
Expose metadata descriptions via PostgreSQL UDFs by @lukekim in #11032
fix: route Turso WAL checkpoint through native turso connection (fixes #10657) by @claudespice in #11048
fix: add missing truncate delegation and guards for wrapper TableProviders by @claudespice in #11014
Update DataConnector statuses by @krinart in #11052
remove redundant readonly checks by @Jeadie in #10975
Fix Unity Catalog connector compatibility with OSS Unity Catalog by @ewgenius in #11026
refactor(cdc): reduce CDC sub-batch splits for interleaved upsert/delete workloads by @sgrebnov in #11051
feat(cayenne): allow inline writes with pending deletions (deletes/upserts) by @sgrebnov in #11031
fix(sql-tool): defer read-only gate to caller's API key role by @phillipleblanc in #11040
fix: map Gemini Recitation finish reason to ContentFilter by @claudespice in #11046
feat(cayenne): fast-path CDC deletes by extracting PK values from filters by @sgrebnov in #11049
fix(cluster): gate scheduler readiness on executor partition loads by @phillipleblanc in #10992
fix(snowflake): enforce function deny-list in federation pushdown by @claudespice in #11057
perf(cdc): Last-write-wins dedup in group_into_sub_batches to reduce sub-batch splits by @sgrebnov in #11059
[Bug] Timing between reconnect and AllocateInitialPartitions leaves connection without flight_sql_client by @Jeadie in #10805
Define 'trait QueryEngine' to refactor runtime crate by @Jeadie in #11028
fix(snowflake): apply Spice function deny-list in extracted connector crate by @claudespice in #11071
perf(cayenne): keep CDC upsert PK keysets resident to avoid per-batch full-table rebuilds by @lukekim in #11074
fix(postgres-replication): emit recovery log + reduce reconnect-warn volume by @claudespice in #11084
fix metadata on search indexing by @Jeadie in #11080
perf(cayenne): scale CDC inline flush caps with memory + storage class by @lukekim in #11087
feat(cayenne): merge-on-read position deletes for PK upsert tables + memory-pool accounting by @lukekim in #11085
Support tuple-IN composite PK extraction in Cayenne delete fast-path by @sgrebnov in #11093
feat(cluster): report per-executor table statistics so distributed JoinSelection can size joins by @phillipleblanc in #11089
feat(cluster): NDV-aware executor stats so CDC q18 join swap fires by @phillipleblanc in #11098
Improve HTAP replication diagnostics on non-convergence by @sgrebnov in #11100
Normalize DataType::Null to Int32 in acceleration schema for duckdb by @krinart in #11062
Fix debezium schema evolution support by @ewgenius in #11095
feat(cayenne): incremental write-path executor statistics for distributed join sizing by @phillipleblanc in #11104
fix(cache): periodic moka maintenance to drain invalidation predicates (#11077) by @phillipleblanc in #11106
fix: validate Snowflake account identifiers and auth config by @Jeadie in #11024
fix: trace external mcp server tool calls by @ewgenius in #11058
Remove unwrap from test code; drop clippy::unwrap_used suppressions by @phillipleblanc in #11108
Upgrade to DuckDB 1.5.3 + statically link the VSS (HNSW) extension by @sgrebnov in #11107
Fix fetched_at for HTTP connector by @Jeadie in #11116
fix(search): propagate LIMIT to base TableScan in VectorScanTableProvider (fixes #8368) by @claudespice in #11124
feat(runtime): add spicebench feature to register the Cayenne catalog connector by @phillipleblanc in #11122
fix(cayenne): tombstone inline-checkpointed rows on upsert to prevent duplicate PKs by @sgrebnov in #11129
Remove possibility of a deadlock in RuntimeStatus by @krinart in #11114
Fix Windows build: vendored-vss duckdb-rs + adapt to table-providers mongodb API by @phillipleblanc in #11140
localpod: synchronize child refreshes when parent uses in-memory (arrow) accelerator by @phillipleblanc in #11139
Add datasets dimension to the query_executions metric by @phillipleblanc in #11138
fix(spiceai): keep correlated subqueries out of JOIN ON for Spice Cloud federation by @phillipleblanc in #11143
fix(duckdb): normalize timestamp columns to microsecond precision (fixes #10627) by @claudespice in #11145
feat(cayenne): sharded parallel Vortex encode with key/time clustering by @lukekim in #11144
fix(cluster): prevent DoPut write pipeline self-deadlock under ingest backpressure by @phillipleblanc in #11160
feat(chbench): configurable HTAP concurrency, DuckDB query overrides, and OLTP rate control by @sgrebnov in #11162
fix(http): preserve non-JSON response rows instead of crashing nested decomposition (fixes #11155) by @claudespice in #11161
Use declared schema in DynamoDB/MongoDB/Debezium by @krinart in #11066
fix(cluster): prevent partitioned datasets from staying Refreshing by @phillipleblanc in #11157
fix(runtime): don't list postgres as a valid accelerator engine when postgres-accel is disabled by @sgrebnov in #11169
fix(spark): recover stale or broken Spark Connect sessions on failure by @lukekim in #11171
fix(secrets): don't abort secret lookup precedence walk on a failing store by @phillipleblanc in #11175
feat(cayenne): bound aggregate write concurrency, conservative defaults, and write/read observability by @lukekim in #11170
feat(unity_catalog): support Unity Catalog credential vending for Delta Lake tables by @phillipleblanc in #11180
fix(secrets): keep failed secret stores registered so lookups report the init root cause by @phillipleblanc in #11181
fix(debezium): sign-extend minimal-width base64 decimals instead of zero-padding by @claudespice in #11184
fix(deps): update hickory-resolver to 0.26 (evicts hickory-proto 0.25.x) by @phillipleblanc in #11183
refactor(secrets): derive secret store metadata from a single registry table by @phillipleblanc in #11188
perf(cayenne): cut CDC replication lag on hot upsert tables by @lukekim in #11191
feat(cayenne): async inline-fallback (per-tombstone published flag) + 64c/256GB tuning by @lukekim in #11194
Add HTTP connector mTLS support by @lukekim in #11127
feat(snowflake): push AT TIME ZONE as CONVERT_TIMEZONE and pin session to UTC by @lukekim in #11190
feat(cdc): make cdc_max_coalesce_age_ms a real apply-loop linger by @sgrebnov in #11196
feat(cayenne): delta-write encoding levels (cayenne_delta_encoding, default auto) + make compression_strategy=zstd real by @lukekim in #11199
Add NSQL context endpoint by @lukekim in #11075
fix(federation): respect the Spice function deny-list across all SQL connectors; dialect-aware DuckDB pushdown by @claudespice in #11186
fix: surface unknown/applied cayenne_* runtime.params at startup (fixes #10970) by @claudespice in #11133
perf(cayenne): plain-fsync ordering tier on the staged-commit hot path by @lukekim in #11198
fix(kafka): decode JSON payloads to Arrow directly — fixes lossy Decimal128 + removes double-parse (#11192) by @claudespice in #11207
feat(cayenne): self-tuning accelerator — hardware + schema + closed-loop adaptive (auto/adaptive modes) by @lukekim in #11213
perf(cayenne): CDC throughput — SF-100 @10K txn/s toward <5s lag + 5K QPH by @lukekim in #11206
fix(cluster): distribute accelerated tables wrapped by metadata/index providers by @phillipleblanc in #11226
Improve Cayenne adaptive tuning and schema safety by @lukekim in #11237
feat: Add cayenne_file_pruning param by @peasee in #11239
Debezium connector - handle tombstone messages in kafka topic, with schema evolution enabled by @ewgenius in #11197
feat(cayenne): broadcast small-dimension joins to executors by @phillipleblanc in #11245
fix: scope request context across the managed query runtime by @phillipleblanc in #11253
fix: Strip inference columns from table schema on query by @peasee in #11251
fix(flightsql): don't drop un-pushed FilterExec predicates in distributed pushdown rules by @claudespice in #11256
feat(postgres): share one replication slot across multiple changes-mode datasets by @phillipleblanc in #11255
Upgrade to DataFusion v53.1, Arrow v58.3, Vortex v0.74, and dependencies by @lukekim in #11118
feat(connectors): support file_format: vortex everywhere parquet is supported by @lukekim in #11282
perf(cayenne): metadata-only publish — drop per-key insert records on upsert commit by @lukekim in #11260
fix(kafka): harden fetch_latest_message for multi-partition topics by @ewgenius in #11285
perf(cayenne): bound mem-tier checkpoint churn + O(1) per-scan deletion view by @lukekim in #11249
fix(udfs): length-prefix digest_many values so column boundaries can't collide (fixes #11272) by @claudespice in #11288
fix: restore federation deny-list enforcement regressed by the DataFusion 53 upgrade by @claudespice in #11294
fix(postgres): recover unchanged-TOAST columns from the old tuple; classify walsender contention as transient by @phillipleblanc in #11293
feat: deepen extended schema inference and wire it into cayenne compaction sharding/sorting by @lukekim in #11284
perf(cayenne): in-memory CDC tier follow-ups + write-phase observability by @lukekim in #11278
Support per-dataset CDC tunable overrides by @sgrebnov in #11295
feat(cayenne): harden adaptive auto-tuner (controller hygiene, mem-tier actuator, delete/burst signals, single opt-in) by @lukekim in #11302
fix(cayenne): scan inlined-view capture starvation under sustained CDC (analytical QPH) by @lukekim in #11299
fix(vortex): don't row-evaluate hash-join dynamic filters in the scan by @sgrebnov in #11307
fix(deps): bump rust-postgres crates (RUSTSEC-2026-0178/0179) by @lukekim in #11313
fix: strict validation of Postgres CDC params instead of silent defaults (fixes #11274) by @claudespice in #11304
fix(duckdb): always quote on-refresh sort columns so reserved-word names don't break refresh by @claudespice in #11305
feat(flightsql): fall back to original connection when endpoint location is unreachable by @melks in #11287
perf(cayenne): light-encode transient staged CDC deltas by @lukekim in #11311
feat(acceleration): widening-only schema evolution via on_schema_change by @lukekim in #11261
deps(vortex): bump pin to spiceai-53 HEAD — intra-file decode split + per-execution kernel cache by @lukekim in #11314
feat(github): enhance GitHub component validation and error handling by @lukekim in #11259
feat(cayenne): goal-driven adaptive tuning toward operator SLOs (lag, freshness, query latency, QPH) by @lukekim in #11310
fix(cayenne): broadcast-join rewrite must bail on ambiguous columns, NULL-equal joins, and residual filters by @claudespice in #11252
Add Cayenne maintained aggregates by @lukekim in #11235
perf(cayenne): single-hash composite deletion filter via KeyDeletionIndex::get_batch by @phillipleblanc in #11325
fix(Vortex): decline only the InList membership conjunct of hash-join dynamic filters by @sgrebnov in #11335
fix: scope SQL UDF arg inlining to args-table columns (fixes #11273) by @claudespice in #11337
fix(refresh): restore S3 ETag/Version refresh-skip behind provider wrappers by @phillipleblanc in #11339
fix(cayenne): ref-count in-flight scans so GC can't delete Vortex files mid-read by @phillipleblanc in #11321
fix(runtime): retry object-store dataset load when source files are not yet available by @phillipleblanc in #11342
feat(s3): default to path-style for dotted bucket names on standard AWS by @phillipleblanc in #11347
fix(runtime): resolve accelerated table through metadata-enrichment wrapper by @phillipleblanc in #11345
fix: detect dual-write accelerated tables behind the metadata-enrichment wrapper by @claudespice in #11351
feat(cayenne): incremental seq-prefix bake — shrink the merge-on-read deletion index by @lukekim in #11326
fix(adbc): prevent Spice-specific UDFs from being pushed down to ADBC sources by @krinart in #11297
fix: Query Redshift schema details from svv_redshift tables by @peasee in #11362
perf(cayenne): tune Turso connection PRAGMAs + jitter metastore retries by @lukekim in #11359
Upgrade to DataFusion 54 by @sgrebnov in #11360
feat(runtime): dedicated CDC-apply tokio runtime + per-runtime tokio metrics by @lukekim in #11370
fix(cayenne): spill oversized hash joins via sort-merge to avoid OOM by @lukekim in #11371
fix(cayenne): honor cayenne_metastore: turso for partitioned tables and the dataset checkpoint by @phillipleblanc in #11365
perf(cayenne): in-RAM scan parallelism, query admission control, skip no-op deletion encode, sound scan output_ordering by @lukekim in #11332
fix(catalog): restore DDL after DataFusion 54 broke transparent catalog-provider downcasts by @phillipleblanc in #11375
feat(flightsql): infer schema via SELECT * LIMIT 1 when GetTables is unimplemented by @melks in #11286
fix(cayenne): Utf8View read schema avoids hash-join offset overflow by @lukekim in #11379
feat(cluster): support distributed (Ballista) scans of Iceberg tables by @phillipleblanc in #11378
feat(optimizer): cost-based left-deep join reordering for Cayenne acceleration by @sgrebnov in #11377
cli - fix service-account auth in spice cloud * commands by @ewgenius in #11316
fix: Support external Redshift table schema inference and Hive external type parsing by @peasee in #11399
feat(llms): native GLM support — opt-in flash-attn + surface reasoning_content by @lukekim in #11400
feat(s3_vectors): paginate QueryVectors for topK up to 10,000 by @bjchambers in #11405
fix: surface .env parse errors with line numbers instead of silently skipping by @Oxygen56 in #11306
fix(status): probe metrics endpoint over https when TLS is enabled by @phillipleblanc in #11393
fix(mcp): record task_history spans for tool calls proxied through /v1/mcp by @phillipleblanc in #11397
Properly handle date_trunc in BigQueryDialect by @krinart in #11416
fix(snowflake): honor column scale in Int64 timestamp arm and cast TIME by @claudespice in #11418
fix: /v1/queries chunk row_offset uses cumulative offset, not chunk_index * chunk_size (fixes #11271) by @claudespice in #11398
feat(cayenne): incremental retraction for maintained aggregates + anchor bench by @lukekim in #11389
feat(cluster): support distributed (Ballista) scans of Iceberg catalog tables by @phillipleblanc in #11419
Simplify chat/responses models by @Jeadie in #10997
feat(llms): distributed tensor-parallel GLM inference via mistral.rs ring backend by @lukekim in #11406
Optimize Flight streaming for large result sets by @lukekim in #11420
fix(deps): evict rustls 0.21 / rustls-webpki 0.101.7 (GHSA-82j2-j2ch-gfr8) by @phillipleblanc in #11428
feat(cayenne): in-memory CDC intra-apply sharding (PK-hash shards) by @lukekim in #11421
fix(cayenne): shard CDC upserts with pending deletions so the N>1 slot-ack advances by @lukekim in #11445
fix(cluster): keep built-in avg over Spark avg (distributed aggregate state schema mismatch) by @phillipleblanc in #11434
feat(views): support params.file_format for embedding chunking by @Jeadie in #11424
fix: offload blocking sync calls off the primary async runtime by @phillipleblanc in #11435
fix(udfs): rebind dot_product alias to Spice's inner_product on DataFusion 54 by @lukekim in #11443
feat(secrets): add full-fidelity reference iteration and a resolution-status API by @phillipleblanc in #11195
fix: Deny unsupported array functions for Postgres pushdown by @peasee in #11450
fix(cli): spice query honors --http-endpoint instead of failing on a Flight connect by @phillipleblanc in #11452
fix(cayenne): coordinate query-pool + in-memory CDC tier budgets to prevent adaptive OOM by @lukekim in #11449
Surface mTLS config in Kafka data connector by @v1gnesh in #11372
feat(secrets): check and report secret references at startup by @phillipleblanc in #11457
Default cayenne_force_view_types to false by @sgrebnov in #11459
fix: resolve table-reference qualification in results-cache invalidation (fixes #11266) by @claudespice in #11460
feat(cayenne): metadata aggregate pushdown — fold whole-table SUM/AVG/COUNT/MIN/MAX from statistics by @bjchambers in #11414
fix(search): restore numeric trunc and fix SortPreservingMergeExec planning error (DF54) by @Jeadie in #11415
fix(cluster): route Ballista shuffle/temp to the data PVC by @phillipleblanc in #11454
fix(search): default Elasticsearch kNN candidate pool to 1000 instead of 10 (fixes #11264) by @claudespice in #11467
feat(acceleration): add on_schema_change drop_and_recreate policy by @lukekim in #11462
feat(cayenne): predicate-aware maintained aggregates serve filtered analytical queries from the CDC delta by @lukekim in #11458
feat(cluster): shared Ballista job state with scheduler failover by @phillipleblanc in #11436
feat(cayenne): extend HLL NDV sketching to string and date columns by @bjchambers in #11468
fix(search): persist character chunk offsets so search snippets aren't shifted/garbled (fixes #11269) by @claudespice in #11479
feat(cayenne): storage-aware adaptive CDC tuning — calibration probe, IMDS, I/O-cliff fast path, infeasible-SLO feedback by @lukekim in #11463
fix(cayenne): LIMIT N under-delivers on key-deletion tables by @lukekim in #11490
fix(cayenne): live/tier-accurate join build-side stats (merge-on-read deletes + never-shrink NDV) by @lukekim in #11496
perf(cayenne): kernel-space I/O hygiene — compaction fadvise + staged-commit barrier reduction by @lukekim in #11495
feat(cayenne): feed maintained-aggregate IVM from the staged-disk CDC path by @lukekim in #11491
feat(cayenne): global adaptive-tuning SLOs with per-dataset overrides; QPH global-only by @lukekim in #11497
Update search snapshots by @sgrebnov in #11473
fix: Support reading column types longer than 128 chars in Redshift by @peasee in #11500
fix(cluster): distributed (Ballista) query-execution config + scheduled SF10 bench by @phillipleblanc in #11478
fix(http): retry transient response-body read failures; de-flake backoff test by @claudespice in #11482
Re-land orphaned deletion-vector cleanup during retention deletes by @lukekim in #11501
fix(cayenne): re-upsert over a pending delete tombstone records an insert-record (overwrite resurrection) by @bjchambers in #11469
fix: Flight DoPut silently dropped client batches on early sink completion by @claudespice in #11507
Upgrade OpenTelemetry to 0.32 and reqwest to 0.13 by @phillipleblanc in #11506
fix(queries): run async /v1/queries jobs under the submitting request context by @phillipleblanc in #11505
fix(cayenne): restore append-only current-snapshot compaction by @Jeadie in #11439
perf(cayenne): orphaned deletion-vector cleanup off the write path, behind a knob by @bjchambers in #11517
fix(datafusion): accurate projected scan byte size so hash joins build the smaller side by @sgrebnov in #11503
fix(cayenne): user-visible DELETE WHERE pk IN (...) reports the real row count by @lukekim in #11514
fix(cayenne): seed persisted num_rows for hash-join sizing by @sgrebnov in #11515
fix(cayenne): correctness & memory_limit fixes from perf audit (2 P0, 3 P1) by @lukekim in #11516
feat(cayenne): wire orphaned-DV cleanup knob to spicepod params + doc sync by @bjchambers in #11523
Fix s3 vectors API by @krinart in #11536
fix: Placeholder table initialization lock swap by @peasee in #11540
chore(cluster): bump ballista pin for the null-aware anti-join fix by @phillipleblanc in #11544
fix(runtime-tools): fix memory table identifier validation rejecting valid names by @Jeadie in #11546
Bump datafusion to include spiceai/datafusion#181 by @Jeadie in #11563
Update deny.toml by @krinart in #11571
Bump datafusion (spiceai/datafusion#182) and datafusion-table-providers (#27): fix q16 CollectLeft planning error and SQLite q6 wrong revenue by @Jeadie in #11598

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.0.0...v2.1.0

Spice v2.0-rc.4 (Apr 30, 2026)

May 1, 2026 · 22 min read

William Croxson

Senior Software Engineer at Spice AI

Announcing the release of Spice v2.0-rc.4! 🚀

v2.0.0-rc.4 is the fourth release candidate for advanced testing of v2.0, building on v2.0.0-rc.3.

Highlights in this release candidate include:

Elasticsearch Data Connector (Alpha) with native hybrid search (BM25 full-text + kNN vector + RRF)
PostgreSQL Native CDC via WAL logical replication, eliminating the need for Debezium or Kafka
Multi-vector Embeddings with MaxSim for ColBERT-style late-interaction retrieval
Rerank UDTF for hybrid search pipelines with automatic query propagation
HashiCorp Vault and Azure Key Vault Secret Stores for enterprise secret management
DuckDB Vector Engine with HNSW index support
Azure Cosmos DB Connector (RC), Git Connector promoted to RC
MCP Streamable HTTP transport
Read-only API Key Enforcement on Flight DoGet and async query paths

What's New in v2.0.0-rc.4

Elasticsearch Data Connector (Alpha, Spice.ai Enterprise)

The new Elasticsearch data connector enables querying Elasticsearch indexes as SQL tables with full hybrid search support. Currently available in Spice.ai Enterprise.

Key capabilities:

SQL Table Access: Query any Elasticsearch index with standard SQL via a native DataFusion TableProvider.
kNN Vector Search: Use the vector_search() UDTF against Elasticsearch-backed vector fields.
BM25 Full-Text Search: Use the text_search() UDTF for native Elasticsearch full-text queries.
Hybrid Search: Combine kNN and BM25 results with the rrf() UDTF for reciprocal rank fusion.
Elasticsearch as a Vector Engine: Accelerated datasets can use Elasticsearch as the backing vector engine for embedding storage and retrieval.

Example configuration:

datasets:
  - from: elasticsearch:my_index
    name: my_data
    params:
      elasticsearch_endpoint: https://my-cluster.es.io:9200
      elasticsearch_username: ${secrets:es_user}
      elasticsearch_password: ${secrets:es_password}

PostgreSQL Native Replication via WAL

Postgres datasets configured with refresh_mode: changes can now stream changes directly from PostgreSQL logical replication (WAL) into any local accelerator without Debezium or Kafka required.

Key capabilities:

Native Logical Replication: Uses pgoutput decoding to stream INSERT/UPDATE/DELETE events.
Automatic Slot Management: Each Spice replica creates a distinct replication slot (spice_<dataset>_<hash>), so multi-replica deployments work automatically. Publications are shared.
Bootstrap Snapshot: An initial REPEATABLE READ snapshot seeds the accelerator before replication begins.
LSN Acknowledgement: The LsnCommitter sends durable LSN back to Postgres so WAL segments are reclaimed.
All Accelerators Supported: Works with DuckDB, SQLite, Postgres, Cayenne, and Arrow accelerators.

Example configuration:

datasets:
  - from: postgres:my_table
    name: my_table
    params:
      pg_host: localhost
      pg_port: 5432
      pg_db: mydb
      pg_publication: my_publication   # optional; auto-created if omitted
    acceleration:
      enabled: true
      engine: duckdb
      refresh_mode: changes

Multi-vector Embeddings with MaxSim (Late Interaction)

Column-level embeddings now support list-of-string columns, producing one embedding vector per list element and enabling ColBERT-style late-interaction retrieval.

Key capabilities:

Multi-vector per Row: Columns of type List<String> produce List<FixedSizeList<F32, D>> — one embedding per list element.
MaxSim / Mean / Sum Scoring: Per-row score is the max, mean, or sum cosine over the list elements. Default is MaxSim (ColBERT).
_match Column: Returns the specific list element that produced the highest cosine similarity.
No Schema Changes Required: Works with existing embedding configurations; activates automatically for list-type columns.

Rerank UDTF for Hybrid Search

A new rerank() table-valued function reorders scored results from vector_search, text_search, or rrf by a reranker model's relevance judgements. See Search Functionality for an overview of search UDTFs.

Key capabilities:

Auto Query Propagation: The query string is automatically inherited from a nested search UDTF — no repetition required.
Any Chat Model as Reranker: Any registered chat/completion model can serve as a reranker via the built-in LlmRerank adapter (listwise prompt by default; pointwise available).
Filter and Projection Pushdown: The RerankExec physical node supports pushdown, reducing data movement.
Extensible: A new RerankerModelStore sits alongside ChatModelStore and EmbeddingModelStore; native providers (Cohere, Voyage, BGE) can be added without runtime plumbing changes.

SELECT * FROM rerank(
    rrf(vector_search('my_table', 'query text'), text_search('my_table', 'query text')),
    document => content
) LIMIT 10;

New Secret Stores: HashiCorp Vault and Azure Key Vault

Two new enterprise-grade Secret Stores are now available.

HashiCorp Vault (hashicorp_vault):

KV v2 (default) and KV v1 mount support.
Auth methods: token, approle, kubernetes, jwt.
Token leases are cached and automatically re-acquired on expiry.

secrets:
  - from: hashicorp_vault:secret/my-app
    name: my_secrets
    params:
      hashicorp_vault_addr: https://vault.example.com
      hashicorp_vault_auth_method: approle
      hashicorp_vault_role_id: ${env:VAULT_ROLE_ID}
      hashicorp_vault_secret_id: ${secrets:vault_secret_id}

Azure Key Vault (azure_keyvault):

Per-key caching with single-flight fetch coalescing.
Auth methods: service principal, managed identity, workload identity, Azure CLI, or auto-detect.
Supports sovereign clouds via endpoint parameter.

secrets:
  - from: azure_keyvault:my-vault
    name: my_secrets
    params:
      azure_keyvault_auth_method: managed_identity

DuckDB Vector Engine

DuckDB-accelerated tables can now use DuckDB's HNSW index for vector search via the vector_engine: duckdb option, enabling fast approximate nearest-neighbor search without an external vector store.

Example configuration:

datasets:
  - from: postgres:public.documents
    name: documents
    columns:
      - name: content
        embeddings:
          - from: hf_minilm
            row_id: id
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
    vectors:
      enabled: true
      engine: duckdb
      params:
        duckdb_distance_metric: cosine
        duckdb_hnsw_m: 16
        duckdb_hnsw_ef_construction: 64
        duckdb_hnsw_ef_search: 32

embeddings:
  - from: huggingface:huggingface.co/minishlab/potion-base-2M
    name: hf_minilm

New and Promoted Connectors

Azure Cosmos DB (Alpha):

A new read-only Azure Cosmos DB NoSQL / Core SQL API connector built on the azure_data_cosmos 0.30 SDK. Supports cross-partition scans, schema inference from document samples, and key-based auth (connection string or account endpoint + key).

Git Connector (RC):

The Git data connector is promoted to RC status with HTTPS/SSH auth (git_token, git_username/git_password, git_ssh_key), Git LFS support (enable_lfs), and per-repo connection resilience (semaphore, bounded retries with exponential backoff, permanent-error circuit breaking).

DynamoDB Write Support (DML)

DynamoDB datasets now support write-back via INSERT, UPDATE, and DELETE operations, complementing the existing read and CDC streaming capabilities.

MCP Streamable HTTP Transport

The MCP server has been upgraded to rmcp 1.5.0 and switched to the Streamable HTTP transport (/v1/mcp), replacing the previous SSE-based endpoint. The client-side transport is updated to StreamableHttpClientTransport.

Security Improvements

Read-only API Key Enforcement: API keys with read-only scope are now strictly enforced on the Flight DoGet path and on async query endpoints, preventing write operations from being issued under a read-only key.

GitHub Workflow Hardening: CI workflows have been hardened with improved security posture to reduce supply-chain risk.

Developer Experience Improvements

Actionable Config Errors: Parameter typos, missing secret references, and unknown engine names now produce specific, actionable error messages with Levenshtein-based suggestions, rather than silent drops or generic "missing required parameter" messages.
spice init Improvements: Written spicepods now include a yaml-language-server: $schema=... directive for IDE completions. Creation messages print regardless of log level.
REPL Improvements: Log filter honors RUST_LOG when -v is not passed; version banner moves to stderr and prints only on an interactive TTY.
403 / 401 Routing: HTTP 403 responses route to a new PermissionDenied variant; 401 messages point at spice login / SPICE_API_KEY.

OpenTelemetry Improvements

See Observability & Monitoring and the runtime.telemetry reference for full configuration details.

Metric Name Prefix: Configure a prefix for all exported OTLP metric names via runtime.telemetry.metric_prefix.
Delta Temporality Default: The OTLP push exporter now defaults to delta temporality, matching Prometheus and most backends.
Resource Attributes: runtime.telemetry.properties are applied as OTLP resource attributes on exported metrics.

Full-text Search Performance

Tantivy full-text search ingestion performance is significantly improved with better batch handling and a rollback-on-error path.

SQL and Query Engine

DataFusion Upgrade: Updated to a newer DataFusion revision with additional bug fixes and performance improvements.
Views on DDL Catalogs: DDL-defined catalogs (e.g., Unity Catalog) can now expose and query views.
flatten_json / json_tree / expand_maps UDTFs: New table-valued functions for JSON transformation, map expansion, and schema decomposition in query pipelines. See JSON Functions and Operators.
cosine_distance Pushdown to DuckDB: cosine_distance is now pushed down to DuckDB accelerators via array_cosine_distance.
Snowflake Type Support: Added support for OBJECT, MAP, GEOGRAPHY, GEOMETRY, VECTOR, and TIMESTAMP_LTZ types in the Snowflake connector.
MySQL Zero-Date Behavior: The MySQL connector adds a new mysql_zero_date_behavior parameter (null or error) controlling how MySQL zero-date values (0000-00-00) are handled.
Databricks Timeouts: The Databricks connector adds new connect_timeout and client_timeout parameters for sql_warehouse mode.

Dependency Updates

Dependency / Component	Version / Update
DataFusion	Updated
rmcp	v1.5.0 (from fork pin)
mistral.rs	Updated
openssl	0.10.78

Contributors

Breaking Changes

No breaking changes.

Cookbook Updates

No new cookbook recipes.

The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.

Upgrading

To upgrade to v2.0.0-rc.4, use one of the following methods:

CLI:

spice upgrade v2.0.0-rc.4

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.0.0-rc.4 image:

docker pull spiceai/spiceai:2.0.0-rc.4

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.0.0-rc.4

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed

Changelog

Integrate spiceio and makefile_targets into pr.yml by @lukekim in #10357
ci: skip artifact compression for test binaries/archives by @lukekim in #10381
chore(deps): bump spiceai/candle, spiceai/mistral.rs, aws-lc-rs, tantivy, rand by @lukekim in #10379
Bump datafusion-table-providers (#10375) by @lukekim in #10384
fix: Update Search integration test snapshots by @app/github-actions in #10376
v2.0.0-rc.3 preparation by @ewgenius in #10382
fix(spicepod): JSON schema accepts string or {name: expr} for partition_by by @lukekim in #10352
fix: Use ROUND for Turso decimal BETWEEN comparisons (fixes #9872) by @claudespice in #10360
Revert "v2.0.0-rc.3 preparation" from trunk by @ewgenius in #10386
Add on_schema_resolved dataset ready state by @lukekim in #10368
feat: Add Elasticsearch data connector with hybrid search support by @lukekim in #10258
ci: bump test archive upload compression-level to 1 by @lukekim in #10388
feat(git-connector): promote Git connector to RC status by @lukekim in #10385
feat(postgres): stream WAL directly to Spice accelerators by @lukekim in #10364
Add schema decomposition to the HTTP connector by @lukekim in #10393
fix(cayenne): Skip catalog refresh state reload for existing providers by @sgrebnov in #10396
Make cayenne-flightsql tool by @Jeadie in #10356
build(deps): bump the github-actions-dependencies group with 2 updates by @app/dependabot in #10398
Update openapi.json by @app/github-actions in #10272
Merge develop to trunk — 2026-04-19 by @claudespice in #10407
feat(otel): default OTLP push exporter to delta temporality by @phillipleblanc in #10412
fix: Restore analyzer rule ordering to run federation before type coercion by @sgrebnov in #10415
fix: Map Utf8/LargeUtf8 to STRING in Databricks/Spark SQL dialects by @sgrebnov in #10420
feat(otel): add metric name prefix at runtime.telemetry.metric_prefix by @phillipleblanc in #10418
fix: Map LargeUtf8 to VARCHAR in Athena ODBC dialect by @sgrebnov in #10419
feat(cluster): connector-driven object store registration on executors by @phillipleblanc in #10414
build(deps): bump ubuntu from 22.04 to 24.04 in the docker-dependencies group by @app/dependabot in #10397
fix: Update benchmark snapshots Apr 20 by @app/github-actions in #10417
feat(otel): apply runtime.telemetry.properties as resource attributes on exported metrics by @phillipleblanc in #10416
Publish RC releases to DockerHub; upgrade runners to ubuntu-24.04 by @lukekim in #10428
feat: Add Azure Cosmos DB (NoSQL) data connector (RC) by @lukekim in #10392
feat(datafusion): flatten_json_properties + json_tree UDTFs by @lukekim in #10406
Harden /v1/tools and /v1/nsql against unauthenticated / LLM-driven SQL by @lukekim in #10365
feat(embeddings): multi-vector embeddings with MaxSim + late-interaction by @lukekim in #10408
Update GH runners for CUDA builds by @ewgenius in #10432
fix(delta_lake): register object stores on cluster executors by @phillipleblanc in #10436
DF-native DML by @krinart in #10327
ci: run Build and Test on spiceai-macos; split install jobs by profile by @lukekim in #10434
Improve search UDTFs: text_search, vector_search, rrf by @lukekim in #10387
fix(model2vec): Improve robustness of model loading for sentence-transformers layouts by @sgrebnov in #10444
Merge develop to trunk — 2026-04-21 by @claudespice in #10448
Enable filter pushdown for vector_search UDTF by @sgrebnov in #10447
Support Snowflake OBJECT, MAP, GEOGRAPHY, GEOMETRY, VECTOR, TIMESTAMP_LTZ types by @lukekim in #10451
Fix Databricks tests by @krinart in #10449
fix(cluster): forward register_object_stores through connector wrappers by @phillipleblanc in #10460
Fixes for vector-search by @krinart in #10455
Add expand_maps option and flatten_json UDTF by @lukekim in #10452
fix: Update Search integration test snapshots by @app/github-actions in #10458
Fix physical codec decode ambiguity for empty protobuf messages by @sgrebnov in #10466
chore(logging): demote s3_single_file_cached skip refresh log to debug by @phillipleblanc in #10467
Enable filter pushdown for rrf UDTF by @sgrebnov in #10465
feat(cluster): consolidate distributed state into cluster.json by @phillipleblanc in #10463
feat(cayenne): Add column statistics and data inlining by @lukekim in #10314
docs(copilot): flag missing wrapper delegation when adding default trait methods by @phillipleblanc in #10461
Wire Elasticsearch vector engine write path through acceleration by @lukekim in #10453
Add helm lint CI by @ewgenius in #10468
Fix Azure and GCS acceleration snapshot object store credential handling by @phillipleblanc in #10486
Update spicepod.schema.json by @app/github-actions in #10485
fix(secrets): harden AWS Secrets Manager secret store by @lukekim in #10478
Update datafusion-ballista crate by @sgrebnov in #10488
feat(secrets): add ParameterSpec and more params for AWS secrets manager by @phillipleblanc in #10487
Add rerank UDTF for hybrid search with query auto-propagation by @lukekim in #10469
Fix flatten_json_properties by @krinart in #10475
fix: preserve field and schema metadata in expand_views_schema by @claudespice in #10494
Upgrade rmcp to upstream 1.5.0; switch MCP server to Streamable HTTP by @lukekim in #10491
fix: handle Snowflake TIMESTAMP_LTZ wire format and prevent nanosecond overflow by @claudespice in #10493
Lint parity in Makefile by @krinart in #10492
Add connect_timeout/client_timeout params to Databricks sql_warehouse mode by @lukekim in #10495
fix(tracing): suppress opentelemetry INFO logs at all verbosity levels by @lukekim in #10497
DynamoDB DML by @krinart in #10470
feat(cayenne): native vector search via SIMD similarity UDFs by @lukekim in #10456
fix(cli): suppress banner for all JSON-producing cloud subcommands (fixes #10498) by @claudespice in #10510
fix(deps): bump openssl to 0.10.78 by @phillipleblanc in #10509
fix(s3): quiet AWS SDK credential probe when no region is configured by @phillipleblanc in #10506
fix(cdc): emit ready signal on caught-up Kafka/Debezium streams (#5201) by @phillipleblanc in #10504
runtime-cluster crate + Run partition discovery before forwarding refresh to executors by @krinart in #10490
Update lint-rust target to use --keep-going by @Jeadie in #10508
Add TPC-H SF100 s3[parquet]-duckdb[file] benchmark spicepod by @lukekim in #10524
Remove dev-profile install steps from pr.yml by @Jeadie in #10507
fix: add missing NULL check on Timestamp path in append refresh by @claudespice in #10518
fix: return error on Decimal128/256 overflow instead of silently dropping scale by @claudespice in #10519
fix: delegate update and delete_from in IndexedTableProvider and EmbeddingTable by @claudespice in #10520
feat(devx): make config errors, CLI, and REPL lead users to success by @lukekim in #10489
fix(rerank): defer execution to RerankExec, enable filters and projection pushdown by @sgrebnov in #10514
fix(llms): support Gemma models with missing attention_bias config field by @lukekim in #10523
Fix vector_search silently ignoring named limit/column/include_score args by @sgrebnov in #10527
fix: split unsupported filters locally in scan() for UseSource mode by @ewgenius in #10528
feat(secrets): add Azure Key Vault secret store by @lukekim in #10496
Bump mistralrs by @krinart in #10532
Fix benchmark configurations and CI build issues by @sgrebnov in #10535
Fix catalog query overrides for MySQL and MSSQL benchmarks by @sgrebnov in #10543
For Cayenne, preserve matched columns for MERGE ... ON <cols> by @Jeadie in #10340
build(deps): bump the aws-sdk group across 1 directory with 5 updates by @app/dependabot in #10538
docs: update AI agent instructions (git workflow + Rust 1.94) by @lukekim in #10544
fix: Update tpch benchmark snapshots by @app/github-actions in #10529
fix: Update tpch benchmark snapshots for accelerated/s3[parquet]-duckdb[file].yaml by @app/github-actions in #10525
Extract runtime-datafusion from runtime by @krinart in #10545
Use generic DML extension planner for Cayenne by @Jeadie in #10437
fix: Update Search integration test snapshots by @app/github-actions in #10552
Fix security and correctness audit issues by @lukekim in #10526
fix(MySQL): revert MySQL result column reorder to fix federated query failures by @sgrebnov in #10557
Fix protoc installation by @krinart in #10566
fix: Disable Ballista dynamic filters on HashJoinExec by @peasee in #10548
Support views on DDL catalogs by @Jeadie in #10554
Update datafusion by @Jeadie in #10422
Improve full-text search indexing performance by @sgrebnov in #10464
feat(mysql): add mysql_zero_date_behavior parameter (null|error) by @phillipleblanc in #10573
fix(snowflake): declare private_key in connector PARAMETERS (fixes #10517) by @claudespice in #10559
Honour CARGO_TARGET_DIR in Makefiles by @Jeadie in #10569
Enable cosine_distance pushdown to DuckDB accelerator via array_cosine_distance by @sgrebnov in #10564
fix: Update test snapshots by @app/github-actions in #10570
fix: Update tpch benchmark snapshots by @app/github-actions in #10560
feat(snapshots): make snapshots an optional feature by @phillipleblanc in #10574
Enforce read-only API key restrictions on Flight DoGet and async query paths by @Jeadie in #10551
Improved security posture on Github workflows by @Jeadie in #10556
fix: Update datafusion-table-providers to improve SqlTable filter pushdown by @sgrebnov in #10595
feat(secrets): add HashiCorp Vault secret store by @phillipleblanc in #10561
fix: delegate update() in UpsertDedupTableProvider to inner provider by @claudespice in #10593
Add DuckDB vector engine support by @lukekim in #10562
Sharepoint - add object-store listing connector with expanded auth and write support by @lukekim in #10473
fix: Install protoc from source by @peasee in #10597

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.0.0-rc.3...v2.0.0-rc.4

Spice v1.8.2 (Oct 21, 2025)

October 21, 2025 · 5 min read

Jack Eadie

Token Plumber at Spice AI

Announcing the release of Spice v1.8.2! 🔍

Spice v1.8.2 is a patch release focused on reliability, validation, performance, and bug fixes, with improvements across DuckDB acceleration, S3 Vectors, document tables, and HTTP search.

What's New in v1.8.2

Support Table Relations in `/v1/search` HTTP Endpoint

Spice now supports table relations for the additional_columns and where parameters in the /v1/search endpoint. This enables improved search for multi-dataset use cases, where filters and columns can be used on specific datasets.

Example:

curl 'http://localhost:8090/v1/search' \
    -H 'Content-Type: application/json' \
    -H 'Accept: application/json' -d '{
        "text": "hello world",
        "additional_columns": ["tbl1.foo", "tbl2.bar", "baz"],
        "where": "tbl1.foo > 100000",
        "limit": 5
    }'

In this example, search results from the tbl1 dataset will include columns foo and baz, where foo > 100000. For tbl2, columns bar and baz will be returned.

DuckDB Data Accelerator Table Partitioning & Indexing

Configurable DuckDB Index Scan: DuckDB acceleration now supports configurable duckdb_index_scan_percentage and duckdb_index_scan_max_count parameters, supporting fine-tuning of index scan behavior for improved query performance.

Example:

datasets:
  - from: postgres:my_table
    name: my_table
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      params:
        # When combined, DuckDB will use an index scan when the number of qualifying rows is less than the maximum of these two thresholds
        duckdb_index_scan_percentage: '0.10' # 10% as decimal
        duckdb_index_scan_max_count: '1000'

Hive-Style Partitioning: In file-partitioned mode, the DuckDB data accelerator uses Hive-style partitioning for more efficient file management.
Table-Based Partitioning: Spice now supports partitioning DuckDB accelerations within a single file. This approach maintains ACID guarantees for full and append mode refreshes, while optimizing resource usage and improving query performance. Configure via the partition_mode parameter:

datasets:
  - from: file:test_data.parquet
    name: test_data
    params:
      file_format: parquet
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      params:
        partition_mode: tables
      partition_by:
        - bucket(100, Field1)

S3 Vectors Reliability

Race Condition Fix: Resolved a race condition in S3 Vectors index and bucket creation. The runtime also now checks if an index or bucket exists after a ConflictException, ensuring robust error handling during index creation and improving reliability for large-scale multi-index vector search.

Document Table Improvements

Primary Key Update: Document tables now use the location column as the primary key, improving performance, consistency, and query reliability.

Additional Improvements & Bugfixes

Reliability: Improved error handling and resource checks for S3 Vectors and DuckDB acceleration.
Validation: Expanded validation for partitioning and index creation.
Performance: Optimized partition refresh and index scan logic.
Bugfix: Don't nullify DuckDB release callbacks for schemas.

Contributors

Breaking Changes

No breaking changes.

Cookbook Updates

No major cookbook updates.

The Spice Cookbook includes 81 recipes to help you get started with Spice quickly and easily.

Upgrading

To upgrade to v1.8.2, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.8.2 image:

docker pull spiceai/spiceai:1.8.2

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

🎉 Spice is now available in the AWS Marketplace!

What's Changed

Changelog

Update mongo config for benchmarks by @krinart in #7546
Configurable DuckDB duckdb_index_scan_percentage & duckdb_index_scan_max_count by @lukekim in #7551
Fix race condition in S3 Vectors index and bucket creation by @kczimm in #7577
Use 'location' as primary key for document tables by @Jeadie in #7567
Update official Docker builds to use release binaries by @phillipleblanc in #7597
Hive-style partitioning for DuckDB file mode by @kczimm in #7563
New Generate Changelog workflow by @krinart in #7562
Add support for DuckDB table-based partitioning by @sgrebnov in #7581
DuckDB table partitioning: delete partitions that no longer exist after full refresh by @sgrebnov in #7614
Rename duckdb_partition_mode to partition_mode param by @sgrebnov in #7622
Fix license issue in table-providers by @phillipleblanc in #7620
Make DuckDB table partition data write threshold configurable by @sgrebnov in #7626
fix: Don't nullify DuckDB release callbacks for schemas by @peasee in #7628
Fix integration tests by reverting the use of batch inserts w/ prepared statements by @phillipleblanc in #7630
Return TableProvider from CandidateGeneration::search by @Jeadie in #7559
Handle table relations in HTTP v1/search by @Jeadie in #7615

Spice v1.7.1 (Sep 29, 2025)

September 30, 2025 · 6 min read

Kevin Zimmerman

Principal Software Engineer at Spice AI

Announcing the release of Spice v1.7.1! 🔍

Spice v1.7.1 is a patch release focused on search improvements, bug fixes, and performance enhancements. This release introduces the Reciprocal Rank Fusion (RRF) user-defined table function (UDTF) for hybrid search, improves vector and text search reliability, and resolves several issues across the runtime, connectors, and query engine.

What's New in v1.7.1

Reciprocal Rank Fusion (RRF) UDTF: Spice now supports Reciprocal Rank Fusion (RRF) as a user-defined table function, enabling advanced hybrid search scenarios that combine results from multiple search methods (e.g., vector and text search) for improved relevance ranking.

Features:

Multi-search fusion: Combine results from vector_search, text_search, and other search UDTFs in a single query.
Advanced tuning: Per-query ranking weights, recency boosting, and configurable decay functions.
Performance: Optional user-specified join key for optimal performance.
Automatic joining: Falls back to on-the-fly JOIN key computation when no explicit key is provided.

Example usage:

SELECT id, title, content, fused_score
FROM rrf(
  vector_search(documents, 'machine learning algorithms', rank_weight => 1.5),
  text_search(documents, 'neural networks deep learning', rank_weight => 1.2),
  join_key => 'id',    -- optional join key for optimal performance
  k => 60.0            -- optional smoothing factor
)
WHERE fused_score > 0.01
ORDER BY fused_score DESC;

Learn more in the RRF documentation.

Acceleration Refresh Metrics: Spice now exposes additional Prometheus metrics that provide detailed observability into dataset acceleration refreshes. These metrics help monitor data freshness and ingestion lag for accelerated datasets with a time column.

Reported metrics:

Metric Name	Description
`dataset_acceleration_max_timestamp_before_refresh_ms`	Maximum value of the dataset's time column before refresh (milliseconds).
`dataset_acceleration_max_timestamp_after_refresh_ms`	Maximum value of the dataset's time column after refresh (milliseconds).
`dataset_acceleration_refresh_lag_ms`	Difference between max timestamp after and before refresh (milliseconds).
`dataset_acceleration_ingestion_lag_ms`	Lag between current wall-clock time and max timestamp after refresh (milliseconds).

These metrics are emitted during each acceleration refresh and can be scraped by Prometheus for monitoring and alerting. For more details, see the Observability documentation.

Bug Fixes & Improvements

This release resolves several issues and improves reliability across search, connectors, and query planning:

Full-Text Search (FTS): Ensure FTS metadata columns can be used in projection, fix JOIN-level filters not having columns in schema, and adds support for persistent file-based FTS indexes. Default limit of 1000 results if no limit specified.
Vector Search: Default limit of 1000 results if no limit specified, and fix removing embedding column.
Databricks SQL Warehouse: Improved error handling and support for async queries.
Other: Fixes for Anthropic model regex validation, tweaked AI-model health checks, and improved error messages.

Contributors

Breaking Changes

No breaking changes.

Cookbook Updates

Added Hybrid-Search using RRF - Combine results from multiple search methods (vector and text search) using Reciprocal Rank Fusion for improved relevance ranking.

The Spice Cookbook includes 78 recipes to help you get started with Spice quickly and easily.

Upgrading

To upgrade to v1.7.1, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.7.1 image:

docker pull spiceai/spiceai:1.7.1

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

🎉 Spice is now available in the AWS Marketplace!

What's Changed

Changelog

ensure FTS metadata columns can be used in projection (#7282) by @Jeadie in #7282
Fix JOIN level filters not having columns in schema (#7287) by @Jeadie in #7287
Use file-based fts index (#7024) by @Jeadie in #7024
Remove 'PostApplyCandidateGeneration' (#7288) by @Jeadie in #7288
RRF: Rank and recency boosting (#7294) by @mach-kernel in #7294
RRF: Preserve base ranking when results differ -> FULL OUTER JOIN does not produce time column (#7300) by @mach-kernel in #7300
fix removing embedding column (#7302) by @Jeadie in #7302
RRF: Fix decay for disjoint result sets (#7305) by @mach-kernel in #7305
RRF: Project top scores, do not yield duplicate results (#7306) by @mach-kernel in #7306
RRF: Case sensitive column/ident handling (#7309) by @mach-kernel in #7309
For vector_search, use a default limit of 1000 if no limit specified (#7311) by @lukekim in #7311
Fix Anthropic model regex and add validation tests (#7319) by @ewgenius in #7319
Enhancement: Implement before/after/lag metrics for acceleration refresh (#7310) by @krinart in #7310
Refactor chat model health check to lower tokens usage for reasoning models (#7317) by @ewgenius in #7317
Enable chunking in SearchIndex (#7143) by @Jeadie in #7143
Use logical plan in SearchQueryProvider. (#7314) by @Jeadie in #7314
FTS max search results 100 -> 1000 (#7331) by @Jeadie in #7331
Improve Databricks SQL Warehouse Error Handling (#7332) by @sgrebnov in #7332
use spicepod embedding model name for 'model_name' (#7333) by @Jeadie in #7333
Handle async queries for Databricks SQL Warehouse API (#7335) by @phillipleblanc in #7335
RRF: Fix ident resolution for struct fields, autohashed join key for varying types (#7339) by @mach-kernel in #7339

Spice v1.7.0 (Sep 23, 2025)

September 23, 2025 · 21 min read

Sergei Grebnov

Senior Software Engineer at Spice AI

Announcing the release of Spice v1.7.0! ⚡

Spice v1.7.0 upgrades to DataFusion v49 for improved performance and query optimization, introduces real-time full-text search indexing for CDC streams, EmbeddingGemma support for high-quality embeddings, new search table functions powering the /v1/search API, embedding request caching for faster and cost-efficient search and indexing, and OpenAI Responses API tool calls with streaming. This release also includes numerous bug fixes across CDC streams, vector search, the Kafka Data Connector, and error reporting.

What's New in v1.7.0

DataFusion v49 Highlights

DataFusion Clickbench Performance Graph Source: DataFusion 49.0.0 Release Blog.

Performance Improvements 🚀

Equivalence System Upgrade: Faster planning for queries with many columns, enabling more sophisticated sort-based optimizations.
Dynamic Filters & TopK Pushdown: Queries with ORDER BY and LIMIT now use dynamic filters and physical filter pushdown, skipping unnecessary data reads for much faster top-k queries.
Compressed Spill Files: Intermediate files written during sort/group spill to disk are now compressed, reducing disk usage and improving performance.
WITHIN GROUP for Ordered-Set Aggregates: Support for ordered-set aggregate functions (e.g., percentile_disc) with WITHIN GROUP.
REGEXP_INSTR Function: Find regex match positions in strings.

Spice Runtime Highlights

EmbeddingGemma Support: Spice now supports EmbeddingGemma, Google's state-of-the-art embedding model for text and documents. EmbeddingGemma provides high-quality, efficient embeddings for semantic search, retrieval, and recommendation tasks. You can use EmbeddingGemma via HuggingFace in your Spicepod configuration:

Example spicepod.yml snippet:

embeddings:
  - from: huggingface:huggingface.co/google/embeddinggemma-300m
    name: embeddinggemma
    params:
      hf_token: ${secrets:HUGGINGFACE_TOKEN}

Learn more about EmbeddingGemma in the official documentation.

POST /v1/search API Use Search Table Functions: The /v1/search API now uses the new text_search and vector_search Table Functions for improved performance.

Embedding Request Caching: The runtime now supports caching embedding requests, reducing latency and cost for repeated content and search requests.

Example spicepod.yml snippet:

runtime:
  caching:
    embeddings:
      enabled: true
      max_size: 128mb
      item_ttl: 5s

See the Caching documentation for details.

Real-Time Indexing for Full Text Search: Full Text search indexing is now supported for connectors that enable real-time changes, such as Debezium CDC streams. Adding a full-text index on a column with refresh_mode: changes works as it does for full/append-mode refreshes, enabling instant search on new data.

Example spicepod.yml snippet:

datasets:
  - from: debezium:cdc.public.question
    name: questions
    acceleration:
      enabled: true
      engine: duckdb
      primary_key: id
      refresh_mode: changes # Use 'changes'
    params: *kafka_params
    columns:
      - name: title
        full_text_search:
          enabled: true # Enable full-text-search indexing
          row_id:
            - id

OpenAI Responses API Tool Calls with Streaming: The OpenAI Responses API now supports tool calls with streaming, enabling advanced model interactions such as web_search and code_interpreter with real-time response streaming. This allows you to invoke OpenAI-hosted tools and receive results as they are generated.

Learn more in the OpenAI Model Provider documentation.

Runtime Output Level Configuration: You can now set the output_level parameter in the Spicepod runtime configuration to control logging verbosity in addition to the existing CLI and environment variable support. Supported values are info, verbose, and very_verbose. The value is applied in the following priority: CLI, environment variables, then YAML configuration.

Example spicepod.yml snippet:

runtime:
  output_level: info # or verbose, very_verbose

For more details on configuring output level, see the Troubleshooting documentation.

Bug Fixes

Several bugs and issues have been resolved in this release, including:

CDC Streams: Fixed issues where refresh_mode: changes could prevent the Spice runtime from becoming Ready, and improved support for full-text indexing on CDC streams.
Vector Search: Fixed bugs where vector search HTTP pipeline could not find more than one IndexedTableProvider, and resolved errors with field mismatches in vector_search UDTF.
Kafka Integration: Improved Kafka schema inference with configurable sample size, improved consumer group persistence for SQLite and Postgres accelerations, and added cooperative mode support.
Perplexity Web Search: Fixed bug where Perplexity web search sometimes used incorrect query schema (limit).
Databricks: Fixed issue with unparsing embedded columns.
Error Reporting: ThrottlingException is now reported correctly instead of as InternalError.
Iceberg Data Connector: Added support for LIMIT pushdown.
Amazon S3 Vectors: Fixed ingestion issues with zero-vectors and improved handling when vector index is full.
Tracing: Fixed vector search tracing to correctly report SQL status.

Contributors

New Contributors

@ChrisTomAlxHitachi made their first contribution in github.com/spiceai/spiceai/pull/6932 🎉

Breaking Changes

No breaking changes.

Cookbook Updates

New Spice with Dotnet SDK Recipe - The recipe shows how to query Spice using the Dotnet SDK.

The Spice Cookbook includes 78 recipes to help you get started with Spice quickly and easily.

Upgrading

To upgrade to v1.7.0, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.7.0 image:

docker pull spiceai/spiceai:1.7.0

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

AWS Marketplace:

🎉 Spice is now available in the AWS Marketplace!

What's Changed

Dependencies

Rust: Upgraded from 1.88.0 to 1.89.0
DataFusion: Upgraded from 48.0.1 to 49.0.0
text-embeddings-inference: Upgraded from 1.7.3 to 1.8.2
twox-hash: Upgraded from 1.6.3 to 2.1.0.

Changelog

Fix parameterised query planning in DataFusion by @Jeadie in #6942
fix: Update benchmark snapshots by @app/github-actions in #6944
refactor: Decouple full text search candidate from UDTF by @peasee in #6940
fix: Re-enable search integration tests by @peasee in #6930
Update acknowledgements and spicepod.schema.json by @sgrebnov in #6948
Add enabling the responses API by @lukekim in #6949
Post-release housekeeping by @sgrebnov in #6951
Add missing param in release notes by @Advayp in #6959
Create comprehensive S3vectors test by @Jeadie in #6903
Update ROADMAP after v1.6 release by @sgrebnov in #6955
Update openapi.json by @app/github-actions in #6961
Add build step for new spiced images in end game template by @Jeadie in #6960
refactor: Use text search UDTF in v1/search by @peasee in #6962
Bump Jimver/cuda-toolkit from 0.2.26 to 0.2.27 by @app/dependabot in #6922
Bump notify from 8.0.0 to 8.2.0 by @app/dependabot in #6924
Use model2vec for search integration tests for speed by @Jeadie in #6971
feat: Add initial DuckDB regexp pushdown support by @peasee in #6966
Bump rustyline from 16.0.0 to 17.0.1 by @app/dependabot in #6976
Upgrade delta_kernel to 0.14 by @phillipleblanc in #6977
Consistent snapshots for mongodb by @krinart in #6974
Bump indexmap from 2.10.0 to 2.11.0 by @app/dependabot in #6921
Fix mongo tests: ignore container_registry() when building image name by @krinart in #6983
Implement support for s3 tables for glue DataConnector by @krinart in #6981
Bump serde_json from 1.0.142 to 1.0.143 by @app/dependabot in #6925
Update build_and_release macOS pipeline to skip updating cmake if installed by @phillipleblanc in #6998
Mark Kafka Data Connector Alpha quality by @sgrebnov in #6991
Add v1.6.1 release notes by @lukekim in #7000
Spice CLI trace: make error friendlier when task_history is disabled by @sgrebnov in #6996
Warn when runtime or management is added in spicepod dependency by @Jeadie in #6953
Enable .datasets[].vectors.params.s3_vectors_distance_metric for S3 Vectors by @Jeadie in #6982
Add s3_vectors index support for CDC and Append streams by @sgrebnov in #6986
Find all vector indexes in v1/search by @Jeadie in #7004
Fix RRF; reorder by score by @Jeadie in #7007
Fix for nested VectorScanTableProvider by @krinart in #7017
Add --sql flag to output SQL query for spice trace by @Jeadie in #7002
Make web search params engine-specific by @Advayp in #7022
Add more MTEB benchmark spicepods by @peasee in #7026
Improve error messaging in tools by @Jeadie in #6895
Add retry for exporting task history records by @sgrebnov in #7049
Increase DoPut write timeout for the next batch from 30 to 120 seconds by @sgrebnov in #7054
Avoid redundant search embedding by @peasee in #7053
Truncate text_embed task_history trace by @sgrebnov in #7050
Use the UTC offset for the start_time and end_time fields in the task history by @ewgenius in #7056
Update supported versions in SECURITY.md by @Jeadie in #7060
Add integration test for Kafka S3 Vectors by @sgrebnov in #6988
Enable parameters to enforce the value is one of several options by @Jeadie in #6984
feat(iceberg): lakekeeper catalog - add warehouse param to spicepod by @ChrisTomAlxHitachi in #6932
feat: Add HTTP query concurrency support to testoperator by @peasee in #7025
Ensure no data does not throw error in v1/search by @Jeadie in #7033
Bump github.com/spf13/cobra from 1.9.1 to 1.10.1 by @app/dependabot in #7013
Add QA analytics for 1.6.x releases by @sgrebnov in #7082
Use env variable for HF cache in model2vec by @Jeadie in #7076
chore: upgrade to Rust 1.88 by @kczimm in #7077
Kafka/Debezium: make common errors user-friendlier by @sgrebnov in #7084
Create Apache Datafusion upgrade issue template by @kczimm in #6800
No join predicate pushdown on empty results by @Jeadie in #7075
Bump tract-onnx from 0.21.10 to 0.22.0 by @app/dependabot in #7071
Bump mongodb from 3.2.4 to 3.3.0 by @app/dependabot in #7073
Bump indicatif from 0.17.11 to 0.18.0 by @app/dependabot in #7070
Bump actions/github-script from 7 to 8 by @app/dependabot in #7069
Bump actions/setup-go from 5 to 6 by @app/dependabot in #7068
Bump actions/download-artifact from 4 to 5 by @app/dependabot in #7066
Bedrock: Tool use without inputs must empty Document by @Jeadie in #7036
Bump github.com/stretchr/testify from 1.10.0 to 1.11.1 by @app/dependabot in #7015
Bump actions/setup-python from 5 to 6 by @app/dependabot in #7067
Upgrade dependabot dependencies by @phillipleblanc in #7061
Bump tempfile from 3.20.0 to 3.21.0 by @app/dependabot in #7018
Only call 'list_datasets' once, after initial system/user messages by @Jeadie in #7039
Bump github.com/spf13/pflag from 1.0.7 to 1.0.10 by @app/dependabot in #7062
Bump actions/checkout from 4 to 5 by @app/dependabot in #7065
Bump golang.org/x/mod from 0.27.0 to 0.28.0 by @app/dependabot in #7064
Bump github.com/AzureAD/microsoft-authentication-library-for-go from 1.4.1 to 1.5.0 by @app/dependabot in #7063
Add friendly message for Kafka operation timeout error, improve code by @sgrebnov in #7088
embed UDF by @mach-kernel in #6967
fix: Update benchmark snapshots by @app/github-actions in #7097
Fix SF100 benchmark tests dispatch by @sgrebnov in #7098
chore(logging): add log when iceberg rest catalog fails with ssl cert error by @ChrisTomAlxHitachi in #6909
Add xxhash support for search/sql results by @krinart in #6978
Use proper federation in max_timestamp_df during acceleration refresh by @krinart in #7055
Fix spiced_docker workflows for new actions/download-artifact@v5 behavior by @phillipleblanc in #7108
Fix spiced_docker workflow by @phillipleblanc in #7111
Add filter for zero vectors before writing to S3 Vectors by @phillipleblanc in #7110
Ensure we find vector index when it also has text search by @Jeadie in #7120
Enable unified traceparent override support for HTTP API by @sgrebnov in #7122
Fix ORDER BY: (BytesProcessedExec to avoid pruning ordered execs during physical optimization) by @mach-kernel in #7105
Fix spiced_docker_nightly workflow by @sgrebnov in #7125
Add output_level to runtime config by @krinart in #7119
Add tests for xxhash hashers by @krinart in #7124
Add input option to update snapshots in Integration tests by @Jeadie in #7127
Fix formatting to improve merges by @lukekim in #7128
Add tests to nulling logic by @Jeadie in #7113
Bump chrono from 0.4.41 to 0.4.42 by @app/dependabot in #7131
Bump ctrlc from 3.4.7 to 3.5.0 by @app/dependabot in #7132
Search: RRF UDTF by @mach-kernel in #7090
Update openapi.json by @app/github-actions in #7141
Bump packages to DF49; resolve incompatibilities by @Jeadie in #7101
fix: Don't error for chunked columns when vectors are disabled by @peasee in #7150
Allow bzip2-1.0.6 license in deny.toml by @Jeadie in #7148
Tune retry settings for Kafka/Debezium connectors by @sgrebnov in #7142
Update TEI by @Jeadie in #7152
Use twox-hash version 2.1.2 by @krinart in #7165
Revert "Use proper federation in max_timestamp_df during acceleration refresh (#7055)" by @phillipleblanc in #7156
Bump octocrab from 0.44.1 to 0.45.0 by @app/dependabot in #7158
Bump github.com/spf13/viper from 1.19.0 to 1.21.0 by @app/dependabot in #7130
Bump keyring from 3.6.2 to 3.6.3 by @app/dependabot in #7157
fix: Remove keywords from AI document search by @peasee in #7052
Bump tract-core from 0.21.10 to 0.22.0 by @app/dependabot in #7134
Update TEI by @Jeadie in #7171
Update openapi.json by @app/github-actions in #7172
fix: Ensure vector search UDTF respects the supplied projection by @peasee in #7155
Bump clap from 4.5.45 to 4.5.47 by @app/dependabot in #7135
Bump golang.org/x/sys from 0.35.0 to 0.36.0 by @app/dependabot in #7129
Include 'catalog_id' in Glue catalog parameters by @Jeadie in #7151
fix: Use head ref from merge group event in pulls-with-spice concurrency group by @peasee in #7175
Fix lint for xxhash feature by @phillipleblanc in #7176
Add Kafka-specific metrics for consumer lag and consumed records by @sgrebnov in #7146
Kafka: persist consumer between restarts with SQLite and PG acceleration by @sgrebnov in #7177
Kafka: support specifying a target consumer group ID by @sgrebnov in #7178
Fix timestamp parsing for spice trace by @krinart in #7173
Support full-text indexing on CDC/append streams by @phillipleblanc in #7180
Bump iceberg-rust version to include limit push down by @krinart in #7191
Make full text stream connector more robust by @phillipleblanc in #7193
fix: Update benchmark snapshots by @app/github-actions in #7179
Initial changes for SearchIndex by @Jeadie in #7103
Robustly handle indexing FTS for CDC streams by @phillipleblanc in #7197
Proper handling/mapping for ThrottlingException during embedding calls by @krinart in #7170
Add spicepod.yml by @lukekim in #7202
Delta Lake: Support read pruning on timestamp columns using maxValues stats by @sgrebnov in #7203
feat: Add initial embeddings cache by @peasee in #7194
Make S3vector a FixedSizeListArray by @Jeadie in #7201
Fix projection mismatch issues with RRF calling vector search / text search by @mach-kernel in #7200
feat: Add embeddings cache to all embeddings by @peasee in #7204
Revert "Make S3vector a FixedSizeListArray (#7201)" by @kczimm in #7210
Update duckdb version to make ICU statically linked by default by @krinart in #7215
Change DataType list nullability from true to false by @Jeadie in #7216
Use Instant + saturating_sub to handle time drift by @krinart in #7212
Flatten 'IndexedTableProvider' when adding full-text support by @Jeadie in #7219
Include comments in pulls by @lukekim in #7224
Add github_max_concurrent_connections = 5 by @lukekim in #7225
RRF: Fix scoring by @mach-kernel in #7226
Update RRF search integration snapshots after scoring change by @mach-kernel in #7227
Make S3vector a FixedSizeListArray by @Jeadie in #7230
Proper federation during acceleration refresh + datafusion version bump + integration tests by @krinart in #7228
Use DuckDBDialect for DuckDB non-federated queries by @krinart in #7232
Move chunking out of llms and into new crate chunking by @Jeadie in #7229
Remove duplicate pg_port configuration in test by @lukekim in #7233
Upgrade to Rust 1.89 by @phillipleblanc in #7235
Catalog connection error: fix connector name from 'iceberg' to 'spice.ai' by @sgrebnov in #7240
Create PutVectorsSink by @kczimm in #7199
Benchmark tests: fix API key reference in spicecloud catalog by @sgrebnov in #7239
Add Dotnet SDK sample to end game template by @sgrebnov in #7238
Update spicepod.schema.json by @app/github-actions in #7254
Postgres: Improve Decimals read performance and add Name type support by @sgrebnov in #7255
Add tests for hybrid search on a vector engine by @Jeadie in #7220

Spice v0.18.3-beta (Sep 30, 2024)

September 30, 2024 · 5 min read

Jack Eadie

Token Plumber at Spice AI

Announcing the release of Spice v0.18.3-beta 🛠️

The Spice v0.18.3-beta release includes several quality-of-life improvements including verbosity flags for spiced and the Spice CLI, vector search over larger documents with support for chunking dataset embeddings, and multiple performance enhancements. Additionally, the release includes several bug fixes, dependency updates, and optimizations, including updated table providers and significantly improved GitHub data connector performance for issues and pull requests.

Highlights in v0.18.3-beta

GitHub Query Mode: A new github_query_mode: search parameter has been added to the GitHub Data Connector, which uses the GitHub Search API to enable faster and more efficient query of issues and pull requests when using filters.

Example spicepod.yml:

- from: github:github.com/spiceai/spiceai/issues/trunk
  name: spiceai.issues
  params:
    github_query_mode: search # Use GitHub Search API
    github_token: ${secrets:GITHUB_TOKEN}

Output Verbosity: Higher verbosity output levels can be specified through flags for both spiced and the Spice CLI.

Example command line:

spice -v
spice --very-verbose

spiced -vv
spiced --verbose

Embedding Chunking: Chunking can be enabled and configured to preprocess input data before generating dataset embeddings. This improves the relevance and precision for larger pieces of content.

Example spicepod.yml:

- name: support_tickets
  embeddings:
    - column: conversation_history
      use: openai_embeddings
      chunking:
        enabled: true
        target_chunk_size: 128
        overlap_size: 16
        trim_whitespace: true

For details, see the Search Documentation.

Dependencies

DataFusion Table Providers: Upgraded to rev b0af91992699ecbf5adf2036a07122578f06150e.

Contributors

@Sevenannn
@peasee
@Jeadie
@sgrebnov
@phillipleblanc
@ewgenius
@slyons

What's Changed

- Update datafusion table provider patch by @Sevenannn in https://github.com/spiceai/spiceai/pull/2817
- refactor: Set max_rows_per_batch for ODBC to 4000 by @peasee in https://github.com/spiceai/spiceai/pull/2822
- Use User message for health check by @Jeadie in https://github.com/spiceai/spiceai/pull/2823
- Upgrade Helm chart (Spice v0.18.2-beta) by @sgrebnov in https://github.com/spiceai/spiceai/pull/2820
- Add verbosity flags for spiced, spice: `-v`, `-vv`, `--verbose`, `--very-verbose`. by @Jeadie in https://github.com/spiceai/spiceai/pull/2831
- Rename `spiceai` data connector to `spice.ai` by @sgrebnov in https://github.com/spiceai/spiceai/pull/2680
- Prepare for v0.19.0-beta release (version bump) by @sgrebnov in https://github.com/spiceai/spiceai/pull/2821
- Bump clap from 4.5.17 to 4.5.18 (#2801) by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2848
- Enable "rc" feature for serde in spicepod crate by @ewgenius in https://github.com/spiceai/spiceai/pull/2851
- Update spicepod.schema.json by @github-actions in https://github.com/spiceai/spiceai/pull/2852
- chore: update table providers by @peasee in https://github.com/spiceai/spiceai/pull/2858
- fix: Use GitHub search for issues in GraphQL by @peasee in https://github.com/spiceai/spiceai/pull/2845
- fix: Use GitHub search for pull_requests by @peasee in https://github.com/spiceai/spiceai/pull/2847
- Support chunking dataset embeddings by @Jeadie in https://github.com/spiceai/spiceai/pull/2854
- refactor: Update GraphQL client to be more robust for filter push down by @peasee in https://github.com/spiceai/spiceai/pull/2864
- docs: Update accelerator beta criteria by @peasee in https://github.com/spiceai/spiceai/pull/2865
- Change `BytesProcessedRule` to be an optimizer rather than an analyzer rule by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2867
- Don't run E2E or PR tests on documentation by @Jeadie in https://github.com/spiceai/spiceai/pull/2869
- Verify benchmark query results using snapshot testing (spice.ai connector) by @sgrebnov in https://github.com/spiceai/spiceai/pull/2866
- feat: Add GraphQLOptimizer by @peasee in https://github.com/spiceai/spiceai/pull/2868
- Update quickstarts for Endgame by @Jeadie in https://github.com/spiceai/spiceai/pull/2863
- Update version to v0.18.3-beta by @sgrebnov in https://github.com/spiceai/spiceai/pull/2882
- Update DataFusion: fix coalesce, Aggregation with Window functions unparsing support by @sgrebnov in https://github.com/spiceai/spiceai/pull/2884
- Revert "Rename `spiceai` data connector to `spice.ai`" by @sgrebnov in https://github.com/spiceai/spiceai/pull/2881
- Adding integration test for DuckDB read functions by @slyons in https://github.com/spiceai/spiceai/pull/2857
- Show more informative mysql error message by @Sevenannn in https://github.com/spiceai/spiceai/pull/2883
- Fix `no process-level CryptoProvider available` when using REPL and TLS by @sgrebnov in https://github.com/spiceai/spiceai/pull/2887
- Change UX for chunking and enable overlap_size in chunking by @Jeadie in https://github.com/spiceai/spiceai/pull/2890
- Add `log/slog` to spice CLI tool by @Jeadie in https://github.com/spiceai/spiceai/pull/2859
- feat: Add GitHub GraphQLOptimizer by @peasee in https://github.com/spiceai/spiceai/pull/2870
- Fix mysql invalid tablename error message by @Sevenannn in https://github.com/spiceai/spiceai/pull/2896
- fix: Remove login column rename in pulls and update Optimizer by @peasee in https://github.com/spiceai/spiceai/pull/2897
- Fix require check checking. by @Jeadie in https://github.com/spiceai/spiceai/pull/2898

**Full Changelog**: https://github.com/spiceai/spiceai/compare/v0.18.2-beta...v0.18.3-beta

Resources

Community

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Twitter: @spice_ai
Slack: spiceai.org/slack
Telegram: Spice AI Discussion
Reddit: https://www.reddit.com/r/spiceai
Email: hey@spice.ai

What's New in v2.2.0​

Cloud Connect​

MySQL Change Data Capture​

PostgreSQL Catalog CDC​

Debezium CDC Without Kafka​

Faster Search with Warm In-Memory Indexes​

Operations & Observability​

Other Notable Improvements and Bug Fixes​

SDK Updates​

Dependency Updates​

New Contributors​

Contributors​

Breaking Changes​

Cookbook Updates​

Upgrading​

What's Changed​

Changelog​

What's New in v2.1.0​

High-Throughput Cayenne CDC​

Change Data Capture & HTAP​

Distributed Query​

Performance & Query Engine​

AI & LLM​

Search & Vectors​

SQL & Query Engine​

Security & Connectors​

Adaptive Self-Tuning (Experimental)​

Observability​

Notable Bug Fixes​

Dependency Updates​

Contributors​

Breaking Changes​

Cookbook Updates​

Upgrading​

What's Changed​

Changelog​

What's New in v2.0.0-rc.4​

Elasticsearch Data Connector (Alpha, Spice.ai Enterprise)​

PostgreSQL Native Replication via WAL​

Multi-vector Embeddings with MaxSim (Late Interaction)​

Rerank UDTF for Hybrid Search​

New Secret Stores: HashiCorp Vault and Azure Key Vault​

DuckDB Vector Engine​

New and Promoted Connectors​

DynamoDB Write Support (DML)​

MCP Streamable HTTP Transport​

Security Improvements​

Developer Experience Improvements​

OpenTelemetry Improvements​

Full-text Search Performance​

SQL and Query Engine​

Dependency Updates​

Contributors​

Breaking Changes​

Cookbook Updates​

Upgrading​

What's Changed​

Changelog​

What's New in v1.8.2​

Support Table Relations in /v1/search HTTP Endpoint​

DuckDB Data Accelerator Table Partitioning & Indexing​

S3 Vectors Reliability​

Document Table Improvements​

Additional Improvements & Bugfixes​

Contributors​

Breaking Changes​

Cookbook Updates​

Upgrading​

What's Changed​

Changelog​

What's New in v1.7.1​

Bug Fixes & Improvements​

Contributors​

Breaking Changes​

Cookbook Updates​

Upgrading​

What's Changed​

Changelog​

What's New in v1.7.0​

DataFusion v49 Highlights​

What's New in v2.2.0

Cloud Connect

MySQL Change Data Capture

PostgreSQL Catalog CDC

Debezium CDC Without Kafka

Faster Search with Warm In-Memory Indexes

Operations & Observability

Other Notable Improvements and Bug Fixes

SDK Updates

Dependency Updates

New Contributors

Contributors

Breaking Changes

Cookbook Updates

Upgrading

What's Changed

Changelog

What's New in v2.1.0

High-Throughput Cayenne CDC

Change Data Capture & HTAP

Distributed Query

Performance & Query Engine

AI & LLM

Search & Vectors

SQL & Query Engine

Security & Connectors

Adaptive Self-Tuning (Experimental)

Observability

Notable Bug Fixes

Dependency Updates

Contributors

Breaking Changes

Cookbook Updates

Upgrading

What's Changed

Changelog

What's New in v2.0.0-rc.4

Elasticsearch Data Connector (Alpha, Spice.ai Enterprise)

PostgreSQL Native Replication via WAL

Multi-vector Embeddings with MaxSim (Late Interaction)

Rerank UDTF for Hybrid Search

New Secret Stores: HashiCorp Vault and Azure Key Vault

DuckDB Vector Engine

New and Promoted Connectors

DynamoDB Write Support (DML)

MCP Streamable HTTP Transport

Security Improvements

Developer Experience Improvements

OpenTelemetry Improvements

Full-text Search Performance

SQL and Query Engine

Dependency Updates

Contributors

Breaking Changes

Cookbook Updates

Upgrading

What's Changed

Changelog

What's New in v1.8.2

Support Table Relations in `/v1/search` HTTP Endpoint

DuckDB Data Accelerator Table Partitioning & Indexing

S3 Vectors Reliability

Document Table Improvements

Additional Improvements & Bugfixes

Contributors

Breaking Changes

Cookbook Updates

Upgrading

What's Changed

Changelog

What's New in v1.7.1

Bug Fixes & Improvements

Contributors

Breaking Changes

Cookbook Updates

Upgrading

What's Changed

Changelog

What's New in v1.7.0

DataFusion v49 Highlights