Skip to main content

8 posts tagged with "snowflake"

Snowflake Data Connector related topics and usage

View All Tags

Spice v2.0-rc.5 (May 27, 2026)

· 30 min read
Jack Eadie
Jack Eadie
Token Plumber at Spice AI

Spice v2.0-rc.5 is now available! 🔥

v2.0.0-rc.5 is the fifth release candidate for advanced testing of v2.0, building on v2.0.0-rc.4.

This release completes the mTLS implementation across server endpoints and outbound connectors, adds MongoDB Change Streams and durable Kafka offset persistence as new CDC sources, expands DML write-back to PostgreSQL, Snowflake, and Arrow, promotes DuckLake to Beta, introduces user-defined functions, on-demand dataset loading, unified query cancellation, dynamic HTTP request headers and subquery-driven request parameters, provider-aware LLM prompt caching, and a long list of Cayenne performance improvements.

Highlights in this release candidate include:

  • Spice Cayenne — CDC throughput, compaction and scan caching, synchronized partition commits, join filter propagation, parallel Vortex writes, lock-free deletion caches
  • Mutual TLS (mTLS) — TLS cert hot-reload, public mTLS for HTTP and Flight (channel + identity modes), mTLS client certs for FlightSQL and Spice.ai connectors
  • MongoDB Change Streams — native real-time CDC for MongoDB, no Debezium or Kafka required
  • Kafka CDC offsets — offsets persisted in sidecar tables for durable, resumable Kafka CDC
  • PostgreSQL DML — INSERT, UPDATE, DELETE write-back on PostgreSQL datasets
  • Snowflake DML — INSERT, UPDATE, DELETE write-back on Snowflake datasets
  • Arrow Primary Key Upserts — native upsert path using primary key matching
  • DuckLake promoted to Beta — with INSERT support on catalog tables
  • User-Defined Functions — define SQL UDFs in spicepods, plus remote UDFs over HTTP (Spice.ai Enterprise)
  • Spatial SQL UDFs — optional geospatial UDFs (ST_*) for geometry workloads
  • On-Demand Dataset Loading — datasets can be deferred and loaded on first reference
  • Unified Query Cancellation — Ctrl-C and HTTP request cancellation propagate across all execution paths
  • Dynamic HTTP Connector — pass-through request headers, subquery-driven params, and JSON schema decomposition
  • HTTP Rate-Control persistence — rate-limit state persisted in object storage across restarts
  • refresh_mode: snapshot — point-in-time snapshot acceleration with SQLite/Turso WAL flushing
  • Storage-profile accelerator tuning — accelerators auto-tune defaults based on local SSD, EBS-class disk, or tmpfs
  • Provider-Aware LLM Prompt Caching — automatic prompt caching for OpenAI-compatible providers that support it
  • Responses API — support across all model providers with streaming response.output_text.delta, plus Authorization: Bearer header support

What's New in v2.0.0-rc.5​

Cayenne Improvements​

Significant performance work across Spice Cayenne-backed catalogs and accelerators.

  • Ingest throughput: End-to-end improvements to CDC ingest, background compaction, and a new scan-result cache for hot reads; parallel Vortex partition writes; lock-free deletion caches with bloom-prefiltered probes; background retention with CDC pipelining; SQLite metastore pool scaled to 32 for high-concurrency mutation workloads.
  • Data inlining: Small writes are serialized as Arrow IPC and committed directly into the Cayenne metastore (cayenne_inlined_data), bypassing the staged Vortex write path for low-latency ingest. Inline upserts atomically rewrite existing inline rows instead of emitting side delete markers, and inline data remains query-visible via an in-memory union scan with a generation-keyed decode cache. Inline rows are checkpointed to Vortex when row, segment, or byte thresholds are reached. Defaults are refresh-mode aware: inline writes are enabled by default for high-frequency caching, changes, and fast append workloads and disabled for full, snapshot, and slower append.
  • Query planning: Join filter propagation across equi-join keys (gated behind runtime.params.cayenne_filter_propagation), range fallback for large join filters, hot-path clone elimination, and IN-list rewrites for large filter lists.
  • Correctness: Synchronized partition commits across partitions, correct NULL-sentinel handling for nullable partition expressions (e.g. bucket(N, col)), Vortex panic fix on highly compressible data, and live reads through expired protected snapshots.
  • Catalog and platform: Refresh-mode-aware compaction defaults, rejection of non-distributed Cayenne catalog configurations, and a vendored Vortex DataFusion integration for faster iteration on the Cayenne planner.

Mutual TLS (mTLS)​

Spice.ai Enterprise feature. See Enterprise Security.

Spice now supports full mutual TLS for both HTTP and Arrow Flight endpoints.

TLS cert hot-reload (#10727): The Spice runtime watches for SIGHUP and reloads TLS certificates without restarting, enabling cert rotation with zero downtime.

Public mTLS for HTTP and Flight (#10753): Two client_auth_mode values control how the server handles client certificates:

  • request — optional mTLS: the server requests a client cert but accepts connections without one (useful for migration windows).
  • required — strict mTLS: the server requires a valid client cert signed by the configured CA.

mTLS client certs for FlightSQL and Spice.ai connectors (#10764): Outbound connections from the FlightSQL and Spice.ai data connectors can now present client certificates for mutual authentication with upstream services.

Example configuration:

runtime:
  tls:
    enabled: true
    certificate_file: /etc/spice/tls/server.crt
    key_file: /etc/spice/tls/server.key
    client_auth_mode: required
    client_auth_ca_file: /etc/spice/tls/client-ca.crt

MongoDB Change Streams​

MongoDB datasets configured with refresh_mode: changes now stream changes from MongoDB Change Streams into any local accelerator (#10813), providing real-time CDC without Debezium or Kafka.

Example configuration:

datasets:
  - from: mongodb:my_collection
    name: my_collection
    params:
      host: my-cluster.mongodb.net
      db: mydb
    acceleration:
      enabled: true
      engine: duckdb
      refresh_mode: changes

CDC Improvements​

See Change Data Capture (CDC) for an overview of CDC in Spice.

  • Kafka CDC offset persistence (#10823): Kafka CDC offsets are persisted in sidecar tables for durable, resumable streams. On restart or failover, Spice resumes from the last committed offset.
  • Pipelined CDC ingestion (#10676): Source reads overlap with batch apply, with additional batching, envelope coalescing, and nullability propagation improvements across the apply pipeline.
  • Debezium schema evolution fix (#10144): Schema changes in Debezium-sourced datasets no longer break dataset initialization on reload (fixes #9782).

PostgreSQL DML Support​

The PostgreSQL data connector now supports write-back via INSERT, UPDATE, and DELETE operations (#10446). Combined with the existing read-side federation, PostgreSQL-backed datasets can serve as full read/write tables. The PostgreSQL Catalog connector additionally exposes foreign-key metadata for NSQL and query planning (#10849).

Snowflake DML Support​

The Snowflake data connector now supports write-back via INSERT, UPDATE, and DELETE operations (#10747), complementing its existing read capabilities.

Arrow Primary Key Upserts​

Arrow-accelerated tables now support native upsert operations using primary key matching (#10749), providing efficient update-or-insert semantics for in-memory datasets.

DuckLake Promoted to Beta​

The DuckLake Catalog and Data Connector are promoted to Beta quality (#10743).

DuckLake catalog tables with read_write access now support INSERT operations (#10744), enabling full read/write workflows against DuckLake-backed catalogs. The DuckLake connector also gains a series of correctness fixes for downcast, module registration, schema discovery, and S3 credentials (#10650).

User-Defined Functions​

Spice now supports user-defined functions (UDFs) as a first-class spicepod component (#10571), letting you define reusable SQL functions in the spicepod or invoke remote functions over HTTP. The runtime also gains table user functions with HTTP server gating (#10675).

A security fix closes a remote-UDF SSRF vector (#10757).

Spatial SQL UDFs​

Spice now ships an optional set of geospatial SQL UDFs (ST_*) for geometry workloads (#10833). The functions are gated behind a build feature and can be invoked from any SQL surface.

On-Demand Dataset Loading​

Datasets can now be marked for on-demand loading (#10629). Deferred datasets are registered with a declared schema at startup (#10669) and only fully resolve when first referenced, reducing startup time and memory footprint for spicepods with many seldom-used datasets.

Spicepods also gain columns[].type and columns[].nullable (#10661) with a lenient type parser for declaring schemas inline.

Unified Query Cancellation​

All query execution paths — HTTP, Flight, FlightSQL, MCP, and internal — now honour a unified cancellation signal (#10390). When a client disconnects, presses Ctrl-C in the REPL, or cancels an in-flight HTTP request, the corresponding query is cancelled end-to-end, freeing resources promptly.

Dynamic HTTP Connector​

The HTTP data connector gains dynamic request headers parameterised from query predicates (#10604), subquery-driven request parameters for fan-out queries (#10636), HTTP response metadata as queryable columns via JSON schema decomposition (#10679), no-limit pagination (#10673), and shared rate-control across HTTP-based connectors using the same backend host (#10648).

HTTP Rate-Control Persistence​

The HTTP rate-control state (per-endpoint throttle counters) is now persisted in object storage (#10697), ensuring rate limits survive restarts and are consistent across replicas. Rate-control metrics now use an origin label rather than the connector name for cleaner aggregation (#10689).

The metrics HTTP endpoint (/metrics) is also independently rate-limited (#10162) to prevent scraping from impacting query serving.

refresh_mode: snapshot​

Spice.ai Enterprise feature. See Acceleration Snapshots.

A new refresh_mode: snapshot provides point-in-time snapshot acceleration (#10651), with SQLite and Turso WAL flushing and a Cayenne metastore slice integration so accelerated readers see a consistent snapshot while writes continue.

Storage-Profile Accelerator Tuning​

Acceleration configs gain a new storage_profile field (#10913) with values auto (default), local_ssd, ebs, and tmpfs. Under auto, the runtime detects whether the acceleration store is backed by local SSD, EBS-class network disk, or tmpfs, and applies storage-aware defaults across DuckDB, partitioned DuckDB, SQLite, Turso, and Cayenne file-mode accelerators. Explicit per-accelerator parameters always override the profile defaults.

Provider-Aware LLM Prompt Caching​

LLM calls automatically use provider-aware prompt caching (#10645) when the configured model provider supports it (e.g., Anthropic, OpenAI). System prompts and tool descriptions are marked for caching so repeated invocations within the cache window reuse the provider-side cached prefix, reducing latency and cost.

A new searchable registry mode for LLM tools (#10647) lets agents discover tools by semantic search rather than enumerating all tools in the system prompt, which scales to large tool inventories.

Responses API Improvements​

The Responses API is now supported across all configured model providers (#10724). Streaming delta events via response.output_text.delta are also supported (#10828). The runtime now also accepts Authorization: Bearer headers in addition to x-api-key, bumps async-openai, and stops populating FunctionToolCall.id so OpenAI-compatible servers can assign the ID themselves (#10911).

Distributed Cluster Improvements​

Spice.ai Enterprise feature. See High Availability.

  • Per-request executor readiness gate (#10860): /v1/ready on schedulers waits for a configurable quorum of executors before returning healthy, enabling proper rolling deployments.
  • Ballista S3 shuffle reads under cluster mode (#10910): The shuffle reader builds its S3 client from the executor pod's environment, matching the writer. Async queries with runtime.params.shuffle_location: s3://... now complete instead of failing with AccessDenied on shuffle fetches.
  • Flattened scheduler config (#10450): runtime.scheduler.partition_management.* fields are flattened directly onto runtime.scheduler and renamed under the canonical "partition assignment" terminology. See Breaking Changes.

Caching & Search​

Improvements across Caching and Search:

  • Per-principal cache namespacing (#10702): SQL, search, and caching-accelerator caches are now namespaced per authenticated principal, so cached results never cross identity boundaries.
  • DuckDB HNSW vector indexes (#10695, #10674, #10668): DuckDB-accelerated views support HNSW vector indexes for vector search, vector search SQL is rewritten to activate HNSW_INDEX_SCAN, and HNSW indexes are preserved across data refresh.

Security Improvements​

See Authentication and TLS for configuring Spice security.

  • API key timing-position leak and remote-UDF SSRF (#10757): Closed a timing-based position-disclosure leak in API key comparison and blocked SSRF via remote UDF endpoint parameters.
  • Configurable allowed_hosts for MCP (#10638): MCP servers can be restricted to an explicit allowlist of upstream hosts.

SQL, Query, and Developer Experience​

See the SQL Reference for the full SQL surface area.

  • SQL REPL expanded view (#10797): Toggle \x in the REPL for a vertical key-value layout on wide result sets.
  • FlightSQL Substrait plan support (#10761): The Spice runtime now implements CommandStatementSubstraitPlan, enabling clients that submit plans as Substrait-encoded protobuf.
  • MCP auth for streamable HTTP tools (#10927): Streamable HTTP MCP tools support native authentication via mcp_auth_token and mcp_headers, both with full Spice secret expansion.
  • Elasticsearch FTS engine config and index lifecycle (#10672): Direct FTS engine configuration plus index lifecycle and ingestion controls for the Elasticsearch connector.
  • Self-hosted Spice connector (#10546): Connect Spice to another self-hosted Spice runtime as a federated source.

Connector Bug Fixes​

Notable correctness fixes across the Data Connectors: DynamoDB Streams retry on transient errors (#10794) and typed-NULL handling in DML (#10511); ScyllaDB physical filter pushdown disabled to fix incorrect results (#10772); MSSQL TOP N pushdown for non-nullable sort columns (#10621); DuckLake include filter applied (#10738); DuckDB DELETE/UPDATE on full and caching refresh modes (#10632); checked arithmetic for Turso integer-millis and timestamp-to-nanosecond conversions (#10786, #10666); and Flight GetFlightInfo/DoGet schema parity (#10864). See the Changelog for the full list.

Dependency Updates​

Dependency / ComponentVersion
DuckDBv1.5.2
Icebergv0.9.1
Tursov0.6.0
Vortexv0.69.0

Contributors​

Breaking Changes​

Flattened runtime.scheduler configuration (#10450): The nested runtime.scheduler.partition_management block has been flattened and renamed to use the canonical "partition assignment" terminology. Migrate as follows:

# Before
runtime:
  scheduler:
    partition_management:
      interval: 30s
      max_assignments_per_cycle: 16
      discovery_timeout: 10s

# After
runtime:
  scheduler:
    partition_assignment_interval: 30s
    max_assignments_per_interval: 16
    partition_discovery_timeout: 10s

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.0.0-rc.5, use one of the following methods:

CLI:

spice upgrade v2.0.0-rc.5

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.0.0-rc.5 image:

docker pull spiceai/spiceai:2.0.0-rc.5

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.0.0-rc.5

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

  • Enable DML support for PostgreSQL data connector by @phillipleblanc in #10446
  • feat(postgres): support inline PEM sslrootcert by @claudespice in #10578
  • Add foreign key metadata discovery to PostgreSQL Catalog by @sgrebnov in #10849
  • Add Snowflake DML support by @lukekim in #10747
  • Add MongoDB Change Streams support by @lukekim in #10813
  • Add user-defined functions by @lukekim in #10571
  • Add table user functions and gate HTTP servers by @lukekim in #10675
  • feat: add on-demand dataset loading by @phillipleblanc in #10629
  • feat(runtime): declared-schema deferred datasets by @phillipleblanc in #10669
  • feat(spicepod, runtime): add columns[].type / nullable + lenient type parser by @phillipleblanc in #10661
  • Replace external smb crate with internal SMB 3.1.1 client by @phillipleblanc in #10516
  • Add unified query cancellation across all paths by @lukekim in #10390
  • Add dynamic HTTP request headers by @lukekim in #10604
  • feat(http): Support dynamic HTTP connector request params from subqueries by @lukekim in #10636
  • feat(http): pass through HTTP metadata columns with JSON schema decomposition by @lukekim in #10679
  • Add nolimit HTTP pagination max pages by @lukekim in #10673
  • Add shared HTTP rate control for connectors by @lukekim in #10648
  • Use origin label instead of name for HTTP rate control metrics by @lukekim in #10689
  • fix(http): reject OR across different HTTP filter columns by @lukekim in #10625
  • Add provider-aware LLM prompt caching by @lukekim in #10645
  • Add searchable registry mode for LLM tools by @lukekim in #10647
  • feat: refresh_mode: snapshot + SQLite/Turso WAL flush + Cayenne metastore slice by @phillipleblanc in #10651
  • feat: per-principal cache namespacing for SQL/search/caching-accelerator by @lukekim in #10702
  • Add self-hosted Spice connector support by @phillipleblanc in #10546
  • Add Delta Lake Azure tenant parameter by @phillipleblanc in #10671
  • Support OAuth2 client credentials in 'spice cloud login' by @ewgenius in #10586
  • Add configurable allowed_hosts for MCP by @lukekim in #10638
  • fix: make Helm chart probes configurable by @peasee in #10696
  • Strip high-cardinality datasets dim from anonymous telemetry by @lukekim in #10711
  • feat(elasticsearch): direct FTS engine config + index lifecycle and ingestion controls by @lukekim in #10672
  • Add DuckDB HNSW vector index support for accelerated views by @sgrebnov in #10695
  • Rewrite DuckDB vector search SQL to activate HNSW_INDEX_SCAN by @sgrebnov in #10674
  • Fix DuckDB HNSW vector indexes lost after data refresh by @sgrebnov in #10668
  • Fix DuckDB DELETE/UPDATE on full and caching refresh mode datasets by @phillipleblanc in #10632
  • Fix DuckLake connector: downcast, module registration, schema discovery, and S3 credentials by @sgrebnov in #10650
  • Fix federation pushing denied functions inside subqueries to remote engines by @phillipleblanc in #10692
  • fix(caching): honour refresh_on_startup: always in caching mode by @phillipleblanc in #10594
  • fix(iceberg): rebuild storage factory when Hadoop catalog scheme is inferred by @sgrebnov in #10601
  • Pipeline CDC ingestion: overlap source reads with batch apply by @lukekim in #10676
  • fix: add NULL check to CDC primary key extraction by @lukekim in #10684
  • Properly handle nullability during CDC processing by @krinart in #10803
  • Flatten scheduler config and rename partition management → partition assignment by @lukekim in #10450
  • Improve NSQL UX and harden internal LLM tools by @lukekim in #10715
  • Support Responses API across model providers by @lukekim in #10724
  • Update xAI default model and handle Grok model retirements by @Jeadie in #10723
  • Improve cli table layout by @krinart in #10725
  • TLS cert hot-reload (mTLS plan M1) by @phillipleblanc in #10727
  • Fix DuckLake catalog include filter being ignored by @phillipleblanc in #10738
  • Promote DuckLake Catalog and Data Connector to Beta quality by @sgrebnov in #10743
  • feat(ducklake): Support INSERT on catalog tables with read_write access by @sgrebnov in #10744
  • perf(cdc): coalesce envelopes and overlap commits in apply pipeline by @lukekim in #10745
  • feat: Allow full version tags in spicepod version by @peasee in #10748
  • Add Arrow primary key upserts by @lukekim in #10749
  • fix(snapshot): keep refresh_mode snapshot read-only by @phillipleblanc in #10752
  • feat(tls): public mTLS for HTTP and Flight (channel + identity modes) by @phillipleblanc in #10753
  • perf(cayenne): lock-free deletion caches with bloom-prefiltered probe by @lukekim in #10756
  • fix(security): close API key timing-position leak and remote-UDF SSRF by @lukekim in #10757
  • Fix 'wait_until_dependent_tables_are_ready' for catalogs by @phillipleblanc in #10758
  • Fixes for views and resolved tables on 'spice refresh' CLI by @phillipleblanc in #10759
  • Implement FlightSQL CommandStatementSubstraitPlan support by @lukekim in #10761
  • feat(connectors): mTLS client cert support for flightsql and spiceai connectors by @phillipleblanc in #10764
  • Allow arbitrary filenames when specifying spicepod path + kind validation by @krinart in #10777
  • fix: ignore field metadata in schema compatibility check in index_table_scan by @Jeadie in #10778
  • Display pushed-down limits in EXPLAIN TREE output by @lukekim in #10779
  • fix: enable streaming append for Kafka with Cayenne accelerator by @lukekim in #10780
  • fix: bound chunked-index intermediate batch size to prevent OOM by @phillipleblanc in #10783
  • fix: label all columns in spice cloud metrics table output by @claudespice in #10784
  • fix: use checked arithmetic for Turso integer-millis timestamp read path by @claudespice in #10786
  • fix: use checked arithmetic in timestamp-to-nanosecond conversions by @claudespice in #10666
  • Upgrade to DuckDB v1.5.2 by @sgrebnov in #10788
  • Improve CDC ingestion performance by @lukekim in #10789
  • Fix tool_search/tool_invoke spans by @lukekim in #10791
  • Add Cayenne inline mutations and benchmark coverage by @lukekim in #10792
  • Ensure we always resolve table names in distributed mode/metadata by @Jeadie in #10793
  • Remove permanent errors from DynamoDB Streams by @krinart in #10794
  • Add expanded view mode for wide table display in SQL REPL by @lukekim in #10797
  • Fix Cayenne CDC schema mismatch error by @sgrebnov in #10800
  • Executors should create catalog tables on join by @Jeadie in #10807
  • Add compressed file support for listing connectors by @lukekim in #10809
  • Improve Cayenne mutation, scan, and inline memtable scaling by @lukekim in #10811
  • Add range fallback for large join filters by @lukekim in #10816
  • Improve Cayenne join filter pushdown by @lukekim in #10818
  • Synchronize Cayenne partition commits across partitions by @phillipleblanc in #10819
  • fix: Deny nondistributed cayenne catalog by @peasee in #10821
  • Enable parallel Cayenne Vortex writes by @lukekim in #10822
  • Expand Arrow type handling in formatting and Elasticsearch by @lukekim in #10825
  • Add response.output_text.delta to responses API by @krinart in #10828
  • feat(cayenne): add join filter propagation and no-spill Q21 planning by @lukekim in #10840
  • Upgrade Turso to v0.6.0 by @sgrebnov in #10843
  • feat(cli): add spice feedback command to open community Slack by @lukekim in #10856
  • Upgrade iceberg to v0.9.1 by @sgrebnov in #10859
  • feat(cluster): per-request executor readiness gate on /v1/ready by @phillipleblanc in #10860
  • fix: Require dim-side statistics for CayennePropagateFilterAcrossEquiJoinKeys by @sgrebnov in #10863
  • fix: Debezium schema evolution breaks dataset init on reload by @claudespice in #10144
  • fix(mssql): Push topK limit to SQL Server for non-nullable sort columns by @Jeadie in #10621
  • fix(ScyllaDB): disable physical filter pushdown by @sgrebnov in #10772
  • fix: handle typed NULLs and prevent overflow in DynamoDB DML type conversions by @krinart in #10511
  • fix: use InsertOp::Overwrite in DynamoDB bootstrap scan_and_overwrite_accelerator by @krinart in #10639
  • Improve DynamoDB Bootstrap performance by @krinart in #10616
  • fix: preserve field and schema metadata in Vortex type transformation by @lukekim in #10628
  • fix: GH connector - explicitly use AWS LC RS crypto provider for jwt by @phillipleblanc in #10619
  • fix: add snapshot mode guards to delete_from/update and delegate DML in SwappableTableProvider by @phillipleblanc in #10685
  • Persist HTTP rate-control state in object storage by @lukekim in #10697
  • Rate limit metrics HTTP endpoint by @lukekim in #10162
  • feat(geo): add optional spatial SQL UDF support by @lukekim in #10833
  • feat(cayenne): CDC throughput, compaction, scan caching, and benchmarks by @lukekim in #10852
  • fix(cayenne): fix Vortex panic on highly compressible data by @sgrebnov in #10855
  • fix(cayenne): Read live protected snapshots after cleanup grace period by @sgrebnov in #10901
  • fix: Disable Cayenne HashJoin rewriter optimizer by @sgrebnov in #10882
  • Fix GetFlightInfo vs DoGet Flight Schema by @krinart in #10864
  • fix(search): preserve column casing in /v1/search primary key plumbing by @claudespice in #10909
  • fix(object-store): dedupe s3 url style auto-detection log by @phillipleblanc in #10898
  • Improve Spice CLI manifest editing and direct command modes by @lukekim in #10815
  • Persist Kafka CDC offsets in sidecar tables by @lukekim in #10823
  • feat(task-history): record Ballista stages for distributed queries by @phillipleblanc in #10831
  • Add '#[deny(clippy::missing_trait_methods)]' to wrapper/delegation trait impls by @Jeadie in #10795
  • Optimize Cayenne catalog maintenance paths by @lukekim in #10904
  • Centralize DuckDB settings for accelerator by @ewgenius in #10895
  • deps(ballista): bump to 47e2b494 to fix S3 shuffle reads under cluster mode by @phillipleblanc in #10910
  • Authorization header + Bump async-openai + responses_adapter fix by @krinart in #10911
  • Tune accelerators by storage profile by @lukekim in #10913
  • feat: add dataset-level on_schema_change config by @lukekim in #10908
  • Handle NULL sentinel for nullable partition expressions by @Jeadie in #10880
  • fix: Remove Cayenne Catalog from catalog registration by @peasee in #10914
  • Add catalog name to foreign key metadata in postgres catalog by @Jeadie in #10917
  • Cayenne perf: eliminate redundant clones, PK point-lookup fanout fix, IN-list rewrite + microbench coverage by @lukekim in #10916
  • fix(turso-shared): retry on Turso BEGIN CONCURRENT "Write-write conflict" by @lukekim in #10946
  • Vendor Vortex DataFusion for Cayenne by @lukekim in #10933
  • perf(cayenne): background retention + enable CDC pipelining for retention-configured tables by @lukekim in #10936
  • feat(cayenne): scale metastore pool to 32 + vs_duckdb_scaling benches (1→128 concurrency, sqlite + turso lanes) by @lukekim in #10943
  • feat(mcp): support auth for streamable HTTP tools by @phillipleblanc in #10927
  • Explicit error if v1/search requests a table without search index by @Jeadie in #10968
  • Fix spicepod loading failure when directory name contains dots by @sgrebnov in #10958
  • Extend append tests with arrow engine configurations by @sgrebnov in #10959
  • Remove dataset on_schema_change Policy from rc.5 release notes by @sgrebnov in #10964
  • Skip tpcds_q78 for Cayenne engine at SF100 by @sgrebnov in #10966
  • fix: Update benchmark snapshots May-20 by @app/github-actions in #10952
  • Fix #10951: UdtfExec invariant Vec lengths must match children count by @phillipleblanc in #10953
  • docs(release): update v2.0.0-rc.5 notes with latest trunk PRs by @lukekim in #10949
  • Remove eval related things for v2.0.0 by @Jeadie in #10945
  • build(deps): bump ubuntu from 24.04 to 26.04 in the docker-dependencies group by @app/dependabot in #10883
  • fix: Add publish = false to chbench-driver by @sgrebnov in #10939

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.0.0-rc.4...v2.0.0-rc.5

Spice v2.0-rc.3 (Apr 21, 2026)

· 13 min read
Evgenii Khramkov
Evgenii Khramkov
Senior Software Engineer at Spice AI

Announcing the release of Spice v2.0-rc.3! ⚡

v2.0.0-rc.3 is the third release candidate for advanced testing of v2.0, building on v2.0.0-rc.2.

Highlights in this release candidate include:

  • HTTP Connector Enhancements with OAuth2 refresh-token authentication, query-parameter pagination, and map-to-array conversion for broader API compatibility
  • Databricks and Unity Catalog Reliability Improvements with resilience controls, improved UC-awareness, permission checks, and structured error reporting
  • Snowflake and ADBC Registration Performance Improvements with better observability during dataset registration
  • OpenTelemetry Exporter Improvements with exporter fixes and support for authenticated metrics export
  • Kafka, GitHub, and HTTP Connector Fixes including Kafka reliability improvements, GitHub GraphQL resilience updates, and HTTP JSON union/reload fixes

What's New in v2.0.0-rc.3​

HTTP Connector Enhancements​

The HTTP connector now supports more authentication and API response patterns, making it easier to integrate with modern REST APIs.

Key improvements:

  • OAuth2 Refresh-Token Authentication: Added support for OAuth2 refresh-token flows for APIs that issue short-lived access tokens.
  • Query-Parameter Pagination: Added pagination support using query parameters for APIs that expose page or cursor controls in the URL.
  • Map-to-Array Conversion: Added response transformation support for APIs that return map-shaped payloads that need to be normalized into arrays.
  • Improved JSON Union Handling: Better handling for heterogeneous JSON payloads during ingestion.
  • More Reliable Reloads: Fixed runtime behavior for HTTP-backed datasets during spicepod reloads.

Example configuration of an HTTP connector using the OAuth2 refresh token flow:

datasets:
  - from: https://api.example.com
    name: secure_data
    params:
      file_format: json
      allowed_request_paths: '/v1/**'
      auth_token_url: https://auth.example.com/oauth/token
      http_auth_refresh_token: ${secrets:my_refresh_token}
      http_auth_client_id: ${secrets:my_client_id}
      http_auth_client_secret: ${secrets:my_client_secret}
      auth_scopes: 'read:data offline_access'

Databricks and Unity Catalog Reliability Improvements​

Databricks and Unity Catalog integrations are now more resilient and provide clearer behavior in permission-constrained environments.

Key improvements:

  • Resilience Controls: Added controls to improve reliability when interacting with Databricks services.
  • Unity Catalog Awareness: Improved handling for Unity Catalog-specific behaviors and mixed deployment configurations.
  • Permission Prechecks: Databricks UC permission checks now distinguish explicit denials from ambiguous cases.
  • Structured Error Reporting: Advisory permission failures now surface with more actionable structured errors.
  • Classic SQL Warehouse Compatibility: Improved handling for foreign table scenarios with Classic SQL Warehouse combinations.
  • Task History Instrumentation: Added instrumentation to improve observability for Databricks-related operations.

Snowflake and ADBC Improvements​

Snowflake and ADBC-backed dataset registration is now faster and easier to observe.

Key improvements:

  • Faster Dataset Registration: Improved registration performance for Snowflake and ADBC datasets.
  • Better Observability: Added better instrumentation and visibility into registration workflows.
  • ADBC Alignment: Updated ADBC dependencies and integration points for improved compatibility.
  • Search Schema Fix: Fixed a full-text search schema mismatch issue with the ADBC connector.

OpenTelemetry and Observability Improvements​

Spice improves telemetry export reliability and authenticated metrics delivery for the OpenTelemetry integration.

Key improvements:

  • OTEL Exporter Fixes: Fixed issues in the OpenTelemetry exporter.
  • Authenticated Metrics Export: Added support for authorization headers in the OTEL metrics exporter.
  • Reduced Startup Noise: Suppressed unnecessary AWS SDK noise and improved OTEL-related initialization behavior.
  • Connector Initialization Reliability: Fixed issues that could block connector initialization in telemetry-related code paths.

Dependency and Toolchain Updates​

Dependency / ComponentVersion / Update
Rust toolchainv1.94.1 (from v1.93.1)
DataFusionv52.5.0-rc1
mistral.rs / candlemistral.rs v0.8.x, candle v0.10.1
ADBC Corev0.23

Other Improvements​

  • Improved Query Pushdown: Expanded sort and limit pushdown, including improved pushdown behavior for Oracle and MSSQL connectors.
  • Partitioned Query Planning Improvements: Improved PartitionedTableScanRewrite handling for ORDER BY, partition expressions, and fully qualified table references, while preventing incorrect bucketing partition pushdown to executors.
  • MongoDB SRV Support: Upgraded datafusion-table-providers with MongoDB SRV support.
  • Tantivy Logging: Search logging now defaults to warn unless very verbose logging is enabled.
  • Kafka Connector Fixes: Improved reliability for the Kafka connector behavior.
  • GitHub Connector Resilience: Improved commit fetching for dynamic and slash refs, and reduced GraphQL page sizes on gateway errors for the GitHub connector.
  • GitHub API Efficiency: Lowered default comment fetch sizes to reduce pressure on GitHub GraphQL APIs.
  • Embedding Validation: Added validation for embedding row_id columns during dataset initialization.
  • View Cache Invalidation: Cached plans are now cleared when views are updated.
  • Refresh SQL Dedup Fix: Fixed append refresh deduplication when refresh_sql selects a subset of columns.

Contributors​

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook includes 86 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v2.0.0-rc.3, use one of the following methods:

CLI:

spice upgrade v2.0.0-rc.3

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:2.0.0-rc.3 image:

docker pull spiceai/spiceai:2.0.0-rc.3

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai --version 2.0.0-rc.3

AWS Marketplace:

Spice is available in the AWS Marketplace.

What's Changed​

Changelog​

  • fix: Full Text Search schema mismatch with ADBC connector by @lukekim in #10235
  • docs: Update v2.0.0-rc.2 release notes with latest changes by @lukekim in #10238
  • Fix append refresh dedup failure when refresh_sql selects column subset by @sgrebnov in #10225
  • Revert "Properly mark dataset as Ready on Scheduler (#10215)" by @sgrebnov in #10242
  • Fix failing merge conflicts for benchmarks by @krinart in #10247
  • fix(github): fetch commits for dynamic and slash refs by @lukekim in #10233
  • Upgrade DataFusion to v52.5.0-rc1 by @lukekim in #10249
  • Merge develop to trunk (2026-04-09) by @claudespice in #10248
  • fix: Validate embedding row_id columns during dataset init (fixes #8226) by @claudespice in #10208
  • fix: Update tpch benchmark snapshots for federated/glue[csv].yaml by @app/github-actions in #10244
  • feat(databricks): add resilience controls, UC awareness, and task history instrumentation by @lukekim in #10246
  • fix: Make PartitionManager resilient to bare vs fully qualified table references by @sgrebnov in #10257
  • fix: Update tpch benchmark snapshots for accelerated/s3[parquet]-cayenne[file].yaml by @app/github-actions in #10256
  • Merge develop to trunk (2026-04-10) by @claudespice in #10251
  • Improve Snowflake/ADBC dataset registration performance and observability by @lukekim in #10266
  • Fixes for kafka connector by @krinart in #10263
  • fix(runtime): gate otel code tags, suppress aws sdk noise, and unblock connector init by @lukekim in #10260
  • fix(runtime): avoid regionless AWS SDK loads by @lukekim in #10271
  • Add versioned release install workflow coverage by @lukekim in #10276
  • fix(runtime): handle HTTP JSON unions and spicepod reloads by @lukekim in #10277
  • Databricks UC permission prechecks: explicit denial as permanent error, ambiguous cases advisory by @lukekim in #10274
  • Revert component status changes re-introduced by develop merge (#10248) by @sgrebnov in #10293
  • Fix broken CI workflows by @ewgenius in #10294
  • Group dependabot updates by ecosystem by @lukekim in #10296
  • fix(tests): Replace flaky S3 Vectors snapshot tests with structural validation by @lukekim in #10301
  • Update test_github_workflows snapshot by @lukekim in #10304
  • fix(ci): fix Bedrock runner mismatch and snapshot auto-merge failure by @ewgenius in #10306
  • feat(http): Add map-to-array conversion and query-parameter pagination by @lukekim in #10295
  • New crate: datafusion-ddl by @Jeadie in #10205
  • Make Databricks UC permission checks advisory with structured error reporting by @lukekim in #10283
  • build(deps): bump the github-actions-dependencies group with 4 updates by @app/dependabot in #10298
  • fix: Clear cached plans on view updates by @peasee in #10312
  • build(deps): bump the aws-sdk group with 7 updates by @app/dependabot in #10299
  • Code out of runtime. by @Jeadie in #10178
  • fix: Respect function registry denies for accelerated table filter pushdown by @peasee in #10311
  • fix: Don't block heartbeat when all slots acquired by @peasee in #10322
  • fix: strip only outer parens in get_table_partition_expr_from_ctx by @Jeadie in #10323
  • Upgrade datafusion-table-providers with MongoDB SRV support by @lukekim in #10317
  • fix: Avoid pushing down bucketing partition expressions into executors by @peasee in #10324
  • Upgrade datafusion-table-providers to d1b911a5 and bump adbc to 0.23 by @lukekim in #10329
  • fix: Update Search integration test snapshots by @app/github-actions in #10308
  • Handle foreign table + Classic sql warehouse combination gracefully by @krinart in #10318
  • New crate datafusion-flightsql by @Jeadie in #10201
  • Set tantivy=warn unless very verbose logging by @Jeadie in #10338
  • Remove image registry and image name options from spidapter by @ewgenius in #10241
  • build(deps): bump sysinfo from 0.37.2 to 0.38.4 by @app/dependabot in #10291
  • build(deps): bump futures from 0.3.31 to 0.3.32 by @app/dependabot in #10289
  • New crate 'datafusion-dml' by @Jeadie in #10334
  • Jeadie/26 04 16/spice sql by @Jeadie in #10343
  • Add Teraswitch/Pittsburgh apt mirrors + retry config for CI runners by @lukekim in #10349
  • Implement sort pushdown and fix pushdown gaps across providers by @lukekim in #10337
  • Merge develop to trunk (2026-04-16) by @claudespice in #10345
  • Update candle and mistral.rs lock-step pins by @lukekim in #10278
  • docs: fix status badges in README by @lukekim in #10350
  • Migrate secrets to vars by @krinart in #10354
  • Add limit pushdown and improve sort pushdown for Oracle and MSSQL by @sgrebnov in #10351
  • Fix ubuntu mirror configuration by @ewgenius in #10359
  • fix: Increase throughput test default ready_wait from 30s to 300s (fixes #8207) by @claudespice in #10344
  • Add auth headers support to OTEL metrics exporter by @lukekim in #10347
  • fix(github): shrink GraphQL page size on gateway errors; lower comment defaults by @lukekim in #10355
  • Relax apt mirror substitution failure to warning in CI action by @ewgenius in #10361
  • feat(http): Add OAuth2 refresh-token auth to HTTP connector by @lukekim in #10348
  • Upgrade Rust toolchain to 1.94.1 by @lukekim in #10353
  • Handle order by and sort in PartitionedTableScanRewrite by @Jeadie in #9656
  • Fix OTEL Exporter by @krinart in #10363
  • Pin spiceai candle / TEI forks to merged revs; drop local [patch] overrides by @lukekim in #10362

Full Changelog: https://github.com/spiceai/spiceai/compare/v2.0.0-rc.2...v2.0.0-rc.3

Spice v1.3.2 (June 2, 2025)

· 2 min read
Phillip LeBlanc
Phillip LeBlanc
Co-Founder and CTO of Spice AI

Announcing the release of Spice v1.3.2! ❄️

Spice v1.3.2 is a patch release with fixes to the DuckDB data accelerator and Snowflake data connector.

Changes:

  • DuckDB Data Accelerator: Supports ORDER BY rand() for randomized result ordering and ORDER BY NULL for SQL compatibility.

  • Snowflake Data Connector: Adds TIMESTAMP_NTZ(0) type for timestamps with seconds precision.

Contributors​

Breaking Changes​

No breaking changes.

Cookbook Updates​

No new cookbook recipes.

The Spice Cookbook now includes 67 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.3.2, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.3.2 image:

docker pull spiceai/spiceai:1.3.2

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

What's Changed​

Dependencies​

No major dependency changes.

Changelog​

  • Handle Snowflake Timestamp NTZ with seconds precision (#6084) by @kczimm in #6084
  • Fix DuckDB acceleration ORDER BY rand() and ORDER BY NULL (#6071) by @phillipleblanc in #6071

Full Changelog: https://github.com/spiceai/spiceai/compare/v1.3.1...v1.3.2

Spice v1.1.2 (Apr 14, 2025)

· 4 min read
Phillip LeBlanc
Phillip LeBlanc
Co-Founder and CTO of Spice AI

Announcing the release of Spice v1.1.2! 🔗

Spice v1.1.2 improves Delta Lake Data Connector performance, introduces new Accept headers for the /v1/sql and /v1/nsql endpoints to include query metadata with results, and resolves an issue with the Snowflake Data Connector when handling wide tables (>600 columns).

The official Tableau Connector for Spice.ai v0.1 has been released, making it easy to connect to both self-hosted Spice.ai and Spice Cloud instances using Tableau.

What's New in v1.1.2​

Query Spice.ai using Tableau

  • Delta Lake Data Connector: Upgraded delta_kernel to v0.9, and optimized scan operations, reducing query execution time by up to 20% on large datasets.

  • Snowflake Data Connector: Fixed a bug that caused failures when loading tables with more than 600 columns.

  • Query Metadata (SQL and NSQL): Added support for the application/vnd.spiceai.sql.v1+json Accept header on the /v1/sql endpoint, and the application/vnd.spiceai.nsql.v1+json Accept header on the /v1/nsql endpoint, enabling responses to include metadata such as the executed SQL query and schema alongside results.

Example:

curl -XPOST "http://localhost:8090/v1/nsql" \
  -H "Content-Type: application/json" \
  -H "Accept: application/vnd.spiceai.nsql.v1+json" \
  -d '{
    "query": "What’s the highest tip any passenger gave?"
  }' | jq

Example response:

{
  "row_count": 1,
  "schema": {
    "fields": [
      {
        "name": "highest_tip",
        "data_type": "Float64",
        "nullable": true,
        "dict_id": 0,
        "dict_is_ordered": false,
        "metadata": {}
      }
    ],
    "metadata": {}
  },
  "data": [
    {
      "highest_tip": 428.0
    }
  ],
  "sql": "SELECT MAX(\"tip_amount\") AS \"highest_tip\"\nFROM \"spice\".\"public\".\"taxi_trips\""
}

For details, see the SQL Query API and NSQL API documentation.

Contributors​

Breaking Changes​

No breaking changes in this release.

Cookbook Updates​

No major cookbook additions.

The Spice Cookbook now includes 65 recipes to help you get started with Spice quickly and easily.

Upgrading​

To upgrade to v1.1.2, use one of the following methods:

CLI:

spice upgrade

Homebrew:

brew upgrade spiceai/spiceai/spice

Docker:

Pull the spiceai/spiceai:1.1.2 image:

docker pull spiceai/spiceai:1.1.2

For available tags, see DockerHub.

Helm:

helm repo update
helm upgrade spiceai spiceai/spiceai

What's Changed​

Dependencies​

Changelog​

  • Backport - Fix schema inference for Snowflake tables with large number of columns #5348 by @ewgenius in #5350
  • Upgrade delta_kernel to 0.9 (#5343) by @phillipleblanc in #5356
  • Add basic support for application/vnd.spiceai.sql.v1+json format (#5333) by @sgrebnov in #5333
  • Convert DataFusion filters to Delta Kernel predicates by @phillipleblanc in #5362
  • revert to previous pdf-extract; remove test for encrypted pdf support by @kczimm in #5355
  • Turn off delta_kernel::log_segment logging and refactor log filtering by @phillipleblanc in #5367
  • Extend application/vnd.spiceai.sql.v1+json with schema and row_count fields by @sgrebnov in #5365
  • Make separate vnd.spiceai.sql.v1+json and vnd.spiceai.nsql.v1+json MIME types by @sgrebnov in #5382

Full Changelog: v1.1.1...v1.1.2

Spice v1.0-rc.5 (Jan 13, 2025)

· 9 min read
Evgenii Khramkov
Evgenii Khramkov
Senior Software Engineer at Spice AI

Announcing the release of Spice v1.0-rc.5 🛠️

Spice v1.0.0-rc.5 is the fifth release candidate for the first major version of Spice.ai OSS. This release focuses production readiness and critical bug fixes. In addition, a new DynamoDB data connector has been added along with automatic detection for GPU acceleration when running Spice using the CLI.

Highlights in v1.0-rc.5​

  • Automatic GPU Acceleration Detection: Automatically detect and utilize GPU acceleration when running by CLI. Install AI components locally using the CLI command spice install ai. Currently supports NVIdia CUDA and Apple Metal (M-series).

  • DynamoDB Data Connector: Query AWS DynamoDB tables using SQL with the new DynamoDB Data Connector.

datasets:
  - from: dynamodb:users
    name: users
    params:
      dynamodb_aws_region: us-west-2
      dynamodb_aws_access_key_id: ${secrets:aws_access_key_id}
      dynamodb_aws_secret_access_key: ${secrets:aws_secret_access_key}
    acceleration:
      enabled: true
sql> describe users;
+----------------+-----------+-------------+
| column_name    | data_type | is_nullable |
+----------------+-----------+-------------+
| created_at     | Utf8      | YES         |
| date_of_birth  | Utf8      | YES         |
| email          | Utf8      | YES         |
| account_status | Utf8      | YES         |
| updated_at     | Utf8      | YES         |
| full_name      | Utf8      | YES         |
| ...                                      |
+----------------+-----------+-------------+

  • File Data Connector: Graduated to Stable.

  • Dremio Data Connector: Graduated to Release Candidate (RC).

  • Spice.ai, Spark, and Snowflake Data Connectors: Graduated to Beta.

Dependencies​

No major dependency changes.

Contributors​

  • @Jeadie
  • @phillipleblanc
  • @ewgenius
  • @peasee
  • @Sevenannn
  • @lukekim

What's Changed​

* Update acknowledgements by @github-actions in https://github.com/spiceai/spiceai/pull/4190
* Ensure non-nullity of primary keys in `MemTable`; check validity of initial data. by @Jeadie in https://github.com/spiceai/spiceai/pull/4158
* Bump version to v1.0.0 stable by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4191
* Fix metal + models download by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4193
* Update spice.ai connector beta roadmap by @ewgenius in https://github.com/spiceai/spiceai/pull/4194
* feat: verify on zero results snapshots by @peasee in https://github.com/spiceai/spiceai/pull/4195
* Add throughput test module to `test-framework` by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4196
* Update Spice.ai TPCH snapshots by @ewgenius in https://github.com/spiceai/spiceai/pull/4202
* Replace all usage of `lazy_static!` with `LazyLock` by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4199
* Fix model + metal download by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4200
* Run Clickbench for Dremio by @Sevenannn in https://github.com/spiceai/spiceai/pull/4138
* Update openapi.json by @github-actions in https://github.com/spiceai/spiceai/pull/4205
* Fix the typo in connector stable criteria by @Sevenannn in https://github.com/spiceai/spiceai/pull/4213
* feat: Add throughput test example by @peasee in https://github.com/spiceai/spiceai/pull/4214
* feat: calculate throughput test query percentiles by @peasee in https://github.com/spiceai/spiceai/pull/4215
* feat: Add throughput test to actions by @peasee in https://github.com/spiceai/spiceai/pull/4217
* Implement DynamoDB Data Connector by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4218
* 1.0 doc updates by @lukekim in https://github.com/spiceai/spiceai/pull/4181
* Improve clarity and concison of use-cases by @lukekim in https://github.com/spiceai/spiceai/pull/4220
* Remove macOS Intel build by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4221
* fix: Test operator throughput test workflow by @peasee in https://github.com/spiceai/spiceai/pull/4222
* DynamoDB: Automatically load AWS credentials from IAM roles if access key not provided by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4226
* File connector clickbench snapshots results by @ewgenius in https://github.com/spiceai/spiceai/pull/4225
* Spice.ai Catalog Connector by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4204
* feat: Add test framework metrics collection by @peasee in https://github.com/spiceai/spiceai/pull/4227
* Add badges for build/test status on README.md by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4228
* Release Dremio to RC by @Sevenannn in https://github.com/spiceai/spiceai/pull/4224
* feat: Add more test spicepods by @peasee in https://github.com/spiceai/spiceai/pull/4229
* feat: Add load test to testoperator by @peasee in https://github.com/spiceai/spiceai/pull/4231
* Add TSV format to all `object_store`-based connectors by @Jeadie in https://github.com/spiceai/spiceai/pull/4192
* Move test-framework to dev-dependencies for Runtime by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4230
* Document limitation for correlated subqueries in TPCH for Spice.ai connector by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4235
* Changes for CUDA by @Jeadie in https://github.com/spiceai/spiceai/pull/4130
* fix: Collect batches from test framework, load test updates by @peasee in https://github.com/spiceai/spiceai/pull/4234
* Suppress opentelemetry_sdk warnings - they aren't useful by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4243
* fix: Set dataset status first, update test framework by @peasee in https://github.com/spiceai/spiceai/pull/4244
* feat: Re-enable defaults on test spicepods by @peasee in https://github.com/spiceai/spiceai/pull/4248
* Add usage for streaming local models; Fix spice chat usage bar TPS expansion by @Jeadie in https://github.com/spiceai/spiceai/pull/4232
* refactor: Use composite testoperator setup, add query overrides by @peasee in https://github.com/spiceai/spiceai/pull/4246
* Enable expand_views_at_output for DF optimizer and transform schema to expanded view types by @ewgenius in https://github.com/spiceai/spiceai/pull/4237
* Add throughput test spicepod for databricks delta mode connector by @Sevenannn in https://github.com/spiceai/spiceai/pull/4241
* Spark data connector - update and enable TPCH and TPCDS benchmarks by @ewgenius in https://github.com/spiceai/spiceai/pull/4240
* Increase the timeout minutes of load test to 10 hours by @Sevenannn in https://github.com/spiceai/spiceai/pull/4254
* Improve partition column counts error for delta table by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4247
* Add e2e test for databricks catalog connector (mode: delta_lake) by @Sevenannn in https://github.com/spiceai/spiceai/pull/4255
* Spark connector integration tests by @ewgenius in https://github.com/spiceai/spiceai/pull/4256
* Run benchmark test with the new test framework by @Sevenannn in https://github.com/spiceai/spiceai/pull/4245
* Configure databricks delta secrets to run load test by @Sevenannn in https://github.com/spiceai/spiceai/pull/4257
* Support `properties` for emitted telemetry by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4249
* feat: Add `ready_wait` test operator workflow input by @peasee in https://github.com/spiceai/spiceai/pull/4259
* Handle 'LargeStringArray' for embedding tables by @Jeadie in https://github.com/spiceai/spiceai/pull/4263
* `llms` tests for alpha/beta model criteria  by @Jeadie in https://github.com/spiceai/spiceai/pull/4261
* Configurable runner type for load and throughput tests by @ewgenius in https://github.com/spiceai/spiceai/pull/4262
* Handle NULL partition columns for Delta Lake tables by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4264
* Add integration test for Snowflake by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4266
* Add Snowflake TPCH queries by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4268
* Handle `LargeStringArray` in `v1/search`. by @Jeadie in https://github.com/spiceai/spiceai/pull/4265
* Fix `build_cuda` in Update spiced_docker.yml by @Jeadie in https://github.com/spiceai/spiceai/pull/4269
* Run Snowflake benchmark in GitHub Actions by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4270
* Allow Snowflake query override for CI tests by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4271
* Don't run GPU builds for trunk by @Jeadie in https://github.com/spiceai/spiceai/pull/4272
* Fix InvalidTypeAction not working by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4273
* Add xAI key to llm integration tests by @Jeadie in https://github.com/spiceai/spiceai/pull/4274
* Update openai snapshots by @Jeadie in https://github.com/spiceai/spiceai/pull/4275
* Fix federation bug for correlated subqueries by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4276
* Update end_game.md by @ewgenius in https://github.com/spiceai/spiceai/pull/4278
* Promote Snowflake to Beta by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4277
* Set version to 1.0.0-rc.5 by @ewgenius in https://github.com/spiceai/spiceai/pull/4283
* Update cargo.lock by @ewgenius in https://github.com/spiceai/spiceai/pull/4285
* Update spice.ai data connector snapshots by @ewgenius in https://github.com/spiceai/spiceai/pull/4281
* Promote the Spice.ai Data Connector to Beta by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4282
* Revert change to  `integration_models__models__search__openai_chunking_response.snap` by @Jeadie in https://github.com/spiceai/spiceai/pull/4279
* Allow for a subset of build artifacts to be published to minio by @Jeadie in https://github.com/spiceai/spiceai/pull/4280
* Promote File Data Connector to Stable by @ewgenius in https://github.com/spiceai/spiceai/pull/4286
* Add Iceberg to Supported Catalogs by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4287
* Update openapi.json by @github-actions in https://github.com/spiceai/spiceai/pull/4289
* Fix Spark benchmark credentials, add back overrides by @ewgenius in https://github.com/spiceai/spiceai/pull/4295
* Promote Spark Data Connector to Beta by @ewgenius in https://github.com/spiceai/spiceai/pull/4296
* Add Dremio throughput test spicepod by @Sevenannn in https://github.com/spiceai/spiceai/pull/4233
* Add error message for invalid databricks mode parameter by @Sevenannn in https://github.com/spiceai/spiceai/pull/4299
* Fix pre-release check to look for `build` string by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4300
* Promote databricks catalog connector (mode: delta_lake) to beta by @Sevenannn in https://github.com/spiceai/spiceai/pull/4301
* Properly delegate `load_table` to Rest Catalog by @phillipleblanc in https://github.com/spiceai/spiceai/pull/4303
* Update acknowledgements by @github-actions in https://github.com/spiceai/spiceai/pull/4302
* docs: Update ROADMAP.md by @peasee in https://github.com/spiceai/spiceai/pull/4306
* v1.0.0-rc.5 Release Notes by @ewgenius in https://github.com/spiceai/spiceai/pull/4298

**Full Changelog**: https://github.com/spiceai/spiceai/compare/v1.0.0-rc.4...v1.0.0-rc.5

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Spice v0.17.4-beta (Sep 9, 2024)

· 5 min read
Luke Kim
Luke Kim
Founder and CEO of Spice AI

Announcing the release of Spice v0.17.4-beta.

The v0.17.4-beta release adds compatibility, performance, and reliability improvements to the DuckDB and SQLite accelerators. The GitHub data connector adds a Stargazers table, Snowflake and Clickhouse data connectors have improved resiliency for empty tables, and core data processing and quality has been improved.

Highlights in v0.17.4-beta​

Improved benchmarking, testing, and robustness of data accelerators: Continued compatibility, performance, and reliability improvements for SQLite and DuckDB data accelerators and expanded performance and quality testing.

GitHub Stargazers: The GitHub Data Connector adds support for a /stargazers table making it easy to query GitHub Stargazers using SQL!

Breaking Changes​

None.

Contributors​

  • @phillipleblanc
  • @Jeadie
  • @lukekim
  • @sgrebnov
  • @peasee
  • @eltociear
  • @Sevenannn
  • @ewgenius

New Contributors​

What's Changed​

- Change to sql lang by @ewgenius in https://github.com/spiceai/spiceai/pull/2484
- Update acknowledgements by @github-actions in https://github.com/spiceai/spiceai/pull/2487
- Bump rustyline from 13.0.0 to 14.0.0 by @dependabot in https://github.com/spiceai/spiceai/pull/2473
- Update spicepod.schema.json by @github-actions in https://github.com/spiceai/spiceai/pull/2490
- Native schema inference for snowflake (and support timezone_tz, better numeric support) by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2493
- Add checks for GitHub quickstart and docs banner to endgame template by @ewgenius in https://github.com/spiceai/spiceai/pull/2489
- Prepare for v0.18.0-beta by @Jeadie in https://github.com/spiceai/spiceai/pull/2488
- Add logo to README.md by @lukekim in https://github.com/spiceai/spiceai/pull/2497
- Add stargazers to GitHub data connector by @lukekim in https://github.com/spiceai/spiceai/pull/2502
- Enable federation for accelerated queries (sqlite and duckdb) by @sgrebnov in https://github.com/spiceai/spiceai/pull/2511
- Load SQLite decimal extension by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2498
- fix: Support INTERVAL in SQLite by @peasee in https://github.com/spiceai/spiceai/pull/2513
- Add refresh jitter to refreshing dataset acceleration by @Jeadie in https://github.com/spiceai/spiceai/pull/2510
- Update to use DuckDB streaming by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2514
- Add more MySQL types in E2E testing by @sgrebnov in https://github.com/spiceai/spiceai/pull/2512
- Update tpc loading script to support automatic loading into postgres by @Sevenannn in https://github.com/spiceai/spiceai/pull/2509
- docs: update README.md by @eltociear in https://github.com/spiceai/spiceai/pull/2516
- Bump quinn-proto from 0.11.6 to 0.11.8 in the cargo group by @dependabot in https://github.com/spiceai/spiceai/pull/2501
- Script for loading clickbench data into arrow / postgres and run clickbench queries by @Sevenannn in https://github.com/spiceai/spiceai/pull/2500
- Fix run query script to correctly record all errors by @Sevenannn in https://github.com/spiceai/spiceai/pull/2529
- Add support for DuckDB engine to setup-tpc-spicepod.bash by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2530
- Upgrade `datafusion` (fixes subquery alias table unparsing for SQLite) by @sgrebnov in https://github.com/spiceai/spiceai/pull/2532
- Make dataset acceleration delay `period +- jitter` by @Jeadie in https://github.com/spiceai/spiceai/pull/2534
- Add refresh options to `POST /v1/datasets/:name/acceleration/refresh` by @Jeadie in https://github.com/spiceai/spiceai/pull/2515
- Add E2E for GitHub Connector by @sgrebnov in https://github.com/spiceai/spiceai/pull/2505
- Add on-conflict integration test for file based and memory based sqlite by @Sevenannn in https://github.com/spiceai/spiceai/pull/2533
- Upgrade to Rust v.1.81.0 and fix resulting compile error by @Sevenannn in https://github.com/spiceai/spiceai/pull/2539
- Remove unneeded `RwLock` from `EmbeddingModelStore` by @Jeadie in https://github.com/spiceai/spiceai/pull/2541
- Remove unneeded RwLock from LlmModelStore by @Jeadie in https://github.com/spiceai/spiceai/pull/2537
- Add sqlite to the setup tpc benchmark script by @Sevenannn in https://github.com/spiceai/spiceai/pull/2540
- Add sqlite to setup clickbench script by @Sevenannn in https://github.com/spiceai/spiceai/pull/2548
- Update version for v0.17.4-beta release by @ewgenius in https://github.com/spiceai/spiceai/pull/2563
- Sharepoint data connector by @Jeadie in https://github.com/spiceai/spiceai/pull/2294
- Fix predicate/projection push-down for BytesProcessedNode by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2564
- fix out of order projections in sharepoint scans by @Jeadie in https://github.com/spiceai/spiceai/pull/2569
- Use Decimal instead of Float64 for SQLite Decimal columns by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2566
- Add snapshot tests for EXPLAIN plans in integration tests by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2570
- Set refresh.period from `refresh_data_window` by @ewgenius in https://github.com/spiceai/spiceai/pull/2578
- Add snapshot tests for EXPLAIN plans in benchmark tests by @phillipleblanc in https://github.com/spiceai/spiceai/pull/2580
- Disable federation for accelerated queries by @sgrebnov in https://github.com/spiceai/spiceai/pull/2581
- Add manual refresh payload to 'spice refresh...' by @Jeadie in https://github.com/spiceai/spiceai/pull/2565
- Update acknowledgements by @github-actions in https://github.com/spiceai/spiceai/pull/2586

**Full Changelog**: https://github.com/spiceai/spiceai/compare/v0.17.3-beta...v0.17.4-beta

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Spice v0.12.2-alpha (May 13, 2024)

· 5 min read
Sergei Grebnov
Sergei Grebnov
Senior Software Engineer at Spice AI

The v0.12.2-alpha release introduces data streaming and key-pair authentication for the Snowflake data connector, enables general append mode data refreshes for time-series data, improves connectivity error messages, adds nested folders support for the S3 data connector, and exposes nodeSelector and affinity keys in the Helm chart for better Kubernetes management.

Highlights​

  • Improved Connectivity Error Messages: Error messages provide clearer, actionable guidance for misconfigured settings or unreachable data connectors.

  • Snowflake Data Connector Improvements: Enables data streaming by default and adds support for key-pair authentication in addition to passwords.

  • API for Refresh SQL Updates: Update dataset Refresh SQL via API.

  • Append Data Refresh: Append mode data refreshes for time-series data are now supported for all data connectors. Specify a dataset time_column with refresh_mode: append to only fetch data more recent than the latest local data.

  • Docker Image Update: The spiceai/spiceai:latest Docker image now includes the ODBC data connector. For a smaller footprint, use spiceai/spiceai:latest-slim.

  • Helm Chart Improvements: nodeSelector and affinity keys are now supported in the Helm chart for improved Kubernetes deployment management.

Breaking Changes​

  • API to trigger accelerated dataset refreshes has changed from POST /v1/datasets/:name/refresh to POST /v1/datasets/:name/acceleration/refresh to be consistent with the Spicepod.yaml structure.

Contributors​

  • @mach-kernel
  • @y-f-u
  • @sgrebnov
  • @ewgenius
  • @Jeadie
  • @Sevenannn
  • @digadeesh
  • @phillipleblanc
  • @lukekim

What's Changed​

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.12.1-alpha...v0.12.2-alpha

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Spice v0.12.1-alpha (May 6, 2024)

· 5 min read
Phillip LeBlanc
Phillip LeBlanc
Co-Founder and CTO of Spice AI

The v0.12.1-alpha release introduces a new Snowflake data connector, support for UUID and TimestampTZ types in the PostgreSQL connector, and improved error messages across all data connectors. The Clickhouse data connector enables data streaming by default. The public SQL interface now restricts DML and DDL queries. Additionally, accelerated tables now fully support NULL values, and issues with schema conversion in these tables have been resolved.

Highlights​

  • Snowflake Data Connector: Initial support for Snowflake as a data source.

  • Clickhouse Data Streaming: Enables data streaming by default, eliminating in-memory result collection.

  • Read-only SQL Interface: Disables DML (INSERT/UPDATE/DELETE) and DDL (CREATE/ALTER TABLE) queries for improved data source security.

  • Error Message Improvements: Improved the error messages for commonly encountered issues with data connectors.

  • Accelerated Tables: Supports NULL values across all data types and fixes schema conversion errors for consistent type handling.

Contributors​

  • @ahirner
  • @y-f-u
  • @sgrebnov
  • @ewgenius
  • @Jeadie
  • @gloomweaver
  • @Sevenannn
  • @digadeesh
  • @phillipleblanc

What's Changed​

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.12.0-alpha...v0.12.1-alpha

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.