The v0.15-alpha release introduces support for streaming databases changes with Change Data Capture (CDC) into accelerated tables via a new Debezium connector, configurable retry logic for data refresh, and the release of a new C# SDK to build with Spice in Dotnet.

Highlights in v0.15-alpha​

• Debezium data connector with Change Data Capture (CDC): Sync accelerated datasets with Debezium data sources over Kafka in real-time.

• Data Refresh Retries: By default, accelerated datasets attempt to retry data refreshes on transient errors. This behavior can be configured using refresh_retry_enabled and refresh_retry_max_attempts.

• C# Client SDK: A new C# Client SDK has been released for developing applications in Dotnet.

Debezium data connector with Change Data Capture (CDC)​

Integrating Debezium CDC is straightforward. Get started with the Debezium CDC Sample, read more about CDC in Spice, and read the Debezium data connector documentation.

Example Spicepod using Debezium CDC:

datasets:
  - from: debezium:cdc.public.customer_addresses
    name: customer_addresses_cdc
    params:
      debezium_transport: kafka
      debezium_message_format: json
      kafka_bootstrap_servers: localhost:19092
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      refresh_mode: changes

Data Refresh Retries​

Example Spicepod configuration limiting refresh retries to a maximum of 10 attempts:

datasets:
  - from: eth.blocks
    name: blocks
    acceleration:
      refresh_retry_enabled: true
      refresh_retry_max_attempts: 10
      refresh_check_interval: 30s

Breaking Changes​

None.

New Contributors​

• @rupurt made their first contribution in https://github.com/spiceai/spiceai/pull/1791

Contributors​

• @ewgenius
• @phillipleblanc
• @sgrebnov
• @y-f-u
• @rupurt
• @Jeadie
• @peasee

What's Changed​

Dependencies​

No major dependency updates.

Commits​

• Update version to 0.15.0-alpha by @ewgenius in https://github.com/spiceai/spiceai/pull/1784
• Update helm for v0.14.1-alpha by @ewgenius in https://github.com/spiceai/spiceai/pull/1786
• Run PR checks on PRs merging into feature-- branches by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1788
• Enable retries for accelerated table refresh by @sgrebnov in https://github.com/spiceai/spiceai/pull/1762
• enable more tpch benchmark queries as a result of decimal unparsing by @y-f-u in https://github.com/spiceai/spiceai/pull/1790
• add nix flake by @rupurt in https://github.com/spiceai/spiceai/pull/1791
• Support local and HF embedding models by @Jeadie in https://github.com/spiceai/spiceai/pull/1789
• fix(bin/spice): Implement custom Unmarshaller for DatasetOrReference by @peasee in https://github.com/spiceai/spiceai/pull/1787
• For windows, move symlink -> symlink_file. by @Jeadie in https://github.com/spiceai/spiceai/pull/1793
• docs: Add PULL_REQUEST_TEMPLATE.md by @peasee in https://github.com/spiceai/spiceai/pull/1794
• Fix Unsupported DataType: conversion for time predicates by @sgrebnov in https://github.com/spiceai/spiceai/pull/1795
• Use incremental backoff for initial dataset registration retries by @sgrebnov in https://github.com/spiceai/spiceai/pull/1805
• Basic HTTP/S connector by @Jeadie in https://github.com/spiceai/spiceai/pull/1792
• Scale support for Snowflake fixed-point numbers by @sgrebnov in https://github.com/spiceai/spiceai/pull/1804
• bump datafusion federation to resolve the join query failures by @y-f-u in https://github.com/spiceai/spiceai/pull/1806
• fix: Stream PostgreSQL data in by @peasee in https://github.com/spiceai/spiceai/pull/1798
• Remove clippy::module_name_repetitions lint by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1812
• Improve Snowflake fixed-point numbers casting by @sgrebnov in https://github.com/spiceai/spiceai/pull/1809
• Case insensitive secret getter by @ewgenius in https://github.com/spiceai/spiceai/pull/1813
• refactor: Format TOML with Taplo by @peasee in https://github.com/spiceai/spiceai/pull/1808
• feat: Update PR template, add label enforcement in PR by @peasee in https://github.com/spiceai/spiceai/pull/1815
• fix bug that append may miss updates when the incremental changes are not able to be contained in one record batch by @y-f-u in https://github.com/spiceai/spiceai/pull/1817
• add integration test for inner join across federated table and accelerated table by @y-f-u in https://github.com/spiceai/spiceai/pull/1811
• Unify spicepod.llms into spicepod.models and refactor UX of spicepod.models by @Jeadie in https://github.com/spiceai/spiceai/pull/1818
• Fix issue with querying accelerated tables where the dataset name has a schema by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1823
• Fix schema support for refresh_sql and improve e2e tests by @sgrebnov in https://github.com/spiceai/spiceai/pull/1826
• feat: Add GraphQL unnesting by @peasee in https://github.com/spiceai/spiceai/pull/1822
• fix: Allow kind/optimization labels, increase Postgres test timeout by @peasee in https://github.com/spiceai/spiceai/pull/1830
• Implement Real-time acceleration updates via Debezium CDC by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1832
• Remove println statement from PG Connector by @sgrebnov in https://github.com/spiceai/spiceai/pull/1835
• Don't try to "hot reload" Debezium accelerated datasets by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1837
• Create v1/search that performs vector search. by @Jeadie in https://github.com/spiceai/spiceai/pull/1836
• Align spicepod UX of embeddings with models by @Jeadie in https://github.com/spiceai/spiceai/pull/1829
• Add "cmake-build" feature to rdkafka for windows by @Jeadie in https://github.com/spiceai/spiceai/pull/1840
• Add a better error message when trying to configure refresh_mode=changes on a data connector that doesn't support it. by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1839

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.14.1-alpha...v0.15.0-alpha

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

• Twitter: @spice_ai
• Slack: spiceai.org/slack
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: hey@spice.ai

The v0.14.1-alpha release is focused on quality, stability, and type support with improvements in PostgreSQL, DuckDB, and GraphQL data connectors.

Highlights​

• PostgreSQL acceleration and data connector: Support for Composite Types and UUID data types.
• DuckDB acceleration and data connector: Support for LargeUTF8 and DuckDB functions.
• GraphQL data connector: Improved error handling on invalid query syntax.
• Refresh SQL: Improved stability when overwriting STRUCT data types.

Breaking Changes​

None.

New Contributors​

• @phungleson made their first contribution in https://github.com/spiceai/spiceai/pull/1750
• @peasee made their first contribution in https://github.com/spiceai/spiceai/pull/1769

Contributors​

• @lukekim
• @y-f-u
• @ewgenius
• @phillipleblanc
• @Jeadie
• @sgrebnov
• @gloomweaver
• @phungleson
• @peasee
• @digadeesh

What's Changed​

Dependencies​

No major dependency updates.

Commits​

• Update Helm to v0.14.0-alpha by @sgrebnov in https://github.com/spiceai/spiceai/pull/1720
• Update version to 0.14.1-alpha by @sgrebnov in https://github.com/spiceai/spiceai/pull/1721
• Use spiceai/async-openai to solve Deserialize issue in v1/embed by @Jeadie in https://github.com/spiceai/spiceai/pull/1707
• Add greatest least user defined functions by @y-f-u in https://github.com/spiceai/spiceai/pull/1722
• default timeunit to be seconds when time column is a numeric column by @y-f-u in https://github.com/spiceai/spiceai/pull/1727
• use system conf to construct dns resolver by @y-f-u in https://github.com/spiceai/spiceai/pull/1728
• fix a bug that dataset refresh api does not work for table with schema by @y-f-u in https://github.com/spiceai/spiceai/pull/1729
• Move secret crate to runtime module by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1723
• Return schema in get_flight_info_simple by @gloomweaver in https://github.com/spiceai/spiceai/pull/1724
• Refactor vector search component of v1/assist into a VectorSearch struct by @Jeadie in https://github.com/spiceai/spiceai/pull/1699
• Update ROADMAP.md. Fix a broken link for the "Get in touch" link. by @digadeesh in https://github.com/spiceai/spiceai/pull/1725
• Secret keys in params should be case insensitive by @ewgenius in https://github.com/spiceai/spiceai/pull/1737
• expose error log when refresh encountered some issue, also add more debug logs by @y-f-u in https://github.com/spiceai/spiceai/pull/1739
• Support Struct in PostgreSQL accelerator by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1733
• rewrite refresh append update dedup logic using arrow comparators by @y-f-u in https://github.com/spiceai/spiceai/pull/1743
• Add health checks when loading (llms, embeddings) by @Jeadie in https://github.com/spiceai/spiceai/pull/1738
• Support DuckDB function in DuckDB datasets by @Jeadie in https://github.com/spiceai/spiceai/pull/1742
• Update version of spiceai/duckdb-rs, support LargeUTF8 by @Jeadie in https://github.com/spiceai/spiceai/pull/1746
• Split refresh into coordination and execution layers by @sgrebnov in https://github.com/spiceai/spiceai/pull/1744
• bump duckdb rs git sha to resolve duckdb incorrect null value issue by @y-f-u in https://github.com/spiceai/spiceai/pull/1747
• cargo.lock file update with #1747 duckdb-rs sha by @y-f-u in https://github.com/spiceai/spiceai/pull/1748
• Fix error when GraphQL error locations is missing by @phungleson in https://github.com/spiceai/spiceai/pull/1750
• Tweak refresh scheduling logic by @sgrebnov in https://github.com/spiceai/spiceai/pull/1749
• Ensure tonic package is in duckdb feature by @Jeadie in https://github.com/spiceai/spiceai/pull/1756
• Change tonic::async_trait -> async_trait::async_trait by @Jeadie in https://github.com/spiceai/spiceai/pull/1757
• Streaming in v1/chat/completion by @Jeadie in https://github.com/spiceai/spiceai/pull/1741
• Add refresh_retry_enabled/max_attempts acceleration params by @sgrebnov in https://github.com/spiceai/spiceai/pull/1753
• Implement refresh retry based on fibonacci backoff (not enabled) by @sgrebnov in https://github.com/spiceai/spiceai/pull/1752
• Add VSCode debug target to debug runtime benchmark test by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1760
• update spiceai datafusion to include more unparser rules by @y-f-u in https://github.com/spiceai/spiceai/pull/1764
• Show UUID types as String instead of base64 binary. by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1767
• docs: Add linux contributor guide for setup by @peasee in https://github.com/spiceai/spiceai/pull/1769
• Do not expose connection url on object store error by @ewgenius in https://github.com/spiceai/spiceai/pull/1761
• Support secrets in llm and embeddings params by @ewgenius in https://github.com/spiceai/spiceai/pull/1770
• Bump github.com/hashicorp/go-retryablehttp from 0.7.1 to 0.7.7 by @dependabot in https://github.com/spiceai/spiceai/pull/1775
• Update ROADMAP.md with latest roadmap changes for v0.15.0 by @digadeesh in https://github.com/spiceai/spiceai/pull/1773
• Update acknowledgements by @github-actions in https://github.com/spiceai/spiceai/pull/1776
• Strip kwarg '=' in DuckDB function parsing by @Jeadie in https://github.com/spiceai/spiceai/pull/1777

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.14.0-alpha...v0.14.1-alpha

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

• Twitter: @spice_ai
• Slack: spiceai.org/slack
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: hey@spice.ai

The v0.14-alpha release focuses on enhancing accelerated dataset performance and data integrity, with support for configuring primary keys and indexes. Additionally, the GraphQL data connector been introduced, along with improved dataset registration and loading error information.

Highlights​

• Accelerated Datasets: Ensure data integrity using primary key and unique index constraints. Configure conflict handling to either upsert new data or drop it. Create indexes on frequently filtered columns for faster queries on larger datasets.

• GraphQL Data Connector: Initial support for using GraphQL as a data source.

Example Spicepod showing how to use primary keys and indexes with accelerated datasets:

datasets:
  - from: eth.blocks
    name: blocks
    acceleration:
      engine: duckdb # Use DuckDB acceleration engine
      primary_key: '(hash, timestamp)'
      indexes:
        number: enabled # same as `CREATE INDEX ON blocks (number);`
        '(number, hash)': unique # same as `CREATE UNIQUE INDEX ON blocks (number, hash);`
      on_conflict:
        '(hash, number)': drop # possible values: drop (default), upsert
        '(hash, timestamp)': upsert

Primary Keys, constraints, and indexes are currently supported when using SQLite, DuckDB, and PostgreSQL acceleration engines.

Learn more with the indexing quickstart and the primary key sample.

Read the Local Acceleration documentation.

Breaking Changes​

None.

Contributors​

• @phillipleblanc
• @ewgenius
• @sgrebnov
• @Jeadie
• @digadeesh
• @gloomweaver
• @y-f-u
• @lukekim
• @edmondop

What's Changed​

Dependencies​

• Apache DataFusion: Upgraded from 38.0.0 to 39.0.0
• Apache Arrow/Parquet: Upgraded from 51.0.0 to 52.0.0
• Rust: Upgraded from 1.78.0 to 1.79.0

Commits​

• Update Helm chart for v0.13.3-alpha by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1671
• Bump version to v0.14.0-alpha by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1673
• Dependency upgrades: DataFusion 39, Arrow/Parquet 52, object_store 0.10.1, arrow-odbc 11.1.0 by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1674
• Generate unique runtime instance name and store in runtime.metrics table by @ewgenius in https://github.com/spiceai/spiceai/pull/1678
• Proper support for Snowflake TIMESTAMP_NTZ by @sgrebnov in https://github.com/spiceai/spiceai/pull/1677
• Enable tpch_q2 and tpch_q21 in the benchmark queries by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1679
• Start runtime metrics recorder after loading secrets and extensions by @ewgenius in https://github.com/spiceai/spiceai/pull/1680
• Validate table constraints (Primary Keys/Unique Index) on accelerated tables by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1658
• Store labels as JSON string in runtime.metrics by @ewgenius in https://github.com/spiceai/spiceai/pull/1681
• Atomic updates for DuckDB tables with constraints by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1682
• Rename metrics column labels to properties and make it nullable by @ewgenius in https://github.com/spiceai/spiceai/pull/1686
• Fix federation_optimizer_rule schema error for tpch_q7, tpch_q8, tpch_q9, tpch_q14 by @sgrebnov in https://github.com/spiceai/spiceai/pull/1683
• Better prompt for /v1/assist by @Jeadie in https://github.com/spiceai/spiceai/pull/1685
• Support stream in v1/assist by @Jeadie in https://github.com/spiceai/spiceai/pull/1653
• Fix cache hit rate chart loading for Grafana v9.5 by @sgrebnov in https://github.com/spiceai/spiceai/pull/1691
• Update ROADMAP.md to include data connector statuses by @digadeesh in https://github.com/spiceai/spiceai/pull/1684
• Support primary_key in Spicepod and create in accelerated table by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1687
• Datasets with schema support for availability monitoring by @sgrebnov in https://github.com/spiceai/spiceai/pull/1690
• Improve dataset registration output by @sgrebnov in https://github.com/spiceai/spiceai/pull/1692
• Readme: update dataset registration traces by @sgrebnov in https://github.com/spiceai/spiceai/pull/1694
• Improved error logging for datasets load error by @edmondop in https://github.com/spiceai/spiceai/pull/1695
• Improve ArrayDistance scalar UDF by @Jeadie in https://github.com/spiceai/spiceai/pull/1697
• Implement on_conflict behavior for accelerated tables with constraints by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1688
• Fix datasets live update (Spice file watcher) by @sgrebnov in https://github.com/spiceai/spiceai/pull/1702
• Grafana Dashboard: replace Quantile with Percentile filter by @sgrebnov in https://github.com/spiceai/spiceai/pull/1703
• refresh with append overlap by @y-f-u in https://github.com/spiceai/spiceai/pull/1706
• Fix error message on DuckDB constraint violation by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1709
• Add warning when configuring indexes/primary_key/on_conflict for Arrow engine. by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1710
• ensure schema to be existing when query timestamp during refresh by @y-f-u in https://github.com/spiceai/spiceai/pull/1711
• Improve README clarity and add comparison table by @lukekim in https://github.com/spiceai/spiceai/pull/1713
• Update acknowledgements by @github-actions in https://github.com/spiceai/spiceai/pull/1716
• Update README.md to include GraphQL data connector in supported table by @digadeesh in https://github.com/spiceai/spiceai/pull/1717
• Fix quoting issue for databricks identifier by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1718

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.13.3-alpha...v0.14.0-alpha

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

• Twitter: @spice_ai
• Slack: spiceai.org/slack
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: hey@spice.ai

The v0.13.3-alpha release is focused on quality and stability with improvements to metrics, telemetry, and operability.

Highlights​

Ready API: - Add /v1/ready API that returns success once all datasets and models are loaded and ready.

Enhanced Grafana dashboard: The dashboard now includes charts for query duration and failures, the last update time of accelerated datasets, the count of refresh errors, and the last successful time the runtime was able to access federated datasets

Contributors​

• @Jeadie
• @ewgenius
• @phillipleblanc
• @sgrebnov
• @gloomweaver
• @y-f-u
• @mach-kernel

What's Changed​

Dependencies​

• DuckDB 1.0.0: Upgrades embedded DuckDB to 1.0.0.

Commits​

• Scalar UDF array_distance as euclidean distance between Float32[] by @Jeadie in https://github.com/spiceai/spiceai/pull/1601
• Update version to v0.14.0-alpha by @ewgenius in https://github.com/spiceai/spiceai/pull/1614
• Update helm for v0.13.2-alpha by @ewgenius in https://github.com/spiceai/spiceai/pull/1618
• Upgrade duckdb-rs to DuckDB 1.0.0 by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1615
• initial idea for 'POST v1/assist' by @Jeadie in https://github.com/spiceai/spiceai/pull/1585
• openai server trait and move HTTP endpoints to crates/runtime/src/http/v1/ by @Jeadie in https://github.com/spiceai/spiceai/pull/1619
• Add branching policy & updated endgame instructions by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1620
• Update Cargo.lock & add CI check for updated Cargo.lock by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1627
• Add first-class support for views by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1622
• Add /v1/ready API that returns 200 when all datasets have loaded by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1629
• Separate NQL logic from LLM Chat messages, and add OpenAI compatiblility per LLM trait. by @Jeadie in https://github.com/spiceai/spiceai/pull/1628
• Log queries failing on get_flight_info step (Flight Api) by @sgrebnov in https://github.com/spiceai/spiceai/pull/1626
• Graphql Data Connector by @gloomweaver in https://github.com/spiceai/spiceai/pull/1624
• GraphQL improved Error formatting, proper format request body by @gloomweaver in https://github.com/spiceai/spiceai/pull/1637
• Fix v1/assist response and panic bug. Include primary keys in response too by @Jeadie in https://github.com/spiceai/spiceai/pull/1635
• skip integration test if no secret by @y-f-u in https://github.com/spiceai/spiceai/pull/1638
• [append] Refresher::get_latest_timestamp / get_df to add refresh_sql predicates to scan by @mach-kernel in https://github.com/spiceai/spiceai/pull/1636
• GraphQL integration test by @gloomweaver in https://github.com/spiceai/spiceai/pull/1600
• Add err_code to query_failures metric by @sgrebnov in https://github.com/spiceai/spiceai/pull/1639
• use epoch_ms to replace epoch to work with timestamptz by @y-f-u in https://github.com/spiceai/spiceai/pull/1641
• fix the schema mismatch issue on the fallback plan use schema casting by @y-f-u in https://github.com/spiceai/spiceai/pull/1642
• bug report template update by @y-f-u in https://github.com/spiceai/spiceai/pull/1640
• Add query duration, failures and accelerated dataset metrics to Grafana dashboard by @sgrebnov in https://github.com/spiceai/spiceai/pull/1598
• Fix FTP/sftp support for ObjectStoreMetadataTable & ObjectStoreTextTable by @Jeadie in https://github.com/spiceai/spiceai/pull/1649
• Support accelerated embedding tables in v1/assist by @Jeadie in https://github.com/spiceai/spiceai/pull/1648
• GraphQL pagination, limit pushdown and refactor by @gloomweaver in https://github.com/spiceai/spiceai/pull/1643
• Support indexes in accelerated tables by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1644
• Federated datasets availability monitoring by @sgrebnov in https://github.com/spiceai/spiceai/pull/1650
• Reset federated dataset availability during dataset registration by @sgrebnov in https://github.com/spiceai/spiceai/pull/1661
• Change to v0.13.3-alpha by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1666
• Add Time Since Offline chart to Grafana dashboard by @sgrebnov in https://github.com/spiceai/spiceai/pull/1664

• readme fix to correct the number of rows for show tables by @y-f-u in https://github.com/spiceai/spiceai/pull/1667
• Update acknowledgements by @github-actions in https://github.com/spiceai/spiceai/pull/1668
• Add missing dependency on arrow_sql_gen from duckdb data_component by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1669

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.13.2-alpha...v0.13.3-alpha

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

• Twitter: @spice_ai
• Slack: spiceai.org/slack
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: hey@spice.ai

The v0.13.2-alpha release is focused on quality and stability with improvements to federated query push-down, telemetry, and query history.

Highlights​

• Filesystem Data Connector: Adds the Filesystem Data Connector for directly using files as data sources.

• Federated Query Push-Down: Improved stability and schema compatibility for federated queries.

• Enhanced Telemetry: Runtime Metrics now include last update time for accelerated datasets, count of refresh errors, and new metrics for query duration and failures.

• Query History: Enabled query history logging for Arrow Flight queries in addition to HTTP queries.

Contributors​

• @lukekim
• @y-f-u
• @ewgenius
• @phillipleblanc
• @Jeadie
• @Sevenannn
• @sgrebnov
• @gloomweaver
• @mach-kernel

What's Changed​

• Update ROADMAP.md May 27, 2024 by @lukekim in https://github.com/spiceai/spiceai/pull/1535
• update helm chart version and use v0.13.1-alpha by @y-f-u in https://github.com/spiceai/spiceai/pull/1536
• version correction in v0.13.1 release note by @y-f-u in https://github.com/spiceai/spiceai/pull/1538
• update version to v0.14.0-alpha by @y-f-u in https://github.com/spiceai/spiceai/pull/1539
• Update spice_cloud - connect to cloud api by @ewgenius in https://github.com/spiceai/spiceai/pull/1523
• Update spice_cloud extension params, and remove logging by @ewgenius in https://github.com/spiceai/spiceai/pull/1541
• Update MSRV to 1.78 and remove unused Rust Version parameter in CI by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1540
• Improve llm UX in spicepod.yaml by @Jeadie in https://github.com/spiceai/spiceai/pull/1545
• Store local runtime metrics in Timestamp with nanoseconds precision and UTC time by @ewgenius in https://github.com/spiceai/spiceai/pull/1548
• Object store metadata Table provider by @Jeadie in https://github.com/spiceai/spiceai/pull/1518
• Remove clickhouse password requirement by @Sevenannn in https://github.com/spiceai/spiceai/pull/1547
• pretty print loaded rows number by @y-f-u in https://github.com/spiceai/spiceai/pull/1553
• Fix UNION ALL federated push down by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1550
• Update mistral, fix bugs and improve local file DX by @Jeadie in https://github.com/spiceai/spiceai/pull/1552
• Cast runtime.metrics schema, if remote (spiceai) data connector provided by @ewgenius in https://github.com/spiceai/spiceai/pull/1554
• Use proper MySQL dialect during federation push-down by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1555
• parallel load dataset when starting up by @y-f-u in https://github.com/spiceai/spiceai/pull/1551
• fix linter warning on Scanf return value by @y-f-u in https://github.com/spiceai/spiceai/pull/1556
• Update spice cloud connect api endpoint by @ewgenius in https://github.com/spiceai/spiceai/pull/1557
• Create new HTTP endpoint to create embeddings. by @Jeadie in https://github.com/spiceai/spiceai/pull/1558
• Query History support for Flight API by @sgrebnov in https://github.com/spiceai/spiceai/pull/1549
• Don't cache queries for runtime tables by @sgrebnov in https://github.com/spiceai/spiceai/pull/1561
• Fix schema incompatibility on federated push-down queries by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1560
• move 'embeddings' to top-level concept in spicepod.yaml by @Jeadie in https://github.com/spiceai/spiceai/pull/1564
• object_store table provider for UTF8 data formats by @Jeadie in https://github.com/spiceai/spiceai/pull/1562
• Improve connectivity for JDBC clients, like Tableau by @sgrebnov in https://github.com/spiceai/spiceai/pull/1563
• Enable datasets from local filesystem by @Jeadie in https://github.com/spiceai/spiceai/pull/1584
• Adds benchmarking tests for Spice by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1577
• Push down correct timestamp expr to SQLite, add binary type mapping by @mach-kernel in https://github.com/spiceai/spiceai/pull/1566
• Add query_duration_seconds and query_failures metrics by @sgrebnov in https://github.com/spiceai/spiceai/pull/1575
• Use /app as a default workdir in spiceai docker image by @ewgenius in https://github.com/spiceai/spiceai/pull/1586
• Add support for both file:// and file:/ by @Jeadie in https://github.com/spiceai/spiceai/pull/1587
• put load_datasets as the latest step along with start_servers by @y-f-u in https://github.com/spiceai/spiceai/pull/1559
• Embedding columns (from embedding providers) are now run inside datafusion plans. by @Jeadie in https://github.com/spiceai/spiceai/pull/1576
• Support BinaryArray in DuckDB accelerations by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1595
• Add cache header to Flight API and Spice REPL indicator by @sgrebnov in https://github.com/spiceai/spiceai/pull/1591
• Add accelerated datasets refresh metrics by @sgrebnov in https://github.com/spiceai/spiceai/pull/1589
• update the error when starting spice sql with no runtime to be actionable by @digadeesh in https://github.com/spiceai/spiceai/pull/1597
• add odbc integration test by @y-f-u in https://github.com/spiceai/spiceai/pull/1590
• Fix bug in instantiating EmbeddingConnector by @Jeadie in https://github.com/spiceai/spiceai/pull/1592
• readme change to reflect new cli output by @y-f-u in https://github.com/spiceai/spiceai/pull/1602
• Update version v0.13.2 by @ewgenius in https://github.com/spiceai/spiceai/pull/1604
• Roadmap changes Jun 3, 2024 by @lukekim in https://github.com/spiceai/spiceai/pull/1609

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.13.1-alpha...v0.13.2

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

• Twitter: @spice_ai
• Slack: spiceai.org/slack
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: hey@spice.ai

The v0.13.1-alpha release of Spice is a minor update focused on stability, quality, and operability. Query result caching provides protection against bursts of queries and schema support for datasets has been added logical grouping. An issue where Refresh SQL predicates were not pushed down underlying data sources has been resolved along with improved Acceleration Refresh logging.

Highlights in v0.13.1-alpha​

• Results Caching: Introduced query results caching to handle bursts of requests and support caching of non-accelerated results, such as refresh data returned on zero results. Results caching is enabled by default with a 1s item time-to-live (TTL). Learn more.

• Query History Logging: Recent queries are now logged in the new spice.runtime.query_history dataset with a default retention of 24-hours. Query history is initially enabled for HTTP queries only (not Arrow Flight queries).

• Dataset Schemas: Added support for dataset schemas, allowing logical grouping of datasets by separating the schema name from the table name with a .. E.g.

  datasets:
    - from: mysql:app1.identities
      name: app.users
  
    - from: postgres:app2.purchases
      name: app.purchases
  

  In this example, queries against app.users will be federated to my_schema.my_table, and app.purchases will be federated to app2.purchases.

Contributors​

@y-f-u @Jeadie @sgrebnov @ewgenius @phillipleblanc @lukekim @gloomweaver @Sevenannn

New in this release​

• Add more type support on mysql connector by @y-f-u in https://github.com/spiceai/spiceai/pull/1449
• Add in-memory caching support for Arrow Flight queries by @sgrebnov in https://github.com/spiceai/spiceai/pull/1450
• Fix the table reference to use the full table reference, not just the table by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1460
• Make file_format parameter required for S3/FTP/SFTP connector by @ewgenius in https://github.com/spiceai/spiceai/pull/1455
• Add more verbose logging when acceleration refresh update is finished by @y-f-u in https://github.com/spiceai/spiceai/pull/1453
• Fix snowflake dataset path when using federation query by @y-f-u in https://github.com/spiceai/spiceai/pull/1474
• Update cargo to use spiceai datafusion fork by @y-f-u in https://github.com/spiceai/spiceai/pull/1475
• Enable in-memory results caching by default by @sgrebnov in https://github.com/spiceai/spiceai/pull/1473
• Add basic integration test for MySQL federation by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1477
• Update results_cache config names per final spec by @sgrebnov in https://github.com/spiceai/spiceai/pull/1487
• Add DuckDB quickstart to E2E tests by @lukekim in https://github.com/spiceai/spiceai/pull/1461
• Add X-Cache header for http queries by @sgrebnov in https://github.com/spiceai/spiceai/pull/1472
• Add telemetry for in-memory caching by @sgrebnov in https://github.com/spiceai/spiceai/pull/1456
• Pin Git dependencies to a specific commit hash by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1490
• Detect file_format from dataset path by @ewgenius in https://github.com/spiceai/spiceai/pull/1489
• Add file_format to helm chart sample dataset by @ewgenius in https://github.com/spiceai/spiceai/pull/1493
• Improve duckdb data connector error messages by @Sevenannn in https://github.com/spiceai/spiceai/pull/1486
• Add file_format prompt for s3 and ftp datasets in Dataset Configure CLI if no extension detected by @ewgenius in https://github.com/spiceai/spiceai/pull/1494
• Add llms to the spicepod definition and use throughout by @Jeadie in https://github.com/spiceai/spiceai/pull/1447
• Fix duckdb acceleration converting null into default values. by @y-f-u in https://github.com/spiceai/spiceai/pull/1500
• Separate runtime Dataset from spicepod Dataset by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1503
• Duckdb e2e test OSX support by @y-f-u in https://github.com/spiceai/spiceai/pull/1505
• Use TableReference for dataset name by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1506
• Tweak Results Cache naming and output by @lukekim in https://github.com/spiceai/spiceai/pull/1509
• Fix refresh_sql not properly passing down filters by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1510
• Allow datasets to specify a schema by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1507
• Cache invalidation for accelerated tables by @sgrebnov in https://github.com/spiceai/spiceai/pull/1498
• Improve spark data connector error messages by @Sevenannn in https://github.com/spiceai/spiceai/pull/1497
• Parse postgres table schema from prepare statement to support empty tables by @ewgenius in https://github.com/spiceai/spiceai/pull/1445
• Improve clarity of README and add FAQ by @lukekim in https://github.com/spiceai/spiceai/pull/1512
• Use binary data transfer for ftp by @gloomweaver in https://github.com/spiceai/spiceai/pull/1517
• Add support for time64 for SQL insertion statement by @y-f-u in https://github.com/spiceai/spiceai/pull/1519
• Add Spice Extensions PoC by @ewgenius in https://github.com/spiceai/spiceai/pull/1476
• Add results cache metrics, pod and quantile filters to Grafana dashboard by @sgrebnov in https://github.com/spiceai/spiceai/pull/1513
• Add unit tests for results caching utils by @sgrebnov in https://github.com/spiceai/spiceai/pull/1514
• Add E2E tests for results caching by @sgrebnov in https://github.com/spiceai/spiceai/pull/1515
• Pass table_reference full string into spark_session table so it can query across schemas or catalogs by @y-f-u in https://github.com/spiceai/spiceai/pull/1521
• Trace on debug level for tables in runtime schema by @ewgenius in https://github.com/spiceai/spiceai/pull/1524
• Update SparkSessionBuilder::remote and update spark fork hash by @Sevenannn in https://github.com/spiceai/spiceai/pull/1495
• Fix federation push-down for datasets with schemas by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1526
• Store history of queries in 'spice.runtime.query_history' by @Jeadie in https://github.com/spiceai/spiceai/pull/1501
• Disable cache for system queries by @sgrebnov in https://github.com/spiceai/spiceai/pull/1528
• Register runtime tables with runtime schema by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1532
• Fix acknowledgments workflow to include all cargo features by @Jeadie in https://github.com/spiceai/spiceai/pull/1531

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.13.0-alpha...v0.13.1-alpha

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

• Twitter: @spice_ai
• Slack: spiceai.org/slack
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: hey@spice.ai

The v0.13.0-alpha release significantly improves federated query performance and efficiency with Query Push-Down. Query push-down allows SQL queries to be directly executed by underlying data sources, such as joining tables using the same data connector. Query push-down is supported for all SQL-based and Arrow Flight data connectors. Additionally, runtime metrics, including query duration, collected and accessed in the spice.runtime.metrics table. This release also includes a new FTP/SFTP data connector and improved CSV support for the S3 data connector.

Highlights​

• Federated Query Push-Down (#1394): All SQL and Arrow Flight data connectors support federated query push-down.

• Runtime Metrics (#1361): Runtime metric collection can be enabled using the --metrics flag and accessed by the spice.runtime.metrics table.

• FTP & SFTP data connector (#1355) (#1399): Added support for using FTP and SFTP as data sources.

• Improved CSV support (#1411) (#1414): S3/FTP/SFTP data connectors support CSV files with expanded CSV options.

Contributors​

• @Jeadie
• @digadeesh
• @ewgenius
• @gloomweaver
• @lukekim
• @phillipleblanc
• @sgrebnov
• @y-f-u

What's Changed​

• Remove milestones from Enhancement template by @lukekim in https://github.com/spiceai/spiceai/pull/1373
• Update version.txt and Cargo.toml to 0.13.0-alpha by @sgrebnov in https://github.com/spiceai/spiceai/pull/1375
• Helm chart for Spice v0.12.2-alpha by @sgrebnov in https://github.com/spiceai/spiceai/pull/1374
• Add release cargo feature to docker builds by @ewgenius in https://github.com/spiceai/spiceai/pull/1377
• FTP connector by @gloomweaver in https://github.com/spiceai/spiceai/pull/1355
• Provide ability to specify timeout for s3 data connector by @gloomweaver in https://github.com/spiceai/spiceai/pull/1378
• clickhouse-rs use tag instead of branch by @gloomweaver in https://github.com/spiceai/spiceai/pull/1313
• Store runtime metrics in spice.runtime.metrics table by @ewgenius in https://github.com/spiceai/spiceai/pull/1361
• Update bug_report.md to include the kind/bug label by @digadeesh in https://github.com/spiceai/spiceai/pull/1381
• Remove redundant [refresh] in log by @lukekim in https://github.com/spiceai/spiceai/pull/1384
• Implement federation for DuckDB Data Connector (POC) by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1380
• Update wording for spice cloud connection by @ewgenius in https://github.com/spiceai/spiceai/pull/1386
• fix dataset refreshing status by @y-f-u in https://github.com/spiceai/spiceai/pull/1387
• clickhouse friendly error by @y-f-u in https://github.com/spiceai/spiceai/pull/1388
• Initial work for NQL crate and API by @Jeadie in https://github.com/spiceai/spiceai/pull/1366
• Fully implement federation for all SqlTable-based Data Connectors by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1394
• use df logical plan to query latest timestamp when refreshing incrementally by @y-f-u in https://github.com/spiceai/spiceai/pull/1393
• Refactor datafusion.write_data to use table reference by @ewgenius in https://github.com/spiceai/spiceai/pull/1402
• Add federation to FlightTable based DataConnectors by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1401
• SFTP Data Connector by @gloomweaver in https://github.com/spiceai/spiceai/pull/1399
• Use GPT3.5 for NSQL task by @Jeadie in https://github.com/spiceai/spiceai/pull/1400
• Update ROADMAP May 16, 2024 by @lukekim in https://github.com/spiceai/spiceai/pull/1405
• Add ftp/sftp connector to readme by @gloomweaver in https://github.com/spiceai/spiceai/pull/1404
• Add FlightSQL federation provider by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1403
• Refactor runtime metrics to use localhost accelerated table by @ewgenius in https://github.com/spiceai/spiceai/pull/1395
• Use JSON response in OpenAI, text -> SQL model by @Jeadie in https://github.com/spiceai/spiceai/pull/1407
• support more common csv options by @y-f-u in https://github.com/spiceai/spiceai/pull/1411
• add a TLS error message in data connector and implement it for clickhouse by @y-f-u in https://github.com/spiceai/spiceai/pull/1413
• Add CSV to s3 data formats by @gloomweaver in https://github.com/spiceai/spiceai/pull/1414
• fix up dependencies now 0.5.0 disappeared by @Jeadie in https://github.com/spiceai/spiceai/pull/1417
• Add NSQL to FlightRepl by @Jeadie in https://github.com/spiceai/spiceai/pull/1409
• Update Cargo.lock by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1418
• Enable spice.ai replication for runtime.metrics table by @ewgenius in https://github.com/spiceai/spiceai/pull/1408
• Restructure the runtime struct to make it easier to test by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1420
• Make it easier to construct an App programatically by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1421
• Add an integration test for federation by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1426
• wait 2 seconds for the status to turn ready in refreshing status test by @y-f-u in https://github.com/spiceai/spiceai/pull/1419
• Add functional tests for federation push-down by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1428
• Enable push-down federation by default by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1429
• Add guides and examples about error handling by @ewgenius in https://github.com/spiceai/spiceai/pull/1427
• Add LRU cache support for http-based queries by @sgrebnov in https://github.com/spiceai/spiceai/pull/1410
• Update README.md - Remove bigquery from tablet of connectors by @digadeesh in https://github.com/spiceai/spiceai/pull/1434
• Update acknowledgements by @github-actions in https://github.com/spiceai/spiceai/pull/1433
• CLI wording and logs change reflected on readme by @y-f-u in https://github.com/spiceai/spiceai/pull/1435
• Add databricks_use_ssl parameter by @Sevenannn in https://github.com/spiceai/spiceai/pull/1406
• Update helm version and use v0.13.0-alpha by @Jeadie in https://github.com/spiceai/spiceai/pull/1436
• Don't include feature 'llms/candles' by default by @Jeadie in https://github.com/spiceai/spiceai/pull/1437
• Correctly map NullBuilder for Null arrow types by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1438
• Propagate object store error by @gloomweaver in https://github.com/spiceai/spiceai/pull/1415

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.12.2-alpha...v0.13.0-alpha

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

• Twitter: @spice_ai
• Slack: spiceai.org/slack
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: hey@spice.ai

The v0.12.2-alpha release introduces data streaming and key-pair authentication for the Snowflake data connector, enables general append mode data refreshes for time-series data, improves connectivity error messages, adds nested folders support for the S3 data connector, and exposes nodeSelector and affinity keys in the Helm chart for better Kubernetes management.

Highlights​

• Improved Connectivity Error Messages: Error messages provide clearer, actionable guidance for misconfigured settings or unreachable data connectors.

• Snowflake Data Connector Improvements: Enables data streaming by default and adds support for key-pair authentication in addition to passwords.

• API for Refresh SQL Updates: Update dataset Refresh SQL via API.

• Append Data Refresh: Append mode data refreshes for time-series data are now supported for all data connectors. Specify a dataset time_column with refresh_mode: append to only fetch data more recent than the latest local data.

• Docker Image Update: The spiceai/spiceai:latest Docker image now includes the ODBC data connector. For a smaller footprint, use spiceai/spiceai:latest-slim.

• Helm Chart Improvements: nodeSelector and affinity keys are now supported in the Helm chart for improved Kubernetes deployment management.

Breaking Changes​

• API to trigger accelerated dataset refreshes has changed from POST /v1/datasets/:name/refresh to POST /v1/datasets/:name/acceleration/refresh to be consistent with the Spicepod.yaml structure.

Contributors​

• @mach-kernel
• @y-f-u
• @sgrebnov
• @ewgenius
• @Jeadie
• @Sevenannn
• @digadeesh
• @phillipleblanc
• @lukekim

What's Changed​

• Fix list type support in spark connect by @y-f-u in https://github.com/spiceai/spiceai/pull/1341
• Add nested folder support in S3 Parquet connector by @y-f-u in https://github.com/spiceai/spiceai/pull/1342
• Improves S3 connector using DataFusion ListingTable table provider by @y-f-u in https://github.com/spiceai/spiceai/pull/1326
• Update ROADMAP May 6, 2024 by @lukekim in https://github.com/spiceai/spiceai/pull/1315
• List flightsql and snowflake as supported connectors in README.md by @sgrebnov in https://github.com/spiceai/spiceai/pull/1317
• Helm chart for v0.12.1-alpha by @ewgenius in https://github.com/spiceai/spiceai/pull/1323
• Read sqlite_file param and use it as path by @Sevenannn in https://github.com/spiceai/spiceai/pull/1309
• Compile spiced with release feature in docker image by @ewgenius in https://github.com/spiceai/spiceai/pull/1324
• Add support for Snowflake key-pair authentication by @sgrebnov in https://github.com/spiceai/spiceai/pull/1314
• Wrap postgres errors in common DataConnectorError by @ewgenius in https://github.com/spiceai/spiceai/pull/1327
• Fix TPCH tests runner by @sgrebnov in https://github.com/spiceai/spiceai/pull/1330
• Spice CLI support for Snowflake key-pair auth by @sgrebnov in https://github.com/spiceai/spiceai/pull/1325
• sql_provider_datafusion: Support TimestampMicrosecond, Date32, Date64 by @mach-kernel in https://github.com/spiceai/spiceai/pull/1329
• Resolve dangling reference for SQLite by @Sevenannn in https://github.com/spiceai/spiceai/pull/1312
• Select columns from Spark Dataframe according to projected_schema by @Sevenannn in https://github.com/spiceai/spiceai/pull/1336
• Expose nodeselector and affinity keys in Helm chart by @mach-kernel in https://github.com/spiceai/spiceai/pull/1338
• Use streaming for Snowflake queries by @sgrebnov in https://github.com/spiceai/spiceai/pull/1337
• Publish ODBC images by @mach-kernel in https://github.com/spiceai/spiceai/pull/1271
• Include Postgres acceleration engine to types support tests by @sgrebnov in https://github.com/spiceai/spiceai/pull/1343
• Refactor dataconnector providers getters to return common DataConnectorResult and DataConnectorError by @ewgenius in https://github.com/spiceai/spiceai/pull/1339
• s3 csv support to validate the listing table extensibility by @y-f-u in https://github.com/spiceai/spiceai/pull/1344
• Move model code into separate, feature-flagged crate by @Jeadie in https://github.com/spiceai/spiceai/pull/1335
• Initial setup for federated queries by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1350
• Refactor dbconnection errors, and catch invalid postgres table name case by @ewgenius in https://github.com/spiceai/spiceai/pull/1353
• Rename default datafusion catalog to "spice", add internal "spice.runtime" schema by @ewgenius in https://github.com/spiceai/spiceai/pull/1359
• Add API to set Refresh SQL for accelerated table by @sgrebnov in https://github.com/spiceai/spiceai/pull/1356
• Set next version to v0.12.2 by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1367
• Upgrade to DataFusion 38 by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1368
• Incremental append based on time column by @y-f-u in https://github.com/spiceai/spiceai/pull/1360
• Update README.md to include correct output when running show tables from quickstart by @digadeesh in https://github.com/spiceai/spiceai/pull/1371

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.12.1-alpha...v0.12.2-alpha

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

• Twitter: @spice_ai
• Slack: spiceai.org/slack
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: hey@spice.ai

The v0.12.1-alpha release introduces a new Snowflake data connector, support for UUID and TimestampTZ types in the PostgreSQL connector, and improved error messages across all data connectors. The Clickhouse data connector enables data streaming by default. The public SQL interface now restricts DML and DDL queries. Additionally, accelerated tables now fully support NULL values, and issues with schema conversion in these tables have been resolved.

Highlights​

• Snowflake Data Connector: Initial support for Snowflake as a data source.

• Clickhouse Data Streaming: Enables data streaming by default, eliminating in-memory result collection.

• Read-only SQL Interface: Disables DML (INSERT/UPDATE/DELETE) and DDL (CREATE/ALTER TABLE) queries for improved data source security.

• Error Message Improvements: Improved the error messages for commonly encountered issues with data connectors.

• Accelerated Tables: Supports NULL values across all data types and fixes schema conversion errors for consistent type handling.

Contributors​

• @ahirner
• @y-f-u
• @sgrebnov
• @ewgenius
• @Jeadie
• @gloomweaver
• @Sevenannn
• @digadeesh
• @phillipleblanc

What's Changed​

• Add schema types check for query result by @sgrebnov in https://github.com/spiceai/spiceai/pull/1212
• helm chart for v0.12.0-alpha by @y-f-u in https://github.com/spiceai/spiceai/pull/1235
• Update acknowledgements by @github-actions in https://github.com/spiceai/spiceai/pull/1232
• Bump spiceai version to v0.12.1-alpha by @ewgenius in https://github.com/spiceai/spiceai/pull/1239
• Update ROADMAP.md - remove v0.12.0-alpha by @ewgenius in https://github.com/spiceai/spiceai/pull/1241
• Raise errors in InsertBuilder by @Jeadie in https://github.com/spiceai/spiceai/pull/1242
• Update endgame template by @ewgenius in https://github.com/spiceai/spiceai/pull/1240
• Add E2E tests for acceleration engines types support by @sgrebnov in https://github.com/spiceai/spiceai/pull/1218
• Stream blocks to arrow by @gloomweaver in https://github.com/spiceai/spiceai/pull/1203
• Update enhancement.md to include a checklist item have a release notes entry for each enhancement. by @digadeesh in https://github.com/spiceai/spiceai/pull/1245
• arrow_sql_gen data column conversion by @Sevenannn in https://github.com/spiceai/spiceai/pull/1230
• Implement the Localhost Data Connector & fix DoPut by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1266
• Update postgres parameter check by @Sevenannn in https://github.com/spiceai/spiceai/pull/1244
• Record batch casting to fix SQLite data type issues by @y-f-u in https://github.com/spiceai/spiceai/pull/1261
• typo fix on Decimal in postgres arrow_sql_gen by @y-f-u in https://github.com/spiceai/spiceai/pull/1277
• Move verify_schema to arrow_tools by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1284
• Support UUID and TimestampTZ type for Postgres as Data Connector by @ahirner & @y-f-u https://github.com/spiceai/spiceai/pull/1276
• Fix linter warnings by @ewgenius in https://github.com/spiceai/spiceai/pull/1286
• Add Snowflake data connector by @sgrebnov in https://github.com/spiceai/spiceai/pull/1278
• Add Snowflake login support (username and password) by @sgrebnov in https://github.com/spiceai/spiceai/pull/1272
• convert timestamp properly in sql gen by @y-f-u in https://github.com/spiceai/spiceai/pull/1291
• Add if not exists clause to create statement on when creating a table using duckdb acceleration. by @digadeesh in https://github.com/spiceai/spiceai/pull/1290
• Disable DML & DDL queries in the public SQL interface by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1294
• Refactor duckdb to properly set access_mode for connection by @ewgenius in https://github.com/spiceai/spiceai/pull/1285
• do not insert batch for sqlite and postgres if no records in the record batch by @y-f-u in https://github.com/spiceai/spiceai/pull/1293
• Postgres - add custom error message for invalid error table by @ewgenius in https://github.com/spiceai/spiceai/pull/1295
• SQLite/Accelerators handle null values by @gloomweaver in https://github.com/spiceai/spiceai/pull/1298
• Add command to attach to running process by @gloomweaver in https://github.com/spiceai/spiceai/pull/1297
• Use the GITHUB_TOKEN environment variable in the installation script, if available, to avoid rate limiting in CI workflows by @ewgenius in https://github.com/spiceai/spiceai/pull/1302
• Fix unsupported SSL mode options for PostgreSQL connection string by @ewgenius in https://github.com/spiceai/spiceai/pull/1300
• Add CLI cmd spice login spark by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1303
• Check only the latest published release to avoid installing pre-release versions by @ewgenius in https://github.com/spiceai/spiceai/pull/1301
• Postgres data connector - handle invalid host/port and username/password errors by @ewgenius in https://github.com/spiceai/spiceai/pull/1292
• Fix the panic on bad clickhouse connection by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1306
• Improve Snowflake Data Connector by @sgrebnov https://github.com/spiceai/spiceai/pull/1296

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.12.0-alpha...v0.12.1-alpha

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

• Twitter: @spice_ai
• Slack: spiceai.org/slack
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: hey@spice.ai

The v0.12-alpha release introduces Clickhouse and Apache Spark data connectors, adds support for limiting refresh data periods for temporal datasets, and includes upgraded Spice Client SDKs compatible with Spice OSS.

Highlights​

• Clickhouse data connector: Use Clickhouse as a data source with the clickhouse: scheme.

• Apache Spark Connect data connector: Use Apache Spark Connect connections as a data source using the spark: scheme.

• Refresh data window: Limit accelerated dataset data refreshes to the specified window, as a duration from now configuration setting, for faster and more efficient refreshes.

• ODBC data connector: Use ODBC connections as a data source using the odbc: scheme. The ODBC data connector is currently optional and not included in default builds. It can be conditionally compiled using the odbc cargo feature when building from source.

• Spice Client SDK Support: The official Spice SDKs have been upgraded with support for Spice OSS.

  • spice-rs v2.0.0
  • spice.js v2.0.0
  • gospice v6.0.0
  • spicepy v2.0.0

Breaking Changes​

• Refresh interval: The refresh_interval acceleration setting and been changed to refresh_check_interval to make it clearer it is the check versus the data interval.

Contributors​

• @phillipleblanc
• @Jeadie
• @ewgenius
• @sgrebnov
• @y-f-u
• @lukekim
• @digadeesh
• @gloomweaver
• @edmondop
• @mach-kernel

New Contributors​

• Thanks to @mach-kernel who made their first contribution in https://github.com/spiceai/spiceai/pull/1204 by adding the ODBC data connector!

What's Changed​

• Update helm version by @Jeadie in https://github.com/spiceai/spiceai/pull/1167
• Handle and trace errors in secret stores by @ewgenius in https://github.com/spiceai/spiceai/pull/1149
• bump the release versions to 0.12.0 by @y-f-u in https://github.com/spiceai/spiceai/pull/1171
• Don't fail acknowledgments flow if no changes detected by @ewgenius in https://github.com/spiceai/spiceai/pull/1170
• Allow Spice CLI to control runtime installation on Windows by @sgrebnov in https://github.com/spiceai/spiceai/pull/1173
• Allow SELECT count(*) for Sqlite Data Accelerator by @sgrebnov in https://github.com/spiceai/spiceai/pull/1166
• add refresh_period param in acceleration by @y-f-u in https://github.com/spiceai/spiceai/pull/1180
• Properly support Spark Connect filter pushdown by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1186
• Avoid rate-limiting on arduino/setup-protoc@v3 by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1189
• Clickhouse DataConnector base implementation by @gloomweaver in https://github.com/spiceai/spiceai/pull/1168
• rename refresh_interval to refresh_check_interval by @y-f-u in https://github.com/spiceai/spiceai/pull/1190
• Fix timestamp & add support for Decimal to Databricks/Spark by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1194
• Convert temporal column and refresh period to datafusion expr by @y-f-u in https://github.com/spiceai/spiceai/pull/1187
• Hot reload accelerated table on dataset update by @ewgenius in https://github.com/spiceai/spiceai/pull/1195
• Upgrade DataFusion to 37.1 & DuckDB to 10.2 by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1200
• Update version.txt for 0.11.2 release by @digadeesh in https://github.com/spiceai/spiceai/pull/1199
• Clickhouse E2E by @gloomweaver in https://github.com/spiceai/spiceai/pull/1193
• Clickhouse: fix darwin ci pipeline by @gloomweaver in https://github.com/spiceai/spiceai/pull/1201
• Add table_type to show tables in Spice SQL & update next version to v0.12.0-alpha by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1206
• print WARN if time_column does not exists in federated schema by @y-f-u in https://github.com/spiceai/spiceai/pull/1207
• Add FallbackOnZeroResultsScanExec for executing an input ExecutionPlan and optionally falling back to a TableProvider.scan() if the input has zero results by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1196
• Clickhouse refactor connection code and set secure option by @gloomweaver in https://github.com/spiceai/spiceai/pull/1198
• E2E: reusable Spice installation by @sgrebnov in https://github.com/spiceai/spiceai/pull/1205
• Clickhouse block_to_arrow unit test by @gloomweaver in https://github.com/spiceai/spiceai/pull/1202
• rename refresh_period to refresh_data_period by @y-f-u in https://github.com/spiceai/spiceai/pull/1210
• Refactor E2E tests: dataset verification and PostgreSQL installation by @sgrebnov in https://github.com/spiceai/spiceai/pull/1211
• Add BI dashboard acceleration video to README.md by @lukekim in https://github.com/spiceai/spiceai/pull/1219
• Improve clarity and consistency of output messages by @lukekim in https://github.com/spiceai/spiceai/pull/1214
• Update ROADMAP Apr 29, 2024 by @lukekim in https://github.com/spiceai/spiceai/pull/1220
• Stand-alone Spark Connect: Isolate Spark Connect from Databricks Connect to make it reusable by @edmondop in https://github.com/spiceai/spiceai/pull/1213
• Optimize build time in dev mode by @gloomweaver in https://github.com/spiceai/spiceai/pull/1215
• Feature: Support ODBC reads using unixodbc by @mach-kernel in https://github.com/spiceai/spiceai/pull/1204
• Use non-fork deltalake by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1223
• Support Date32 & Date64 in arrow_sql_gen by @Jeadie in https://github.com/spiceai/spiceai/pull/1217
• Update REPL output to be consistent with the latest Spice version by @sgrebnov in https://github.com/spiceai/spiceai/pull/1231
• rename refresh_data_period to refresh_data_window by @y-f-u in https://github.com/spiceai/spiceai/pull/1233
• Update README.md to include ODBC, Spark Connect, and Clickhouse data connectors in support data connector matrix. by @digadeesh in https://github.com/spiceai/spiceai/pull/1234

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.11.1-alpha...v0.12.0-alpha

Resources​

• Getting started with Spice.ai
• Documentation

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

• Twitter: @spice_ai
• Slack: spiceai.org/slack
• Telegram: Spice AI Discussion
• Reddit: https://www.reddit.com/r/spiceai
• Email: hey@spice.ai