Skip to main content

Spice v0.11.1-alpha (April 22, 2024)

· 3 min read
Luke Kim
Luke Kim
Founder and CEO of Spice AI

The v0.11.1-alpha release introduces retention policies for accelerated datasets, native Windows installation support, and integration of catalog and schema settings for the Databricks Spark connector. Several bugs have also been fixed for improved stability.

Highlights​

  • Retention Policies for Accelerated Datasets: Automatic eviction of data from accelerated time-series datasets when a specified temporal column exceeds the retention period, optimizing resource utilization.

  • Windows Installation Support: Native Windows installation support, including upgrades.

  • Databricks Spark Connect Catalog and Schema Settings: Improved translation between DataFusion and Spark, providing better Spark Catalog support.

Contributors​

  • @phillipleblanc
  • @Jeadie
  • @ewgenius
  • @sgrebnov
  • @y-f-u
  • @lukekim
  • @digadeesh
  • @Sevenannn
  • @gloomweaver

New in this release​

What's Changed​

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.11.0-alpha...v0.11.1-alpha

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Spice v0.11-alpha (April 15, 2024)

· 4 min read
Sergei Grebnov
Sergei Grebnov
Senior Software Engineer at Spice AI

The Spice v0.11-alpha release significantly improves the Databricks data connector with Databricks Connect (Spark Connect) support, adds the DuckDB data connector, and adds the AWS Secrets Manager secret store. In addition, enhanced control over accelerated dataset refreshes, improved SSL security for MySQL and PostgreSQL connections, and overall stability improvements have been added.

Highlights in v0.11-alpha​

DuckDB data connector: Use DuckDB databases or connections as a data source.

AWS Secrets Manager Secret Store: Use AWS Secrets Managers as a secret store.

Custom Refresh SQL: Specify a custom SQL query for dataset refresh using refresh_sql.

Dataset Refresh API: Trigger a dataset refresh using the new CLI command spice refresh or via API.

Expanded SSL support for Postgres: SSL mode now supports disable, require, prefer, verify-ca, verify-full options with the default mode changed to require. Added pg_sslrootcert parameter for setting a custom root certificate and the pg_insecure parameter is no longer supported.

Databricks Connect: Choose between using Spark Connect or Delta Lake when using the Databricks data connector for improved performance.

Improved SSL support for Postgres: ssl mode now supports disable, require, prefer, verify-ca, verify-full options with default mode changed to require. Added pg_sslrootcert parameter to allow setting custom root cert for postgres connector, pg_insecure parameter is no longer supported as redundant.

Internal architecture refactor: The internal architecture of spiced was refactored to simplify the creation data components and to improve alignment with DataFusion concepts.

New Contributors​

@edmondop's first contribution github.com/spiceai/spiceai/pull/1110!

Contributors​

  • @phillipleblanc
  • @Jeadie
  • @ewgenius
  • @sgrebnov
  • @y-f-u
  • @lukekim
  • @digadeesh
  • @Sevenannn
  • @gloomweaver
  • @ahirner

New in this release​

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.10.2-alpha...v0.11.0-alpha

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Spice.ai v0.10.2-alpha

· 2 min read
Jack Eadie
Jack Eadie
Token Plumber at Spice AI

Announcing the release of Spice v0.10.2-alpha (Apr 9, 2024)! 🔥

The v0.10.2-alpha release adds the MySQL data connector and makes external data connections more robust on initialization.

Highlights in v0.10.2-alpha​

  • MySQL data connector: Connect to any MySQL server, including SSL support.

  • Data connections verified at initialization: Verify endpoints and authorization for external data connections (e.g. databricks, spice.ai) at initialization.

New Contributors​

Contributors​

  • @phillipleblanc
  • @y-f-u
  • @ewgenius
  • @sgrebnov
  • @lukekim
  • @digadeesh
  • @jeadie

New in this release​

Full Changelog: https://github.com/spiceai/spiceai/compare/v0.10.1-alpha...v0.10.2-alpha

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Spice.ai v0.10.1-alpha

· 2 min read
Luke Kim
Luke Kim
Founder and CEO of Spice AI

Announcing the release of Spice v0.10.1-alpha! 🔥

The v0.10.1-alpha release focuses on stability, bug fixes, and usability by improving error messages when using SQLite data accelerators, improving the PostgreSQL support, and adding a basic Helm chart.

Highlights in v0.10.1-alpha​

Improved PostgreSQL support for Data Connectors TLS is now supported with PostgreSQL Data Connectors and there is improved VARCHAR and BPCHAR conversions through Spice.

Improved Error messages Simplified error messages from Spice when propagating errors from Data Connectors and Accelerator Engines.

Spice Pods Command The spice pods command can give you quick statistics about models, dependencies, and datasets that are loaded by the Spice runtime.

Kubernetes Helm Deployment​

Spice.ai can be deployed to Kubernetes using Helm. Here's a quick guide to get started:

Step 1. (Optional) Start a local kind cluster:

go install sigs.k8s.io/kind@v0.22.0
kind create cluster

Step 2. Install Spice in your Kubernetes cluster using Helm:

helm repo add spiceai https://helm.spiceai.org
helm install spiceai spiceai/spiceai

Step 3. Verify that the Spice pods are running:

kubectl get pods
kubectl logs deploy/spiceai

Step 4. Run the Spice SQL REPL inside the running pod:

kubectl exec -it deploy/spiceai -- spiced --repl

Learn more about deploying Spice.ai to Kubernetes

Contributors​

  • @phillipleblanc
  • @mitchdevenport
  • @ewgenius
  • @sgrebnov
  • @lukekim
  • @digadeesh

New in this release​

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Spice.ai v0.10-alpha

· 2 min read
Phillip LeBlanc
Phillip LeBlanc
Co-Founder and CTO of Spice AI

Announcing the release of Spice v0.10-alpha! 🧙‍♂️

The Spice.ai v0.10-alpha release focused on additions and updates to improve stability, usability, and the overall Spice developer experience.

Highlights in v0.10-alpha​

Public Bucket Support for S3 Data Connector: The S3 Data Connector now supports public buckets in addition to buckets requiring an access id and key.

JDBC-Client Connectivity: Improved connectivity for JDBC clients, like Tableau.

User Experience Improvements:

  • Friendlier error messages across the board to make debugging and development better.
  • Added a spice login postgres command, streamlining the process for connecting to PostgreSQL databases.
  • Added PostgreSQL connection verification and connection string support, enhancing usability for PostgreSQL users.

Grafana Dashboard: Improving the ability to monitor Spice deployments, a standard Grafana dashboard is now available.

Contributors​

  • @phillipleblanc
  • @mitchdevenport
  • @Jeadie
  • @ewgenius
  • @sgrebnov
  • @y-f-u
  • @lukekim
  • @digadeesh

New in this release​

  • Fixes Gracefully handle Arrow Flight DoExchange connection resets
  • Adds Grafana Dashboard
  • Adds Flight SQL CommandGetTableTypes Command support (improves JDBC-client connectivity)
  • Adds Friendlier error messages
  • Adds spice login postgres command
  • Adds PostgreSQL connection verification
  • Adds PostgreSQL connection string support
  • Adds Linux aarch64 build
  • Updates Improves spice status with dataset metrics
  • Updates CLI REPL improved show tables output
  • Updates CLI REPL limit output to 500 rows
  • Updates Improved README.md with architecture diagram updates
  • Updates Improved CI run time.
  • Updates Use macOS hosted Actions runner

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Spice.ai v0.9.1-alpha

· 2 min read
Luke Kim
Luke Kim
Founder and CEO of Spice AI

Announcing the release of Spice v0.9.1-alpha! 🧙‍♂️

The v0.9.1 release focused on stability, bug fixes, and usability by adding spice CLI commands for listing Spicepods (spice pods), Models (spice models), Datasets (spice datasets), and improved status (spice status) details. In addition, the Arrow Flight SQL (flightsql) data connector and SQLite (sqlite) data store were added.

Highlights in v0.9.1-alpha​

FlightSQL data connector: Arrow Flight SQL can now be used as a connector for federated SQL query.

SQLite data backend: SQLite can now be used as a data store for acceleration.

Contributors​

  • @phillipleblanc
  • @mitchdevenport
  • @Jeadie
  • @ewgenius
  • @sgrebnov
  • @y-f-u
  • @lukekim

New in this release​

  • Adds FlightSQL data connector (flightsql).
  • Adds SQLite data store, supports both in-memory and file based (sqlite).
  • Adds support for date, varchar, bpchar, and primitive list types for the PostgreSQL data connector and data store.
  • Adds spice pods, spice status, spice datasets, and spice models CLI commands.
  • Adds GET /v1/spicepods API for listing loaded Spicepods.
  • Adds spiced Docker CI build and release.
  • Adds E2E tests for release installation and local acceleration.
  • Adds E2E tests and instructions to run basic TPC-H benchmark tests.
  • Adds linux/arm64 binary build.
  • Fixes spice sql REPL panics when query result is too large. (https://github.com/spiceai/spiceai/pull/875)
  • Fixes --access-secret in spice s3 login. (https://github.com/spiceai/spiceai/pull/894)
  • Fixes version check upgrade logic.

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Spice.ai v0.9-alpha

· 2 min read
Luke Kim
Luke Kim
Founder and CEO of Spice AI

Announcing the release of Spice v0.9-alpha! 🧙‍♂️

The v0.9 release adds several data connectors including the Spice data connector for the ability to connect to other Spice instances. Improved observability for the Spice runtime has been added with the new /metrics endpoint for monitoring deployed instances.

Highlights in v0.9-alpha​

Arrow Flight SQL endpoint: The Arrow Flight endpoint now supports Flight SQL, including JDBC, ODBC, and ADBC enabling database clients like DBeaver or BI applications like Tableau to connect to and query the Spice runtime.

Spice.ai data connector: Use other Spice runtime instances as data connectors for federated SQL query across Spice deployments and for chaining Spice runtimes.

Keyring secret store: Use the operating system native credential store, like macOS keychain for storing secrets used by the Spice runtime.

PostgreSQL data connector: PostgreSQL can now be used as both a data store for acceleration and as a connector for federated SQL query.

Databricks data connector: Databricks as a connector for federated SQL query across Delta Lake tables.

S3 data connector: S3 as a connector for federated SQL query across Parquet files stored in S3.

Metrics endpoint: Added new /metrics endpoint for Spice runtime observability and monitoring with the following metrics:

- spiced_runtime_http_server_start counter
- spiced_runtime_flight_server_start counter
- datasets_count gauge
- load_dataset summary
- load_secrets summary
- datasets/load_error counter
- datasets/count counter
- models/load_error counter
- models/count counter

Contributors​

New in this release​

  • Adds Keyring secret store (keyring).
  • Adds PostgreSQL data connector (postgres).
  • Adds Spice.ai data connector (spiceai).
  • Adds Arrow Flight SQL (JDBC/ODBC/ADBC) support.
  • Adds Databricks data connector (databricks) - Delta Lake support.
  • Adds S3 data connector (s3) - Parquet support.
  • Adds /v1/models API.
  • Adds /v1/status API.
  • Adds /metrics API.

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Spice.ai v0.8-alpha

· One min read
Luke Kim
Luke Kim
Founder and CEO of Spice AI

Announcing the release of Spice v0.8-alpha! 🏹

This is a minor release that builds on the new Rust-based runtime, adding stability and a preview of new features for the first major release.

Highlights in v0.8-alpha​

Secrets management: Spice 0.8 runtime can now configure and retrieve secrets from local environment variables and in a Kubernetes cluster.

Data tables can be locally accelerated using PostgreSQL

New in this release​

  • Adds Secrets management in local environment variables and Kubernetes clusters.
  • Adds (Preview) PostgreSQL as a data table acceleration engine.

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Spice.ai v0.7-alpha

· 2 min read
Phillip LeBlanc
Phillip LeBlanc
Co-Founder and CTO of Spice AI

Announcing the release of Spice v0.7-alpha! 🏹

Spice v0.7-alpha is an all new implementation of Spice written in Rust. The Spice v0.7 runtime provides developers with a unified SQL query interface to locally accelerate and query data tables sourced from any database, data warehouse, or data lake.

Learn more and get started in minutes with the updated Quickstart in the repository README!

Highlights in v0.7-alpha​

DataFusion SQL Query Engine: Spice v0.7 leverages the Apache DataFusion query engine to provide very fast, high quality SQL query across one or more local or remote data sources.

Data tables can be locally accelerated using Apache Arrow in-memory or by DuckDB.

New in this release​

  • Adds runtime rewritten in Rust for high-performance.
  • Adds Apache DataFusion SQL query engine.
  • Adds The Spice.ai platform as a data source.
  • Adds Dremio as a data source.
  • Adds OpenTelemetry (OTEL) collector.
  • Adds local data table acceleration.
  • Adds DuckDB file or in-memory as a data table acceleration engine.
  • Adds In-memory Apache Arrow as a data table acceleration engine.
  • Removes the built-in AI training engine; now cloud-based and provided by the Spice.ai platform.
  • Removes the built-in dashboard and web-interface; now cloud-based and provided by the Spice.ai platform.

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved.

Spice.ai v0.6.1-alpha

· 2 min read
Luke Kim
Luke Kim
Founder and CEO of Spice AI

Announcing the release of Spice.ai v0.6.1-alpha! 🌶

Building upon the Apache Arrow support in v0.6-alpha, Spice.ai now includes new Apache Arrow data processor and Apache Arrow Flight data connector components! Together, these create a high-performance bulk-data transport directly into the Spice.ai ML engine. Coupled with big data systems from the Apache Arrow ecosystem like Hive, Drill, Spark, Snowflake, and BigQuery, it's now easier than ever to combine big data with Spice.ai.

And we're also excited to announce the release of Spice.xyz! 🎉

Spice.xyz is data and AI infrastructure for web3. It’s web3 data made easy. Insanely fast and purpose designed for applications and ML.

Spice.xyz delivers data in Apache Arrow format, over high-performance Apache Arrow Flight APIs to your application, notebook, ML pipeline, and of course through these new data components, to the Spice.ai runtime.

Read the announcement post at blog.spice.ai.

Spice.xyz

New in this release​

Now built with Go 1.18.

Dependency updates​

  • Updates to React 18
  • Updates to CRA 5
  • Updates to Glide DataGrid 4
  • Updates to SWR 1.2
  • Updates to TypeScript 4.6

Resources​

Community​

Spice.ai started with the vision to make AI easy for developers. We are building Spice.ai in the open and with the community. Reach out on Slack or by email to get involved. We will also be starting a community call series soon!