Merge pull request #3062 from perspective-dev/duckdb

DuckDB Virtual Server

Author: texodus

.github/workflows/build.yaml

@@ -73,7 +73,7 @@ jobs:
                     - ubuntu-22.04
                 python-version:
                     - 3.9
-                node-version: [20.x]
+                node-version: [22.x]
 
         steps:
             - name: Checkout
@@ -125,7 +125,7 @@ jobs:
                     - ubuntu-22.04
                 python-version:
                     - 3.9
-                node-version: [20.x]
+                node-version: [22.x]
 
         steps:
             - name: Checkout
@@ -193,7 +193,7 @@ jobs:
                     - x86_64
                 python-version:
                     - 3.9
-                node-version: [20.x]
+                node-version: [22.x]
                 include:
                     - os: ubuntu-22.04
                       arch: x86_64
@@ -305,7 +305,7 @@ jobs:
                     - windows-2022
                 arch:
                     - x86_64
-                node-version: [20.x]
+                node-version: [22.x]
                 is-release:
                     - ${{ startsWith(github.ref, 'refs/tags/v') || github.ref_name == 'master' }}
                 exclude:
@@ -395,7 +395,7 @@ jobs:
                     - x86_64
                 python-version:
                     - 3.9
-                node-version: [20.x]
+                node-version: [22.x]
         steps:
             - name: Checkout
               uses: actions/checkout@v4
@@ -443,7 +443,7 @@ jobs:
                     - x86_64
                 python-version:
                     - 3.9
-                node-version: [20.x]
+                node-version: [22.x]
         steps:
             - name: Checkout
               uses: actions/checkout@v4
@@ -510,7 +510,7 @@ jobs:
                     - x86_64
                 python-version:
                     - 3.9
-                node-version: [20.x]
+                node-version: [22.x]
                 is-release:
                     - ${{ startsWith(github.ref, 'refs/tags/v') || github.ref_name == 'master' }}
                 exclude:
@@ -621,7 +621,7 @@ jobs:
                     - ubuntu-22.04
                 python-version:
                     - 3.9
-                node-version: [20.x]
+                node-version: [22.x]
 
         steps:
             - name: Checkout
@@ -674,7 +674,7 @@ jobs:
                 python-version:
                     - 3.9
                     # - 3.12
-                node-version: [20.x]
+                node-version: [22.x]
                 is-release:
                     - ${{ startsWith(github.ref, 'refs/tags/v') || github.ref_name == 'master' }}
                 exclude:
@@ -823,7 +823,7 @@ jobs:
             matrix:
                 os: [ubuntu-22.04]
                 python-version: [3.9]
-                node-version: [20.x]
+                node-version: [22.x]
                 arch: [x86_64]
         steps:
             - name: Checkout
@@ -880,7 +880,7 @@ jobs:
         strategy:
             matrix:
                 os: [ubuntu-22.04]
-                node-version: [20.x]
+                node-version: [22.x]
         runs-on: ${{ matrix.os }}
         steps:
             - name: Checkout

Cargo.lock

@@ -1,4 +1,5 @@
 <br />
+
 <a href="https://perspective-dev.github.io">
 <picture>
 <source media="(prefers-color-scheme: dark)" srcset="https://github.com/perspective-dev/perspective/raw/master/docs/static/svg/perspective-logo-dark.svg?raw=true">
@@ -14,27 +15,33 @@
 
 <br/>
 
-Perspective is an <i>interactive</i> analytics and data visualization component,
-which is especially well-suited for <i>large</i> and/or <i>streaming</i>
-datasets. Use it to create user-configurable reports, dashboards, notebooks and
-applications.
+Perspective is an interactive analytics and data visualization component for
+large and streaming datasets. Build user-configurable reports, dashboards,
+notebooks, and applications with a high-performance query engine compiled to
+WebAssembly, Python, and Rust.
 
 ### Features
 
-- A fast, memory efficient streaming query engine, written in C++ and compiled
-  for [WebAssembly](https://webassembly.org/), [Python](https://www.python.org/)
-  and [Rust](https://www.rust-lang.org/), with read/write/streaming for
-  [Apache Arrow](https://arrow.apache.org/), and a high-performance columnar
-  expression language based on [ExprTK](https://github.com/ArashPartow/exprtk).
-
-- A framework-agnostic User Interface packaged as a
+- A framework-agnostic user interface packaged as a
   [Custom Element](https://developer.mozilla.org/en-US/docs/Web/Web_Components/Using_custom_elements),
-  powered either in-browser via WebAssembly or virtually via WebSocket server
-  (Python/Node/Rust).
+  which connects to a Data Model in-browser (via WebAssembly) or remotely (via
+  WebSocket, with integration in Python, Node.js and Rust). Includes a data
+  grid, 10+ chart types line, bar, area, scatter, heatmap, treemap, sunburst,
+  candlestick, and more.
+
+- A Data Model API for pluggable engines, enabling Perspective's UI to query
+  external data sources like [DuckDB](https://duckdb.org/) while translating
+  view configurations into native queries.
+
+- A fast, memory-efficient streaming Data Model built-in, written in C++ and
+  compiled for [WebAssembly](https://webassembly.org/),
+  [Python](https://www.python.org/), and [Rust](https://www.rust-lang.org/).
+  Supports read/write/streaming for [Apache Arrow](https://arrow.apache.org/),
+  with a columnar expression language based on
+  [ExprTK](https://github.com/ArashPartow/exprtk).
 
-- A [JupyterLab](https://jupyter.org/) widget and Python client library, for
-  interactive data analysis in a notebook, as well as _scalable_ production
-  applications.
+- A [JupyterLab](https://jupyter.org/) widget and Python client library for
+  interactive data analysis in notebooks.
 
 ### Documentation
 
@@ -62,7 +69,7 @@ applications.
 ### Examples
 
 <!-- Examples -->
-<table><tbody><tr><td>editable</td><td>file</td><td>fractal</td></tr><tr><td><a href="https://perspective-dev.github.io/block?example=editable"><img height="125" src="https://perspective-dev.github.io/blocks/editable/preview.png?"></img></a></td><td><a href="https://perspective-dev.github.io/block?example=file"><img height="125" src="https://perspective-dev.github.io/blocks/file/preview.png?"></img></a></td><td><a href="https://perspective-dev.github.io/block?example=fractal"><img height="125" src="https://perspective-dev.github.io/blocks/fractal/preview.png?"></img></a></td></tr><tr><td>market</td><td>raycasting</td><td>evictions</td></tr><tr><td><a href="https://perspective-dev.github.io/block?example=market"><img height="125" src="https://perspective-dev.github.io/blocks/market/preview.png?"></img></a></td><td><a href="https://perspective-dev.github.io/block?example=raycasting"><img height="125" src="https://perspective-dev.github.io/blocks/raycasting/preview.png?"></img></a></td><td><a href="https://perspective-dev.github.io/block?example=evictions"><img height="125" src="https://perspective-dev.github.io/blocks/evictions/preview.png?"></img></a></td></tr><tr><td>nypd</td><td>streaming</td><td>covid</td></tr><tr><td><a href="https://perspective-dev.github.io/block?example=nypd"><img height="125" src="https://perspective-dev.github.io/blocks/nypd/preview.png?"></img></a></td><td><a href="https://perspective-dev.github.io/block?example=streaming"><img height="125" src="https://perspective-dev.github.io/blocks/streaming/preview.png?"></img></a></td><td><a href="https://perspective-dev.github.io/block?example=covid"><img height="125" src="https://perspective-dev.github.io/blocks/covid/preview.png?"></img></a></td></tr><tr><td>webcam</td><td>movies</td><td>superstore</td></tr><tr><td><a href="https://perspective-dev.github.io/block?example=webcam"><img height="125" src="https://perspective-dev.github.io/blocks/webcam/preview.png?"></img></a></td><td><a href="https://perspective-dev.github.io/block?example=movies"><img height="125" src="https://perspective-dev.github.io/blocks/movies/preview.png?"></img></a></td><td><a href="https://perspective-dev.github.io/block?example=superstore"><img height="125" src="https://perspective-dev.github.io/blocks/superstore/preview.png?"></img></a></td></tr><tr><td>citibike</td><td>olympics</td><td>dataset</td></tr><tr><td><a href="https://perspective-dev.github.io/block?example=citibike"><img height="125" src="https://perspective-dev.github.io/blocks/citibike/preview.png?"></img></a></td><td><a href="https://perspective-dev.github.io/block?example=olympics"><img height="125" src="https://perspective-dev.github.io/blocks/olympics/preview.png?"></img></a></td><td><a href="https://perspective-dev.github.io/block?example=dataset"><img height="125" src="https://perspective-dev.github.io/blocks/dataset/preview.png?"></img></a></td></tr></tbody></table>
+<table><tbody><tr><td>editable</td><td>file</td><td>duckdb</td></tr><tr><td><a href="https://perspective-dev.github.io/block?example=editable"><img height="125" src="https://perspective-dev.github.io/blocks/editable/preview.png?"></img></a></td><td><a href="https://perspective-dev.github.io/block?example=file"><img height="125" src="https://perspective-dev.github.io/blocks/file/preview.png?"></img></a></td><td><a href="https://perspective-dev.github.io/block?example=duckdb"><img height="125" src="https://perspective-dev.github.io/blocks/duckdb/preview.png?"></img></a></td></tr><tr><td>fractal</td><td>market</td><td>raycasting</td></tr><tr><td><a href="https://perspective-dev.github.io/block?example=fractal"><img height="125" src="https://perspective-dev.github.io/blocks/fractal/preview.png?"></img></a></td><td><a href="https://perspective-dev.github.io/block?example=market"><img height="125" src="https://perspective-dev.github.io/blocks/market/preview.png?"></img></a></td><td><a href="https://perspective-dev.github.io/block?example=raycasting"><img height="125" src="https://perspective-dev.github.io/blocks/raycasting/preview.png?"></img></a></td></tr><tr><td>evictions</td><td>nypd</td><td>streaming</td></tr><tr><td><a href="https://perspective-dev.github.io/block?example=evictions"><img height="125" src="https://perspective-dev.github.io/blocks/evictions/preview.png?"></img></a></td><td><a href="https://perspective-dev.github.io/block?example=nypd"><img height="125" src="https://perspective-dev.github.io/blocks/nypd/preview.png?"></img></a></td><td><a href="https://perspective-dev.github.io/block?example=streaming"><img height="125" src="https://perspective-dev.github.io/blocks/streaming/preview.png?"></img></a></td></tr><tr><td>covid</td><td>webcam</td><td>movies</td></tr><tr><td><a href="https://perspective-dev.github.io/block?example=covid"><img height="125" src="https://perspective-dev.github.io/blocks/covid/preview.png?"></img></a></td><td><a href="https://perspective-dev.github.io/block?example=webcam"><img height="125" src="https://perspective-dev.github.io/blocks/webcam/preview.png?"></img></a></td><td><a href="https://perspective-dev.github.io/block?example=movies"><img height="125" src="https://perspective-dev.github.io/blocks/movies/preview.png?"></img></a></td></tr><tr><td>superstore</td><td>citibike</td><td>olympics</td></tr><tr><td><a href="https://perspective-dev.github.io/block?example=superstore"><img height="125" src="https://perspective-dev.github.io/blocks/superstore/preview.png?"></img></a></td><td><a href="https://perspective-dev.github.io/block?example=citibike"><img height="125" src="https://perspective-dev.github.io/blocks/citibike/preview.png?"></img></a></td><td><a href="https://perspective-dev.github.io/block?example=olympics"><img height="125" src="https://perspective-dev.github.io/blocks/olympics/preview.png?"></img></a></td></tr><tr><td>dataset</td></tr><tr><td><a href="https://perspective-dev.github.io/block?example=dataset"><img height="125" src="https://perspective-dev.github.io/blocks/dataset/preview.png?"></img></a></td></tr></tbody></table>
 <!-- Examples -->
 
 ### Media

README.md

@@ -12,6 +12,13 @@ In order to _run_ a project in this directory as written:
 2. Run the project with `pnpm run start $PROJECT_NAME` from the repository root
    (_not_ the `/examples` directory).
 
+## VirtualServer Examples
+
+These examples demonstrate custom data source implementations using the VirtualServer API, which allows you to create backends that serve data from any source without loading it into Perspective's tables:
+
+- **[nodejs-virtual-server](nodejs-virtual-server/)** - Node.js example with in-memory data and WebSocket server
+- **[python-duckdb-virtual](python-duckdb-virtual/)** - Python example using DuckDB as a data source
+
 # Optional
 
 Generally, the changes necessary to make these examples run _without_ the

examples/README.md

@@ -13,6 +13,7 @@
 const LOCAL_EXAMPLES = [
     "editable",
     "file",
+    "duckdb",
     "fractal",
     "market",
     "raycasting",

examples/blocks/examples.js

@@ -4,7 +4,7 @@
     "version": "4.0.1",
     "description": "A collection of simple client-side Perspective examples for `http://bl.ocks.org`.",
     "scripts": {
-        "start": "mkdir -p dist && node --experimental-wasm-memory64 --experimental-modules server.mjs",
+        "start": "mkdir -p dist && node --experimental-modules server.mjs",
         "repl": "node --experimental-repl-await"
     },
     "main": "index.mjs",

examples/blocks/package.json

@@ -0,0 +1,15 @@
+An example of [Perspective](https://github.com/perspective-dev/perspective)
+using [DuckDB WASM](https://duckdb.org/docs/api/wasm/overview) as a virtual
+server backend via the `DuckDBHandler` adapter.
+
+Instead of using Perspective's built-in WebAssembly query engine, this example
+demonstrates how to use DuckDB as the data processing layer while still
+leveraging Perspective's visualization components. The `DuckDBHandler` translates
+Perspective's view configuration (group by, split by, sort, filter, expressions,
+aggregates) into DuckDB SQL queries, enabling Perspective to query data stored
+in DuckDB tables.
+
+This example loads the Superstore sample dataset into a DuckDB table, then
+creates a Perspective viewer that queries the data through the DuckDB virtual
+server. A separate log viewer displays the SQL queries being generated in
+real-time, along with a timeline chart showing query frequency.

examples/blocks/src/duckdb/README.md

@@ -0,0 +1,40 @@
+<!doctype html>
+<html>
+    <head>
+        <meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1,minimum-scale=1,user-scalable=no" />
+        <link rel="preload" href="/node_modules/@perspective-dev/server/dist/wasm/perspective-server.wasm" as="fetch" type="application/wasm" crossorigin="anonymous" />
+        <link rel="preload" href="/node_modules/@perspective-dev/viewer/dist/wasm/perspective-viewer.wasm" as="fetch" type="application/wasm" crossorigin="anonymous" />
+        <link rel="preload" href="/node_modules/superstore-arrow/superstore.lz4.arrow" as="fetch" type="arraybuffer" crossorigin="anonymous" />
+        <link rel="stylesheet" crossorigin="anonymous" href="/node_modules/@perspective-dev/viewer/dist/css/themes.css" />
+        <script type="module" src="index.js"></script>
+        <style>
+            #query {
+                position: absolute;
+                top: 0;
+                left: 0;
+                bottom: 0;
+                right: 30%;
+            }
+
+            #logger {
+                position: absolute;
+                top: 40%;
+                left: 70%;
+                bottom: 0%;
+                right: 0;
+            }
+             #logger2 {
+                position: absolute;
+                top: 0%;
+                left: 70%;
+                bottom: 60%;
+                right: 0;
+            }
+        </style>
+    </head>
+    <body>
+        <perspective-viewer id="query"></perspective-viewer>
+        <perspective-viewer id="logger"></perspective-viewer>
+        <perspective-viewer id="logger2"></perspective-viewer>
+    </body>
+</html>