[Public Alpha] Manage Analytics Buckets from the dashboard

[joshenlim]

Manage Analytics Buckets from the dashboard

We previously announced Supabase Analytics Bucket in Launch Week 15 and we've made much progress since then! You can use these buckets to store large datasets for analytics and reporting 📊

Analytics Buckets is currently in Public Alpha, hence expect rapid changes and possible breaking updates as we expand access to make this increasingly available to everyone. 🙂🙏

Vector Buckets are also now in Public Alpha if you might be interested as well 😄

What we'd like to know from you

• Any bugs / issues that you might run into when using Analytics Buckets
• Any UX friction that you might run into when managing Analytics Buckets through the dashboard
• Any use cases for Analytics Buckets, and how we might be able to improve the DX to support your development / workflow

Related Links

Dashboard: https://supabase.com/dashboard/project/_/storage/analytics

Waitlist: https://forms.supabase.com/analytics-buckets
Youtube: https://www.youtube.com/watch?v=8poJ0f3Ucik
Blog: https://supabase.com/blog/analytics-buckets
Documentation: https://supabase.com/docs/guides/storage/analytics/introduction

[danmandle]

Hi Supabase Team,

We're excited to join the Analytics Buckets alpha. Our use case involves archiving millions of rows of vehicle telemetry data from our vectors table, which includes a PostGIS-generated geometry column:

point extensions.geometry(Point, 4326) GENERATED ALWAYS AS (
  ST_SetSRID(ST_MakePoint(longitude, latitude), 4326)
) STORED

Questions:

1. Does the current alpha ETL pipeline preserve PostGIS geometry columns when replicating to Iceberg tables?
2. Can we query these geometry columns spatially via the Postgres Foreign Data Wrapper (e.g., ST_Within, ST_Distance)?
3. Are the Parquet files compatible with DuckDB's spatial extension for geospatial queries?

Our goal is to archive cold storage data while maintaining geofence query capabilities. Happy to provide feedback as we test!

[iambriccardo]

1. We are not supporting PostGIS directly but we convert any unknown data type to String: https://github.com/supabase/etl/blob/1b99cecf2e91c2b161294306d038f95a7b7008e3/etl/src/conversions/text.rs#L215-L215, if you want you can try to test with ETL locally on your machine how it replicates PostGIS geometry columns. Feel free to open an issue if you then find problems!

[riderx]

very interested to test also for capgo.app we have telemetry of 50M devices a day right now we use Cloudflare analytics engine, to handle the scale but it would be nice to have proper alternative of our OSS users

[ana17777]

Hola tengo ya medio año y no puedo transferir cose cómo me puedan apollar

[aquariuslt]

Hi team,

I've been testing Analytics Buckets on my cloud project and it's working well. I'm now looking to integrate it into our local development workflow.

Question: Are there any plans to provide a Docker service for Analytics Buckets in the local Supabase stack (via supabase start)?

Currently, we use the local Supabase environment for development and testing before deploying to production. Having Analytics Buckets available locally would allow us to:

• Test CDC pipelines and Iceberg integrations in development
• Validate hot/cold storage strategies before production deployment
• Maintain dev/prod parity

Is this on the roadmap, or is there a recommended workaround for local development with Analytics Buckets?

Thanks!

[vinicentus]

I noticed that these vault secrets are created automatically when you create an analytics bucket (called cabinet-telemetry):
cabinet_telemetry_fdw_vault_token
cabinet_telemetry_fdw_vault_aws_s3table_bucket_arn (empty)
cabinet_telemetry_fdw_vault_aws_secret_access_key
cabinet_telemetry_fdw_vault_aws_access_key_id

It would be nice if vault secrets for the rest of the parameters were also automatically created, since they are required in this step Connecting to Iceberg REST Catalog + AWS S3 (or compatible) storage:
cabinet_telemetry_fdw_vault_catalog_uri
cabinet_telemetry_fdw_vault_warehouse_name
cabinet_telemetry_fdw_vault_s3_endpoint
Or maybe they could be easily read from somewhere else, since they seem to depend only on the project reference id, and the warehouse name is the same as the bucket name?

Sidenote: If i manually modify the automatically created vault secrets to DUMMY_VALUE, the modified values are shown in the UI. There seems to be no way to recover the correct settings.

(Don't worry, the project id you see in the picture doens't exist anymore.)

[abdullahiabdulhadi605-netizen]

Nice app
Appreciate it

[iForkU]

Apologies in advance for wall of AI text but i figured claude could explain the issues I ran in much better than I could so here goes:

Bug report: Iceberg FDW wedges the backend when one statement touches two foreign tables

Summary

On a managed Supabase project using the Iceberg Foreign Data Wrapper
(wrappers extension 0.6.1) over an Analytics Bucket, any single SQL
statement that references more than one Iceberg foreign table hangs the
executing backend indefinitely and uninterruptibly. The stuck backend
ignores statement_timeout, pg_cancel_backend(), and
pg_terminate_backend(). The only recovery we found is a full project
restart. We hit this four separate times before isolating the trigger.

Environment

• Managed Supabase Postgres project
• wrappers extension 0.6.1
• Iceberg FDW over a Supabase Analytics Bucket (Apache Iceberg), foreign
  tables in a local iceberg schema, created via the Dashboard
  "Query with Postgres" / Connect flow
• Two foreign tables involved: iceberg.transactions (~232k current rows)
  and iceberg.transaction_line_items (~665k rows)

What triggers it

A statement is safe if and only if it references at most one Iceberg
foreign table. The problem appears whenever a single statement references
two Iceberg foreign tables, regardless of read vs write:

• INSERT INTO iceberg.a SELECT ... FROM iceberg.b (write + read)
• INSERT INTO iceberg.a SELECT ..., (SELECT sum(x) FROM iceberg.b WHERE ...)
  (write + read via scalar subquery)
• A read-only JOIN / scalar subquery spanning iceberg.transactions and
  iceberg.transaction_line_items (read + read — no write involved)
• A query joining two views that are each backed by an Iceberg foreign table

The read-only two-table case is the key finding: it rules out a
write/commit-path explanation. The common factor across all four incidents
was two Iceberg foreign relations in one statement, which points to a
connection- or session-level deadlock inside the wrapper when a second
Iceberg scan/cursor is opened while the first is still active.

What is reliably safe

Every statement that touches exactly one Iceberg foreign table has always
completed normally:

• SELECT count(*) FROM iceberg.transactions and full scans (~20–40s for
  232k rows; slow but finite)
• SELECT DISTINCT ON (id) * FROM iceberg.transactions ORDER BY ... into a
  local table
• INSERT INTO iceberg.transactions SELECT ... FROM <local Postgres table>
  (single Iceberg target, non-Iceberg source) — commits in seconds

Our production offload job has done single-target inserts like this every 15
minutes for months without incident.

Symptoms of a wedged backend

• Backend sits in pg_stat_activity as state = 'active', wait_event
  and wait_event_type both NULL, runtime growing without bound (we let one
  run past 11 minutes for a single-row insert that should take <1s).
• statement_timeout does not fire.
• pg_cancel_backend(pid) and pg_terminate_backend(pid) both return true
  but the backend does not die.
• The stuck session holds a lock on extensions.wrappers_fdw_stats (the
  wrapper's per-FDW stats upsert), so every subsequent session that
  initializes the Iceberg FDW blocks on it — e.g. the scheduled offload job
  starts failing each cycle. Impact stays contained (those jobs just error
  and retry next tick), but it will not self-heal.
• Only a project restart clears the stuck backend.

Reproduction sketch

-- Given two Iceberg foreign tables iceberg.a and iceberg.b:

-- WEDGES (read + read, no write):
select a.id, (select count(*) from iceberg.b where b.a_id = a.id)
from iceberg.a
limit 1;

-- SAFE (one Iceberg relation per statement — stage locally, then combine):
create unlogged table _stage_b as
  select a_id, count(*) as n from iceberg.b group by a_id;   -- one Iceberg table
create unlogged table _stage_a as
  select id from iceberg.a;                                   -- one Iceberg table
select a.id, s.n from _stage_a a left join _stage_b s on s.a_id = a.id;  -- local only

Workaround

Never reference two Iceberg foreign tables in one statement. Stage each
Iceberg table into a local (unlogged) table in its own statement, then do any
join/combine against the local copies, and write back with a single-target
insert whose sources are all local. This fully avoids the hang.

Suggested asks

1. Make Iceberg FDW scans respect statement_timeout and
   pg_cancel_backend() / pg_terminate_backend() so a bad query can be
   killed without a project restart.
2. Investigate the apparent connection/session-level deadlock when a second
   Iceberg scan is opened while a first is active within one statement.
3. If concurrent Iceberg relations in one statement genuinely can't be
   supported yet, fail fast with a clear error instead of hanging.
4. Consider not holding the wrappers_fdw_stats lock across the full scan,
   so one stuck query can't block all other Iceberg FDW sessions.

[matthewisaac869-lang]