Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 18 additions & 4 deletions config.template.jsonc
Original file line number Diff line number Diff line change
Expand Up @@ -159,17 +159,30 @@

// ── Database ────────────────────────────────────────────────────────
"database": {
// `sqlite` for single-node / dev; `mysql` for self-host with MariaDB.
// `sqlite` for single-node/dev, `mysql` for MariaDB/MySQL,
// or `postgres` for PostgreSQL.
//
// NOTE: `postgres` support is a community contribution and is not
// exercised by Puter.com production. It boots, passes its own
// integration tests against pgmock, and runs the common user/app/
// fsentry/session/permission/OIDC flows — but less-traveled code
// paths may surface MySQL/SQLite-isms that haven't been ported yet.
// Expect rough edges and please file issues if you hit one. For
// production self-hosting today, `mysql` (MariaDB) and `sqlite` are
// the supported defaults.
"engine": "sqlite",
// sqlite — file path on disk
"path": "volatile/runtime/puter-database.sqlite",
"targetVersion": 0,
// mysql — connection details
// mysql/postgres — connection details. PostgreSQL defaults to port
// 5432 when `engine` is `postgres`; MySQL/MariaDB normally use 3306.
"host": "",
"port": 3306,
"user": "",
"password": "",
"database": "",
// postgres may also use a URL instead of host/user/password fields:
// "connectionString": "postgres://puter:secret@localhost:5432/puter",
// Optional read-replica pool. Reads route here when populated.
"replica": {
"host": "",
Expand All @@ -178,9 +191,10 @@
"password": "",
"database": ""
}
// mysql self-host bootstrap: set `migrationPaths` to apply the bundled
// schema on first boot. Idempotent — safe to leave on.
// mysql/postgres self-host bootstrap: set `migrationPaths` to apply the
// bundled schema on first boot. Idempotent — safe to leave on.
// "migrationPaths": ["./src/backend/clients/database/migrations/mysql"]
// "migrationPaths": ["./src/backend/clients/database/migrations/postgres"]
},

// ── DynamoDB (KV store) ─────────────────────────────────────────────
Expand Down
22 changes: 21 additions & 1 deletion doc/self-hosting.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ Why these knobs:

- `jwt_secret` + `jwt_secret_v2` — Puter signs new auth tokens with `jwt_secret_v2` (v2 token format, `kid: 'v2'` JWT header). `jwt_secret` is verify-only and lets tokens minted before this version (cookies still in users' browsers, stored API tokens) keep working. `allow_v1_tokens: true` keeps that fallback enabled. Fresh installs can leave it as-is; future versions will flip the flag to `false` and retire `jwt_secret` entirely once legacy tokens have drained.
- `env: "prod"` — the bundled `config.default.json` ships with `env: "dev"` (matches the source-tree `npm run start=gui` workflow, which expects webpack-dev-server emitting a CSS manifest). Self-host runs against pre-built static bundles, so `env: "prod"` makes the homepage emit the `/dist/bundle.min.css` `<link>` tag instead of waiting on a manifest that doesn't exist.
- `database.migrationPaths` — Puter applies the bundled MySQL schema on boot. `mysql_mig_1.sql` (tables) and `mysql_mig_2.sql` (default apps: editor, viewer, pdf, camera, player, recorder, git, dev-center, puter-linux). Idempotent — safe to re-run.
- `database.migrationPaths` — Puter applies the bundled MySQL/MariaDB schema on boot. The migration files are idempotent, so it is safe to leave this configured across restarts.
- `dynamo.bootstrapTables: true` — Puter creates its KV table on boot. **Only set against a local emulator**, never real AWS.
- `dynamo.aws` keys are dummies; DynamoDB-local doesn't validate them but the AWS SDK requires _something_. **Note:** DynamoDB uses `access_key` / `secret_key` (snake_case); S3 below uses `accessKeyId` / `secretAccessKey` (camelCase). Not interchangeable.
- `providers.ollama.enabled: false` — Puter auto-probes a local Ollama at `127.0.0.1:11434` by default; without one running you'd see `ECONNREFUSED` on every boot. To run a bundled Ollama, see [Optional: local LLM (Ollama)](#optional-local-llm-ollama) below.
Expand Down Expand Up @@ -232,6 +232,26 @@ Change it in Settings after first login.

All optional. Drop any of the blocks below into `puter/config/config.json` and `docker compose restart puter`. See [config.template.jsonc](../config.template.jsonc) for the full list. Per-key documentation lives in [src/backend/types.ts](../src/backend/types.ts).

### PostgreSQL database

> **Community contribution — expect rough edges.** PostgreSQL support was contributed by the community and is not run by Puter.com production. It boots, applies the bundled schema, and exercises the common user/app/fsentry/session/permission/OIDC flows in the integration tests, but less-traveled SQL call sites may still need porting. If you hit a query that doesn't work on Postgres, please open an issue. For production self-hosting today, MariaDB/MySQL and SQLite are the supported defaults.

The bundled Docker Compose stack still defaults to MariaDB. To use PostgreSQL instead, run a PostgreSQL service yourself, point Puter at it, and use the PostgreSQL migration path:

```json
"database": {
"engine": "postgres",
"host": "postgres",
"port": 5432,
"user": "puter",
"password": "...",
"database": "puter",
"migrationPaths": ["/opt/puter/dist/src/backend/clients/database/migrations/postgres"]
}
```

You may use `"connectionString": "postgres://puter:...@postgres:5432/puter"` instead of the host/user/password fields. Puter only needs normal PostgreSQL connection details and the migration path; CloudNativePG is one possible Kubernetes operator, but no operator-specific manifests are required by Puter.

### Email (SMTP)

Used for password resets, email confirmation, and notifications. Without it those flows silently fail.
Expand Down
2 changes: 1 addition & 1 deletion extensions/appTelemetry.ts
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ export const handleAppTelemetryUsers = async (

const users = await clients.db.read(
`SELECT u.username, u.uuid FROM user_to_app_permissions p
INNER JOIN user u ON p.user_id = u.id
INNER JOIN ${clients.db.quoteIdentifier('user')} u ON p.user_id = u.id
WHERE p.permission = 'flag:app-is-authenticated' AND p.app_id = ?
ORDER BY (p.dt IS NOT NULL), p.dt, p.user_id
LIMIT ? OFFSET ?`,
Expand Down
74 changes: 71 additions & 3 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@
},
"scripts": {
"test:backend": "vitest run --config src/backend/vitest.config.ts ",
"test:backend:postgres": "PUTER_TEST_DB_ENGINE=postgres vitest run --config src/backend/vitest.config.ts ",
"start=gui": "nodemon --exec \"node dev-server.js\" ",
"start": "node --enable-source-maps -r ./dist/src/backend/telemetry.js ./dist/src/backend/index.js",
"prestart": "npm run build:ts",
Expand Down
93 changes: 90 additions & 3 deletions src/backend/clients/database/DatabaseClient.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ export interface BatchEntry {
values: unknown[];
}

type SqlJsonPath = readonly [string, ...string[]];

/**
* Base database client. Subclasses must override every method that throws here.
*
Expand Down Expand Up @@ -101,9 +103,10 @@ export class AbstractDatabaseClient extends PuterClient {
const cols = Object.keys(data);
const values = Object.values(data);
const sql =
`INSERT INTO \`${tableName}\` ` +
`(${cols.map((c) => `\`${c}\``).join(', ')}) ` +
`VALUES (${cols.map(() => '?').join(', ')})`;
`INSERT INTO ${this.quoteIdentifier(tableName)} ` +
`(${cols.map((c) => this.quoteIdentifier(c)).join(', ')}) ` +
`VALUES (${cols.map(() => '?').join(', ')})` +
this.returningIdClause();
return this.write(sql, values);
}

Expand Down Expand Up @@ -161,4 +164,88 @@ export class AbstractDatabaseClient extends PuterClient {
}
return choices.otherwise as T;
}

quoteIdentifier(identifier: string): string {
return identifier
.split('.')
.map((part) => {
if (part === '*') return part;
return `\`${part.replaceAll('`', '``')}\``;
})
.join('.');
}

booleanLiteral(value: boolean): string {
return value ? '1' : '0';
}

booleanValue(value: boolean): boolean | 0 | 1 {
return value ? 1 : 0;
}

insertIgnoreInto(tableName: string): string {
const table = this.quoteIdentifier(tableName);
return this.case({
sqlite: `INSERT OR IGNORE INTO ${table}`,
postgres: `INSERT INTO ${table}`,
otherwise: `INSERT IGNORE INTO ${table}`,
});
}

insertIgnoreSuffix(): string {
return this.case({
postgres: ' ON CONFLICT DO NOTHING',
otherwise: '',
});
}

upsertClause(
conflictColumns: readonly string[],
updateColumns: readonly string[],
): string {
if (updateColumns.length === 0) {
throw new Error('upsertClause requires at least one update column');
}

const updateList = updateColumns
.map((column) => `${this.quoteIdentifier(column)} = ?`)
.join(', ');

return this.case({
mysql: `ON DUPLICATE KEY UPDATE ${updateList}`,
otherwise: `ON CONFLICT(${conflictColumns
.map((column) => this.quoteIdentifier(column))
.join(', ')}) DO UPDATE SET ${updateList}`,
});
}

jsonTextExtract(jsonExpression: string, path: SqlJsonPath): string {
const sqlitePath = `$${path.map((part) => `.${part}`).join('')}`;
return this.case({
sqlite: `json_extract(${jsonExpression}, ${this.sqlStringLiteral(sqlitePath)})`,
mysql: `JSON_UNQUOTE(JSON_EXTRACT(${jsonExpression}, ${this.sqlStringLiteral(sqlitePath)}))`,
postgres: `${jsonExpression} #>> ARRAY[${path
.map((part) => this.sqlStringLiteral(part))
.join(', ')}]`,
otherwise: `JSON_UNQUOTE(JSON_EXTRACT(${jsonExpression}, ${this.sqlStringLiteral(sqlitePath)}))`,
});
}

nullCoalesce(...expressions: readonly string[]): string {
if (expressions.length === 0) {
throw new Error('nullCoalesce requires at least one expression');
}
return `COALESCE(${expressions.join(', ')})`;
}

returningIdClause(): string {
return this.case({
postgres: ' RETURNING id',
otherwise: '',
});
}

protected sqlStringLiteral(value: string): string {
return `'${value.replaceAll("'", "''")}'`;
}
}
32 changes: 2 additions & 30 deletions src/backend/clients/database/MySQLDatabaseClient.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import { createPool, type Pool } from 'mysql2';
import { AbstractDatabaseClient, type WriteResult } from './DatabaseClient';
import { SQLBatcher } from './SQLBatcher.js';
import { splitMysqlStatements } from './splitMysqlStatements.js';
import { compareMigrationFilenames } from './migrationFilenames.js';
import type { IConfig } from '../../types';

const RETRIABLE_ERROR_CODES = new Set([
Expand All @@ -44,36 +45,7 @@ const RETRIABLE_ERROR_MESSAGES = [
'ETIMEDOUT',
];

/**
* Comparator for MySQL migration filenames.
*
* Existing files are named `mysql_mig_<N>.sql` with unpadded N, so a
* plain lexical sort puts `mysql_mig_10.sql` BEFORE `mysql_mig_2.sql`.
* Pull the trailing integer out and sort numerically. Anything that
* doesn't match the `_<digits>.sql` shape (one-off, vendor dump) falls
* back to lexical comparison so the order stays deterministic, and
* unmatched names sort *after* numbered files so future numbered
* migrations don't get interleaved into a one-off's namespace.
*
* Exported only for the unit test — the production caller is the
* `runMigrations()` loop inside this file.
*/
export const compareMigrationFilenames = (a: string, b: string): number => {
const numericIndex = (name: string): number => {
const m = /_(\d+)\.sql$/.exec(name);
return m ? Number.parseInt(m[1], 10) : Number.NaN;
};
const na = numericIndex(a);
const nb = numericIndex(b);
if (Number.isFinite(na) && Number.isFinite(nb)) {
if (na !== nb) return na - nb;
} else if (Number.isFinite(na)) {
return -1;
} else if (Number.isFinite(nb)) {
return 1;
}
return a.localeCompare(b);
};
export { compareMigrationFilenames };

type PoolConfig = Parameters<typeof createPool>[0];

Expand Down
Loading
Loading