update docs

Author: ilyario

README.md

@@ -0,0 +1,58 @@
+# DataFlow Operator Documentation
+
+Documentation site for [DataFlow Operator](https://github.com/dataflow-operator/dataflow), built with [MkDocs Material](https://squidfunk.github.io/mkdocs-material/).
+
+**Live site**: https://dataflow-operator.github.io/docs/
+
+## Languages
+
+- English (`docs/en/`)
+- Russian (`docs/ru/`)
+
+## Local development
+
+```bash
+pip install -r requirements.txt
+mkdocs serve
+```
+
+Open http://localhost:8000.
+
+## Build
+
+```bash
+mkdocs build
+```
+
+Output goes to `site/`.
+
+## Structure
+
+```
+docs/
+├── mkdocs.yml              # MkDocs configuration
+├── requirements.txt        # Python dependencies
+├── docs/
+│   ├── en/                 # English documentation
+│   ├── ru/                 # Russian documentation
+│   ├── images/             # Logo, favicon
+│   ├── stylesheets/        # Custom CSS
+│   └── javascripts/        # Custom JS
+└── .github/workflows/
+    └── docs.yml            # CI: deploy to GitHub Pages
+```
+
+## Topics covered
+
+- Getting Started
+- Architecture
+- Connectors (Kafka, PostgreSQL, ClickHouse, Trino, Nessie)
+- Transformations
+- Configuration examples
+- Error handling and fault tolerance
+- Prometheus metrics
+- Web GUI
+- MCP server
+- Development guide
+- Connector development
+- Semantic versioning and releases

docs/en/connector-development.md

@@ -41,9 +41,9 @@ Used when the connector performs long-running read operations (e.g., SQL queries
 
 ## Step-by-Step Guide: Adding a New Connector
 
-### Step 1. Define Types in API
+### Step 1. Define Types
 
-Add the spec to `api/v1/dataflow_types.go`:
+Define your connector's config struct (it will be parsed from `Config json.RawMessage` at runtime):
 
 ```go
 // MyDBSourceSpec defines MyDB source configuration
@@ -52,14 +52,20 @@ type MyDBSourceSpec struct {
     Table            string `json:"table"`
     // ...
 }
+```
 
-// In SourceSpec:
-type SourceSpec struct {
-    // ...
-    MyDB *MyDBSourceSpec `json:"myDB,omitempty"`
-}
+Users will specify the connector in a DataFlow manifest using the unified `type` + `config` format:
+
+```yaml
+source:
+  type: mydb
+  config:
+    connectionString: "mydb://localhost:3306/db"
+    table: my_table
 ```
 
+The `SourceSpec` and `SinkSpec` structs use `Type string` + `Config json.RawMessage`. The factory parses `Config` into your spec struct.
+
 ### Step 2. Choose baseConnector or baseConnectorRWMutex
 
 **Use `baseConnector`** when:
@@ -233,17 +239,18 @@ Note: rawMode is supported only in sink connectors. Sources always emit plain co
 
 ### Step 6. Register in Factory
 
-In `internal/connectors/factory.go`:
+In `internal/connectors/factory.go`, parse `Config` into your spec struct:
 
 ```go
 func CreateSourceConnector(source *v1.SourceSpec) (SourceConnector, error) {
     switch source.Type {
     // ...
     case "mydb":
-        if source.MyDB == nil {
-            return nil, fmt.Errorf("mydb source configuration is required")
+        var cfg MyDBSourceSpec
+        if err := json.Unmarshal(source.Config.Raw, &cfg); err != nil {
+            return nil, fmt.Errorf("invalid mydb source config: %w", err)
         }
-        return NewMyDBSourceConnector(source.MyDB), nil
+        return NewMyDBSourceConnector(&cfg), nil
     default:
         return nil, fmt.Errorf("unknown source type: %s", source.Type)
     }
@@ -253,10 +260,11 @@ func CreateSinkConnector(sink *v1.SinkSpec) (SinkConnector, error) {
     switch sink.Type {
     // ...
     case "mydb":
-        if sink.MyDB == nil {
-            return nil, fmt.Errorf("mydb sink configuration is required")
+        var cfg MyDBSinkSpec
+        if err := json.Unmarshal(sink.Config.Raw, &cfg); err != nil {
+            return nil, fmt.Errorf("invalid mydb sink config: %w", err)
         }
-        return NewMyDBSinkConnector(sink.MyDB), nil
+        return NewMyDBSinkConnector(&cfg), nil
     default:
         return nil, fmt.Errorf("unknown sink type: %s", sink.Type)
     }
@@ -266,9 +274,9 @@ func CreateSinkConnector(sink *v1.SinkSpec) (SinkConnector, error) {
 ### Step 7. Generate and Test
 
 ```bash
-make generate
-make manifests
-make test-unit
+task generate
+task manifests
+task test-unit
 ```
 
 ## Important Notes

docs/en/connectors.md

@@ -106,7 +106,7 @@ metadata:
 spec:
   source:
     type: kafka
-    kafka:
+    config:
       brokersSecretRef:
         name: kafka-credentials
         key: brokers
@@ -146,7 +146,7 @@ metadata:
 spec:
   source:
     type: postgresql
-    postgresql:
+    config:
       connectionStringSecretRef:
         name: postgres-credentials
         key: connectionString
@@ -213,7 +213,7 @@ metadata:
 spec:
   source:
     type: kafka
-    kafka:
+    config:
       brokers:
         - secure-kafka:9093
       topic: secure-topic
@@ -299,7 +299,7 @@ The Kafka connector supports reading and writing messages from/to Apache Kafka t
 ```yaml
 source:
   type: kafka
-  kafka:
+  config:
     brokers:
       - kafka1:9092
     topic: input-topic
@@ -354,7 +354,7 @@ source:
 ```yaml
 sink:
   type: kafka
-  kafka:
+  config:
     brokers:
       - kafka1:9092
     topic: output-topic
@@ -370,7 +370,7 @@ The PostgreSQL connector supports reading from and writing to PostgreSQL tables.
 ```yaml
 source:
   type: postgresql
-  postgresql:
+  config:
     # Connection string (required, or use connectionStringSecretRef)
     connectionString: "postgres://user:password@localhost:5432/dbname?sslmode=disable"
     # Table to read from (required if query not specified). Supports schema.table (e.g. public.products)
@@ -411,7 +411,7 @@ source:
 ```yaml
 sink:
   type: postgresql
-  postgresql:
+  config:
     connectionString: "postgres://user:password@localhost:5432/dbname?sslmode=disable"
     # Table to write to. Supports schema.table (e.g. public.products_clone)
     table: target_table
@@ -470,15 +470,15 @@ metadata:
 spec:
   source:
     type: postgresFull
-    postgresFull:
+    config:
       connectionString: "postgres://user:pass@source-pg:5432/db?sslmode=disable"
       syncMode: full
       dataMode: schema_and_data   # or schema_only
       # databases: ["public.users", "analytics.mv_report"]  # optional filter
       # excludeObjects: ["view", "function"]               # optional exclude
   sink:
     type: postgresFull
-    postgresFull:
+    config:
       connectionString: "postgres://user:pass@target-pg:5432/db?sslmode=disable"
 ```
 
@@ -517,7 +517,7 @@ The ClickHouse connector supports reading from and writing to ClickHouse tables.
 ```yaml
 source:
   type: clickhouse
-  clickhouse:
+  config:
     # Connection string (required)
     # Native: clickhouse://host:9000?username=default&password=xxx&database=default
     # HTTP: http://host:8123/default?username=default&password=xxx
@@ -546,7 +546,7 @@ source:
 ```yaml
 sink:
   type: clickhouse
-  clickhouse:
+  config:
     connectionString: "clickhouse://default@clickhouse:9000/default?dial_timeout=10s"
     table: output_table
 
@@ -589,14 +589,14 @@ metadata:
 spec:
   source:
     type: kafka
-    kafka:
+    config:
       brokers:
         - kafka:9092
       topic: input-topic
       consumerGroup: dataflow-group
   sink:
     type: clickhouse
-    clickhouse:
+    config:
       connectionString: "clickhouse://default@clickhouse:9000/default"
       table: output_table
       batchSize: 100
@@ -615,13 +615,13 @@ metadata:
 spec:
   source:
     type: clickhouse
-    clickhouse:
+    config:
       connectionString: "clickhouse://dataflow:dataflow@localhost:9000/dataflow?dial_timeout=10s"
       table: products
       pollInterval: 5
   sink:
     type: clickhouse
-    clickhouse:
+    config:
       connectionString: "clickhouse://dataflow:dataflow@localhost:9000/dataflow?dial_timeout=10s"
       table: products_clone
       batchSize: 100
@@ -637,7 +637,7 @@ The Trino connector supports reading from and writing to Trino (formerly PrestoS
 ```yaml
 source:
   type: trino
-  trino:
+  config:
     # Trino server URL (required)
     serverURL: "http://trino:8080"
 
@@ -719,7 +719,7 @@ The response will contain an `access_token` field. Use this token value in the `
 ```yaml
 sink:
   type: trino
-  trino:
+  config:
     # Trino server URL (required)
     serverURL: "http://trino:8080"
 
@@ -774,14 +774,14 @@ metadata:
 spec:
   source:
     type: kafka
-    kafka:
+    config:
       brokers:
         - kafka:9092
       topic: input-topic
       consumerGroup: dataflow-group
   sink:
     type: trino
-    trino:
+    config:
       serverURL: "http://trino:8080"
       catalog: hive
       schema: default
@@ -809,7 +809,7 @@ The Nessie connector reads from and writes to Apache Iceberg tables via the [Nes
 ```yaml
 source:
   type: nessie
-  nessie:
+  config:
     # Nessie server base URL (required), e.g. https://nessie:19120
     baseURL: "http://nessie:19120"
 
@@ -847,7 +847,7 @@ source:
 ```yaml
 sink:
   type: nessie
-  nessie:
+  config:
     baseURL: "http://nessie:19120"
     branch: main
     warehouse: ""
@@ -883,14 +883,14 @@ metadata:
 spec:
   source:
     type: kafka
-    kafka:
+    config:
       brokers:
         - kafka:9092
       topic: input-topic
       consumerGroup: dataflow-group
   sink:
     type: nessie
-    nessie:
+    config:
       baseURL: "http://nessie:19120"
       branch: main
       namespace: analytics