kyma

Appearance

Connectors ​

Connectors are how kyma pulls from sources that don't push to it. The trait is small, the registry is one map, and the runner is the same group-commit loop the OTLP and Kafka frontends use — so a connector's output lands on the same staging buffer, through the same snapshot CAS, into the same kyma extents.

One framework. One reference engine in tree. Three operational-database engines on the way, each shipping in its own milestone.

Framework ​

The Connector trait, the registry, the periodic scheduler, the admin REST API at POST /v1/connectors, and the secret-by-reference resolver. Generic to every connector type — read this once and the five pages below are mostly config.

Prometheus ​

The reference implementation. Scrapes /metrics on a schedule, parses OpenMetrics, lands one row per sample in the configured table. Shipped today; the smallest possible thing that proves the framework.

Postgres 🚧 ​

Federation for live reads, replication-slot CDC for sync, both at once. Pushdown of filters, projection, LIMIT, single-source aggregation. Lands in DB-M1.

MySQL 🚧 ​

Same shape as Postgres. Binlog row events with GTID checkpoints for CDC; collation-safety rules in the planner so case-insensitive columns never silently corrupt federated equality. Lands in DB-M2.

MongoDB 🚧 ​

Change streams with startAtOperationTime for CDC. BSON type coercion, nested-document flattening up to a configurable depth, polymorphic-field demotion to dynamic. Lands in DB-M3.

Multi-source data 🚧 ​

The marquee surface — live(...) for federated reads, the pushdown_summary on every federated response, kyma_connector_health as a queryable kyma table. Cross-references the conceptual model at Multi-source data.

What's the same across all connector types ​

  • One trait. Implement Connector::run_once; the framework owns scheduling, retry classification, secret resolution, and the row sink.
  • One registry. Connector types register at startup; the admin API validates type against it before persisting any row.
  • One write path. Rows produced by a tick go through the same JSON-to-Arrow coercion, the same staging buffer, and the same snapshot CAS as REST/NDJSON ingest. See Extents and snapshots.
  • One health surface. Per-tick metrics, last_error, last_rows_ingested on the connector row, plus the kyma_connector_health view (DB-M0+) for the database engines. See Observability.