Skip to content
Floe

Plugin Architecture

This document describes the plugin system that enables floe’s flexibility.

Overview

Section titled “Overview”

floe uses a plugin architecture for all configurable components. The implementation-truth category list is floe_core.plugin_types.PluginType.

Plugin TypeAlpha-Supported Reference PathPurposeADR
ComputeDuckDBWhere dbt transforms executeADR-0010
OrchestratorDagsterJob scheduling and executionADR-0033
CatalogPolarisIceberg table catalogADR-0008
StorageS3-compatible storage plugin; demo uses MinIOObject storage for Iceberg dataADR-0036
TelemetryBackendJaeger and console telemetry pluginsOTLP telemetry backend (traces, metrics, logs)ADR-0035
LineageBackendMarquezOpenLineage backend (data lineage)ADR-0035
DBTdbt Coredbt compilation environmentADR-0043
Semantic LayerCube reference implementationBusiness intelligence APIADR-0001
Ingestiondlt plugin primitiveData loading from sourcesADR-0020
Qualitydbt expectations and Great Expectations plugin primitivesData quality validationADR-0044
RBACKubernetes RBACNamespace and service-account isolationEpic 7B
Alert ChannelWebhook / Slack / email primitivesContract violation alert deliveryEpic 15
SecretsKubernetes Secrets and Infisical plugin primitivesCredential managementADR-0023/0031
IdentityKeycloak reference implementationAuthentication providerADR-0024

Total: 14 plugin categories, based on floe_core.plugin_types.PluginType.

Note: PolicyEnforcer and DataContract are now core modules in floe-core, not plugins. Policy enforcement tooling is provided via DBTPlugin, and rules are configured via manifest.yaml. Data contracts use ODCS v3 as an enforced standard.

Canonical Registry: The authoritative implementation-truth reference for plugin category counts and entry points is Plugin Catalog.

Plugin Type History

Section titled “Plugin Type History”
VersionCountChanges
current14Added Quality, RBAC, and Alert Channel categories to PluginType; LINEAGE remains an alias for LINEAGE_BACKEND
floe-core 2.111Moved PolicyEnforcer + DataContract to core modules (not plugins)
floe-core 2.013Split ObservabilityPlugin -> TelemetryBackendPlugin + LineageBackendPlugin (ADR-0035)
floe-core 1.512Added DBTPlugin (ADR-0043)
floe-core 1.411Added DataQualityPlugin (ADR-0044)
floe-core 1.011Initial plugin architecture

Plugin Structure

Section titled “Plugin Structure”

Each plugin is a self-contained package:

plugins/floe-orchestrator-dagster/
├── src/
│   ├── __init__.py
│   └── plugin.py           # DagsterOrchestratorPlugin class
├── chart/                   # Helm chart (if service deployment needed)
│   ├── Chart.yaml
│   ├── values.yaml
│   └── templates/
│       ├── deployment.yaml
│       ├── service.yaml
│       └── configmap.yaml
├── tests/
│   ├── test_plugin.py
│   └── conftest.py
└── pyproject.toml          # Entry point registration

Documentation Index

Section titled “Documentation Index”
DocumentDescription
Discovery and RegistryPlugin discovery via entry points, PluginRegistry implementation
Plugin InterfacesAll plugin interface ABCs (ComputePlugin, OrchestratorPlugin, etc.)
Lifecycle and VersioningAPI versioning, compatibility checks, PluginMetadata
Configuration and CLICLI commands, creating custom plugins
Integration PatternsCompute-catalog integration, DuckDB example
Section titled “Related Documents”