LineageBackendPlugin

Purpose: Pluggable OpenLineage backends for data lineage Location: packages/floe-core/src/floe_core/plugins/lineage.py Entry Point: floe.lineage_backends ADR: ADR-0035: Observability Plugin Interface

LineageBackendPlugin separates lineage collection (OpenLineage events, enforced) from backend storage/visualization (pluggable). This enables integration with existing data catalogs and lineage tools.

Note: OpenLineage event emission is enforced across all floe pipelines. This plugin only controls where lineage data is sent.

Interface Definition

The live ABC is LineageBackendPlugin in packages/floe-core/src/floe_core/plugins/lineage.py. The snippet below is a conceptual excerpt for the public contract.

from abc import ABC, abstractmethod
from typing import Any

class LineageBackendPlugin(ABC):
    """Plugin interface for OpenLineage backends.

    Configure backends for storing and visualizing data lineage graphs.
    """

    name: str
    version: str
    floe_api_version: str

    @abstractmethod
    def get_transport_config(self) -> dict[str, Any]:
        """Generate OpenLineage transport configuration.

        Returns:
            Dictionary containing OpenLineage transport config.
            Example for HTTP backend:
            {
                "type": "http",
                "url": "https://lineage.example.com/api/v1/lineage",
                "auth": {"type": "api_key", "api_key": "${LINEAGE_API_KEY}"}
            }
        """
        pass

    @abstractmethod
    def get_namespace_mapping(self) -> dict[str, str]:
        """Generate namespace mapping for lineage events.

        Returns:
            Dictionary mapping floe namespaces to lineage backend namespaces.
            Example:
            {
                "default": "floe_default",
                "analytics": "floe_analytics"
            }
        """
        pass

Reference Implementations

Plugin	Description	Self-Hosted
MarquezLineagePlugin	Local/self-hosted lineage reference path	Yes
AtlanLineagePlugin	SaaS data catalog and governance	No
OpenMetadataLineagePlugin	Open-source data catalog	Yes

Related Documents

• ADR-0035: Observability Plugin Interface
• Plugin Architecture
• TelemetryBackendPlugin - Companion observability plugin