Databricks DE · Topic 4.3

Lakeflow Connect — Managed & Standard Connectors

Built-in, mostly no-code ingestion from SaaS apps and databases — governed by Unity Catalog, serverless, built on Lakeflow Declarative Pipelines. This deep dive pairs the UI flow with config-as-code.

Managed connectorsCDC + gatewayUC connectionAsset Bundle

The one-liner

Managed connectors = fully-managed, UI-based ingestion from Salesforce, Workday, ServiceNow, SQL Server, MySQL, Postgres… Standard/community connectors reach a wider range of sources from inside your pipelines. Managed = least code, built-in monitoring; reproducible as an Asset Bundle.
✈️ Analogy: managed connectors are direct flights — Databricks runs the whole trip with monitoring & recovery built in. Standard/community connectors are connecting flights you book yourself — more reach, but you wire up more of the journey.

Connector architecture — click a type

▸ SaaS connectors are simpler; database connectors add a CDC gateway

connection → pipeline → table; reads only new/changed records (incremental, often via cursor columns).
Salesforce▸UC Connection (OAuth)▸Ingestion pipeline (SDP)▸Bronze Delta
Uses CDC + an ingestion gateway + staging storage to stream changes continuously.
SQL Server▸Gateway (CDC)▸Staging▸Pipeline▸Bronze Delta

1 · The UI flow (Salesforce example)

  1. Data Ingestion in the sidebar → choose Salesforce.
  2. Create or select a connection (OAuth) — a UC securable storing auth.
  3. Pick destination catalog/schema and the objects (e.g. Account, Opportunity).
  4. Create the ingestion pipeline (a Lakeflow SDP) and set a schedule.
💡 Database connectors add one step: an ingestion gateway + staging schema for CDC before the pipeline.

2 · The same thing as code (CI/CD)

UC connection (databases via SQL; SaaS usually via UI OAuth)

CREATE CONNECTION sqlserver_conn TYPE sqlserver
OPTIONS (host 'db.corp.net', port '1433',
         user secret('kv','sql_user'), password secret('kv','sql_pwd'));

Ingestion pipeline as a Databricks Asset Bundle

# databricks.yml
resources:
  pipelines:
    pipeline_sfdc:
      name: salesforce_ingest
      catalog: main                      # destination catalog
      schema: bronze                     # destination schema
      ingestion_definition:
        connection_name: salesforce_conn  # the UC connection
        objects:
          - table:
              source_schema: objects
              source_table: Opportunity    # Salesforce object → bronze table
              destination_catalog: main
              destination_schema: bronze

Deploy, run & schedule (CLI)

databricks bundle deploy -t prod        # create/update the pipeline
databricks bundle run pipeline_sfdc     # trigger an ingestion run
jobs:
  sfdc_daily:
    trigger: { periodic: { interval: 1, unit: DAYS } }
    tasks:
      - task_key: ingest
        pipeline_task: { pipeline_id: ${resources.pipelines.pipeline_sfdc.id} }

3 · Connector tiers — click each

Managed connectors▸
Databricks-maintained, UI-based, serverless, UC-governed. Built-in monitoring, orchestration, failure recovery. SaaS (Salesforce, Workday, ServiceNow, HubSpot, Jira), Database/CDC (SQL Server, MySQL, Postgres), File (SharePoint, Google Drive), query-based & streaming.
Standard connectors▸
Access a wider range of sources from within your pipelines/queries when no managed connector exists.
Community connectors▸
Open-source, community-built. Fill gaps for less common sources but rely on volunteer maintenance and may lack official support.
ManagedStandard / community
SetupUI + bundle, minimal codeMore wiring
Monitoring / recoveryBuilt-inYou implement
Source coverageSpecific listBroader

Why it matters

💼 Real-world use case: finance needs Salesforce opportunities daily. A managed connector ingests them incrementally into bronze — no custom API code, UC-governed, monitored automatically, and captured as a bundle for CI/CD.

Uses, edge cases & limitations

Common gotchas