The wire to DuckDB: Arrow Flight SQL and the client families

What "connect a BI tool to DuckDB" actually requires, the client families one open protocol unlocks, and how Quack on Demand puts a DuckLake behind that wire

At a glance

Seven questions this paper answers, what the native stack does about each, and what Quack on Demand adds.

If you are asking	DuckDB + Quack	+ Quack on Demand	Where
Can a BI tool open a live connection?	No. DuckDB is a library, and Quack's client is another DuckDB. BI vendors do not ship drivers for either.	Yes. One Arrow Flight SQL endpoint, reached through the JDBC, ODBC and ADBC drivers that already exist for the protocol.	Section 1
Is the wire a protocol client vendors already support?	No. Quack is DuckDB to DuckDB, by design, and it is excellent at exactly that.	Yes. Arrow Flight SQL is an open Apache protocol with drivers maintained upstream, not by your database vendor. No QoD-specific driver exists or is needed.	Section 3
Do results stay columnar on the wire?	Yes, in process. Embedded DuckDB and in-process ADBC hand you Arrow without any wire at all.	Yes, over the network. Results stream as Arrow record batches: the layout the engine computes in is the layout the client receives.	Section 3
Can the wire tell users apart?	No. Quack authentication is a shared token, one identity for every client.	Yes. The handshake validates a username and password or an external JWT / OIDC bearer, then issues a session token with a bounded lifetime.	Section 4
Can a generic tool browse the catalog?	No. There are no standard metadata calls for a tool to draw a table tree from.	Yes. The standard Flight SQL catalog calls are answered from the DuckLake catalog, filtered to the tables the caller may see.	Section 4
Do base tables stay server-side when analysts query?	No. An embedded engine runs where the client runs, so the data comes to the analyst's machine to be scanned.	Yes, with a live connection. The query runs on a server-side node and only result rows cross, as Arrow batches. Import and extract modes still copy, by design.	Section 4
Can a session run multi-statement transactions?	Yes. A DuckDB session runs BEGIN and COMMIT like any database.	Yes, in SQL. BEGIN pins the session to one node until COMMIT or ROLLBACK. Only the protocol-level transaction actions are absent, deliberately, so autocommit-off drivers use SQL instead of failing.	Section 4

No platform fee appears anywhere in this table. The protocol is Apache Arrow's, the drivers are open source, the gateway is Apache-2.0, and all of it runs on your infrastructure.

---

1. Why "how do I connect to DuckDB" has three answers, and then a fourth question

DuckDB is a library. That is not a limitation, it is the design: the engine runs inside your process, next to your data structures, with no server to stand up and no wire to cross. "Connecting to DuckDB" has therefore always meant one of three things, and all three are good answers to the questions they answer.

Embed it. The client bindings (Python, R, Java, Node, Go and more) load the engine into your program. This is the fastest possible path: query results are handed over in memory, often as Arrow, with no serialization at all.

Speak ADBC to it, still in process. ADBC is Arrow's database connectivity API: one columnar API, many drivers. DuckDB ships an ADBC driver, so a program written against ADBC can run on DuckDB today and on another Arrow-speaking database tomorrow without changing its data path. The engine still runs in your process; what ADBC standardises here is the API, not a wire.

Attach it over Quack. Quack is the client/server protocol from the DuckDB core team, a stable protocol shipped as an autoloadable core extension. One DuckDB instance attaches a database (a DuckLake, for the rest of this paper), calls quack_serve, and other DuckDB instances ATTACH 'quack:host' and run SQL against it over HTTP. Now there is a wire, and a very good one: one warm server process, many DuckDB clients.

The fourth question arrives with the first client that is not a DuckDB process, and in most organisations it arrives quickly. Power BI wants a live connection. A Java service needs a pooled JDBC source. A Rust job wants Arrow batches without embedding an engine and the credentials that come with it. None of these can embed DuckDB into someone else's product, and none of them speak Quack, because Quack's client is by design another DuckDB.

What they need is a wire protocol their vendors already ship drivers for. That protocol exists, it is open, and it is columnar: Arrow Flight SQL. A second open wire has universal drivers too, the Postgres protocol, and section 3 weighs the gateways that emulate it before settling on Flight SQL. Section 3 is about the protocol and the client families it unlocks; section 4 is about how Quack on Demand puts a DuckLake behind it. One scope statement first, the same one this series always makes: nothing about a wire changes the engine. One query still runs on one DuckDB node, and if a single join over tens of terabytes must be split across machines, that is Trino, Dremio, StarRocks or Spark, not anything in this paper. The companion paper on scaling DuckDB on DuckLake covers what a pool of nodes does and does not give you.

2. What DuckDB already gives you, and where each path is the right one

2.1 In process: the fastest wire is no wire

When the code and the data share a machine, embedding is not a compromise, it is the optimum. The engine reads the lakehouse directly, the result set is handed to your program in memory, and if your program is Arrow-native the handover is columnar from end to end. In-process ADBC adds portability on top: code written against the ADBC API keeps its data path if the engine behind it changes. Nothing in this paper improves on that path for single-machine work, and Quack on Demand does not try to replace it.

The costs appear when the machine stops being yours alone. Every embedding client needs credentials to the catalog and the object store. Every client scans object storage for itself, so the same table is fetched and cached once per analyst. And the engine's identity is the process's identity: there is no per-user anything, because there is no server to enforce it.

2.2 Quack: the right wire between DuckDB processes

Quack turns one DuckDB into a server. That collapses the per-client costs above into one warm process: one page cache, one catalog connection, one set of credentials, shared by every session. Clients attach with one line and run plain SQL. Between DuckDB processes, this is the wire to use, and it is the wire Quack on Demand itself uses internally: QoD nodes are DuckDB processes running quack_serve, and the manager speaks Quack to them.

Quack is also deliberately narrow, and the DuckDB docs are candid about the posture: a Quack server speaks plain HTTP, binds to localhost by default, and anything beyond a local deployment should sit behind a proxy that terminates TLS. The engine is meant to be fronted.

Use it when	Every client is a DuckDB process you control, on your network: pipelines, services, other engines attaching the same lakehouse.
Not here	Flight SQL, JDBC or ODBC (the client is another DuckDB; the DuckDB docs point at separate third-party projects for other remote-access paths); per-user authentication (a shared token is one identity for everyone); standard catalog-browsing calls for generic tools.

3. Arrow Flight SQL and the client families

Arrow Flight SQL is an open protocol for running SQL against a server and getting Arrow back. It is built on Arrow Flight, a gRPC-based framework for moving Arrow record batches over the network, and it standardises the SQL-shaped part: how a client submits a query, how it prepares and binds parameters, and how it asks the catalog what exists. It is an Apache Arrow project, vendor-neutral by construction, and the drivers for it are maintained upstream in the Arrow ecosystem rather than by any one database.

Two properties make it the right wire for an analytical engine.

Results are columnar on the wire. An analytical result set is columnar at both ends: the engine computes over vectors, and the client (a BI tool, a dataframe library, another Arrow program) wants columns. A row-oriented protocol forces two transpositions, rows built on the server and columns rebuilt on the client, plus a type conversion at each end. Flight SQL streams the result as Arrow record batches instead: the layout the engine computes in is the layout that crosses the network and the layout the client keeps. Large results stream batch by batch rather than buffering.

The command surface is standard. Beyond execute-a-query, the protocol defines prepared statements (prepare once, bind parameters, execute) and a catalog metadata family: list catalogs, schemas, tables and table types, describe primary and foreign keys, report type information. That metadata family is what lets a generic tool draw a table tree, autocomplete a column, or generate a join, with no vendor-specific code. It is the difference between "a driver exists" and "the tool actually works".

One protocol then unlocks four client families at once, and this is the part that matters for planning:

• JDBC. The Arrow Flight SQL JDBC driver is one jar. Anything that consumes JDBC gets a connection: DBeaver and the SQL IDEs, Java services and connection pools, JVM schedulers, Spark reading over JDBC.
• ODBC. An Arrow Flight SQL ODBC driver serves the DSN world: Excel, Tableau, and the long tail of tools whose only extensibility is a DSN.
• ADBC. The same ADBC API from section 2.1, now with the Flight SQL driver behind it, so the engine moves off-machine without the code changing. Drivers exist across Python, Go, Rust, C/C++ and more, and results arrive as Arrow with no row-shaped detour. It is also where the BI vendors themselves are heading: Power BI now ships ADBC drivers, the Flight SQL ADBC driver among them, and Microsoft is retiring its embedded ODBC drivers in ADBC's favour.
• Flight-native. Arrow-native programs can skip the database-API shape entirely and speak Flight SQL directly, for example from PyArrow, or from generated gRPC clients in any language gRPC supports.

The asymmetry is the point: a server implements the protocol once and inherits every family, present and future, because the drivers belong to the ecosystem rather than the server. These are generic protocol drivers, not connectors written for one product: the same jar, DSN or ADBC driver that reaches one Flight SQL server reaches any other. That is what "a wire BI vendors already ship drivers for" means in practice.

Where ADBC ends and Flight SQL begins

The two names travel together and are regularly mistaken for competitors, so it is worth being exact: they do different jobs, and they meet in a driver. ADBC is an API, the Arrow-native successor to JDBC and ODBC: it defines the calls your code makes and says nothing about what crosses the network. Flight SQL is a wire protocol: it defines what crosses the network and says nothing about the API your code sees. The ADBC Flight SQL driver is where they connect, implementing the ADBC API on the front and speaking Flight SQL out the back.

	ADBC	Flight SQL
What it is	A client API: connect, execute, fetch Arrow	A wire protocol: what crosses the network
The peer it replaces	JDBC and ODBC, as the API your code targets	Vendor wire protocols, as the bytes on the network
Your code sees it	Yes, it is what you program against	No, the driver speaks it for you
What swaps behind it	The driver: in-process DuckDB today, Flight SQL tomorrow, same code	The client family: ADBC, JDBC, ODBC and Flight-native all reach the same server

This split is exactly what the Power BI transition illustrates: Microsoft changed the API layer its connectors are built on, ODBC to ADBC, while the Flight SQL wire underneath the transitioning connector stayed the same. An architecture standardised on the wire absorbs API-layer churn like that without moving anything server-side.

The other wire with universal drivers: emulating Postgres

One other open wire has drivers in every client: the Postgres protocol. A family of gateways puts DuckDB behind it, accepting Postgres connections, transpiling the SQL, stubbing pg_catalog for introspection, and executing on DuckDB underneath; Duckgres, an MIT-licensed server from PostHog, is the most complete current example. The appeal is real: psql, every Postgres driver, every ORM, nothing new installed.

Two structural costs are why this series does not build on it.

Structural cost	On the Postgres wire	On Flight SQL
Row transposition	The wire is row-oriented: every result pays the double transposition this section opened with, columns become row messages on the server, then columns again in the client.	Results stream as Arrow record batches; the engine's layout crosses unchanged.
Emulation surface	The client believes it is talking to Postgres, so dialect transpilation, catalog stubs and type flattening (DuckDB's MAP and STRUCT fall back to text) must be chased client by client.	The surface is a published protocol with upstream drivers and native Arrow types; nothing is emulated.

Tellingly, Duckgres itself reaches its DuckDB workers over Arrow Flight SQL, keeping Postgres as the client-facing facade. Emulation earns its keep at exactly one hop, in front of a client that can speak nothing else; the family table below carries that row.

Picking a family

Your client	Family	What you install
Power BI	ADBC (ODBC as the legacy path)	Power BI ships the Flight SQL ADBC driver in its transitioning connectors; Microsoft is phasing its embedded ODBC drivers out through 2026 and 2027. Gateway-routed refreshes still run on ODBC today.
Excel, Tableau, DSN-based tools	ODBC	The Arrow Flight SQL ODBC driver, configured as a DSN.
DBeaver, SQL IDEs, Java services, Spark	JDBC	The Arrow Flight SQL JDBC driver, a single jar on the classpath.
Python, Go, Rust, C++ programs and notebooks	ADBC	The language's ADBC manager plus the Flight SQL driver (in Python, adbc_driver_flightsql).
Arrow-native or gRPC-native code	Flight-native	Nothing beyond your Arrow or gRPC toolchain.
DuckDB processes you control	Quack	Nothing: ATTACH 'quack:host'. The right wire between engines stays the right wire.
A tool that can speak only the Postgres wire and can install nothing	Postgres emulation (not this paper's wire)	A Postgres-emulation gateway in front of DuckDB, with the transposition and compatibility caveats above. Move the client to a standard Flight SQL driver when it can take one.

4. Quack on Demand: a DuckLake behind one Flight SQL edge

Quack on Demand is an Apache-2.0 gateway that serves a DuckLake lakehouse over exactly this wire. Clients see one Arrow Flight SQL endpoint. Behind it, statements run on pools of DuckDB nodes attached to the same DuckLake; the routing, pooling and demand-sizing story is the companion paper, and this section stays on the wire itself: what a client experiences from connection to result.

One endpoint, TLS on. The edge is a gRPC endpoint (port 31338 by default) with TLS enabled by default: a self-signed certificate is generated on first boot for development, and production supplies its own chain. Everything a client does, from handshake to result stream, happens on this one endpoint.

No custom driver, on purpose. Quack on Demand ships no driver of its own, and that is a feature, not a gap. Clients connect with the standard artifacts the Arrow ecosystem already maintains: the Apache Arrow Flight SQL JDBC driver, the ADBC Flight SQL driver from the Apache Arrow ADBC project (adbc_driver_flightsql in Python), PyArrow's Flight client, and the Flight SQL ODBC drivers already in circulation, such as the one Dremio maintains. Three things follow. There is nothing QoD-specific to install, so a security team vets a widely deployed Apache artifact rather than a vendor binary. Driver fixes and new language support arrive from the ecosystem, on the ecosystem's cadence, with no gateway release in the loop. And there is no client-side lock-in in either direction: the driver a tool already uses for another Flight SQL backend connects to Quack on Demand unchanged, and leaving takes the connection string with it, not the client estate.

A handshake that knows who you are. The first request carries either a username and password (Basic) or an external bearer token. The edge validates the credential against its configured providers: database-backed users by default, JWT or OIDC providers (Keycloak, Google, Azure, AWS) when enabled. On success it returns a session bearer that the driver attaches to every subsequent call, standard Flight SQL client behaviour that JDBC, ODBC and ADBC drivers all implement without custom code. Sessions have a bounded lifetime: past the TTL, the next call forces a fresh handshake and re-validates the original credential, which bounds how long a revoked credential keeps working.

A scope that says where you are. Every session addresses a specific tenant and pool, passed as two plain gRPC headers. The JDBC driver forwards them from URL query parameters, so the whole connection story for a BI tool is one string:

jdbc:arrow-flight-sql://host:31338?useEncryption=true&user=alice&password=...&tenant=acme&pool=bi

ADBC passes the same pair as connection options, and the ODBC DSN carries them as properties. The tenant decides which DuckLake catalog the session sees; the pool decides which nodes serve it.

A catalog the tool can draw. The edge answers the full Flight SQL metadata family (catalogs, schemas, tables, table types, primary and foreign keys, type info) from the DuckLake catalog, and it answers it per principal: the listing is filtered to the objects the caller holds at least read access on. When DBeaver draws its table tree or a BI tool enumerates fields, that is these calls underneath, and what renders is what the user may actually query. Grants, column masking and row policies are their own paper; what matters on the wire is that enforcement happens at the edge, before SQL ever reaches a node, and that the standard metadata calls respect it.

Statements, prepared and plain, reads and writes. Ad-hoc queries and prepared statements both work, including parameter binding, and DML runs through the same wire with affected-row counts returned the way JDBC and ADBC expect. The default is autocommit: each statement is routed to a node and commits on its own through the DuckLake catalog.

Transactions, in SQL. A session that issues BEGIN (or START TRANSACTION) gets a real multi-statement transaction: the edge pins the session to one write-capable node until COMMIT or ROLLBACK, so every statement in the transaction sees and mutates the same engine's state. The failure mode is honest too: if the pinned node disappears mid-transaction, the transaction is lost and the client gets a retryable error rather than a silent partial commit. One wire-level nuance is deliberate. Flight SQL also defines protocol-level transaction actions, and the edge does not implement them; it advertises no transaction support at the protocol level precisely so that autocommit-off JDBC and ADBC clients fall back to plain SQL transactions instead of calling actions that would fail. The SQL path is the supported one, and it is the one every client family can speak.

What crosses the wire, and what never does

With a live connection (DirectQuery in Power BI, a live data source in Tableau, any ad-hoc SQL client), each interaction issues SQL over the wire, the query runs on a server-side node against DuckLake data in your object storage, and only the result rows stream back, as Arrow batches. The base tables never land on the analyst's machine. For a lakehouse whose point is that data stays in your bucket and your VPC, this is the property that makes BI access compatible with that point.

Be precise about the exception, because it is client behaviour, not wire behaviour: Power BI's Import mode and Tableau's extract mode copy the full dataset to a local file by design. That copy happens with any backend. When server-side residency is the goal, use the live connection mode; the wire then guarantees that what crosses is what the query returned, after the edge's grants, masks and row policies have been applied.

Runnable examples for every family live in the repository's examples/ directory (TypeScript, Python, Java and Rust clients, each running a single query and the TPC-H suite), alongside the connection walkthroughs in the project documentation.

Use it when

Anything that is not a DuckDB process needs a live, authenticated connection to your DuckLake: BI tools, services, notebooks, jobs in other languages.

Not here

The protocol-level transaction actions (transactions work, but as SQL BEGIN / COMMIT; the Flight SQL BeginTransaction / EndTransaction actions are unimplemented and advertised as absent so drivers do not call them); Substrait plans; bulk Arrow ingestion (writes are SQL statements, not uploaded batches); a zero-hop data path (every statement and result crosses the edge); the Quack wire itself (internal only, never exposed to clients).

5. How to choose

Ask the questions in order; the first row that matches decides it.

If	Then
Every client is a DuckDB process you control	Quack alone, fronted with a TLS-terminating proxy exactly as the DuckDB docs recommend. Come back when the first non-DuckDB client asks in; the section 3 family table says what it will need.
A client can speak only the Postgres wire and can install nothing	A Postgres-emulation gateway bridges that one client, with the row-transposition and emulation-surface caveats from section 3. Route everything that can take a standard Flight SQL driver through the columnar wire instead.
Other clients need a live connection	Quack on Demand: a Flight SQL edge already built, Apache-2.0, with the pool behind it as the companion paper describes. The protocol is open, so the client families and the columnar wire stay standard: nothing about the choice locks the clients in.

6. Side by side

One row per dimension. Read it as a recap: the rows only matter once a specific client family or constraint makes them matter.

	DuckDB + Quack	Quack on Demand
Client-facing wire	Quack (HTTP, DuckDB to DuckDB)	Arrow Flight SQL (gRPC, Arrow record batches)
Who ships the drivers	DuckDB itself (the client is the engine)	The Apache Arrow ecosystem: JDBC, ODBC, ADBC, Flight-native. No QoD-specific driver exists or is needed
BI tools connect	No	Yes: live connections via ODBC / JDBC
Result format on the wire	Quack's own bulk path	Arrow record batches, streamed
Authentication	Shared token, one identity	Per-user: database, JWT or OIDC handshake, bounded sessions
Catalog browsing calls	No	Yes, standard Flight SQL metadata, filtered to grants
Prepared statements	SQL through the attached session	Yes, protocol-level, with parameter binding
Writes over the wire	Yes, SQL	Yes, SQL, with affected-row counts as drivers expect
Multi-statement transactions	Yes, in the session	Yes, as SQL BEGIN / COMMIT, session pinned to one node; the protocol-level transaction actions are deliberately absent
TLS	Via the fronting proxy	On the endpoint, enabled by default
Base-table residency for BI	n/a (no BI path)	Live connections move result rows only
Licence	MIT	Apache-2.0

7. What we expect to change

• Quack beyond DuckDB clients. Whether BI vendors will ever speak Quack directly is an open question. If a tool ships native Quack support one day, that client moves to the "DuckDB processes" row of the family table and connects to the same lakehouse; nothing about the architecture moves.
• ADBC coverage keeps widening. ADBC is the youngest of the families and the one gaining drivers and language support fastest. As dataframe libraries and orchestrators adopt it, more clients reach the same endpoint with less installation, not through anything the server changes.
• The Power BI path is switching drivers, not wires. Power BI historically reached Flight SQL through a connector wrapping the ODBC driver; Microsoft is now transitioning its connectors to ADBC, with the Flight SQL ADBC driver among the replacements, and plans to stop shipping its embedded ODBC drivers by 2027 (gateway-routed refreshes stay on ODBC longest). Both paths speak the same Flight SQL wire, which is exactly the point: the packaging above the wire changes, the wire does not.

The wire itself is the settled layer: an open protocol with ecosystem-owned drivers ages at the speed of Apache Arrow, not at the speed of any one engine's release notes.

---

Appendix A. Wire configuration, as of Quack on Demand 0.7.2

These are the parts that move between releases, which is why they are here and not in the body. Every name and default below was read from the release's own configuration. Check them against your version before relying on them.

Setting	Default	What it does
Endpoint
PROXY_HOST / PROXY_PORT	0.0.0.0 / 31338	Bind address and port of the Flight SQL edge.
PROXY_TLS_ENABLED	true	TLS on the edge. A self-signed development certificate is generated on first boot when none is supplied.
PROXY_TLS_CERT_CHAIN / PROXY_TLS_PRIVATE_KEY	certs/server-cert.pem / certs/server-key.pem	Certificate chain and key; point them at your own material in production.
Session
QOD_SESSION_TTL_SEC	3600	Edge session lifetime. Past it, the next call forces a fresh handshake and re-validates the original credential, bounding how long a revoked credential keeps working.
Scoping headers
tenant / pool	required	Plain gRPC headers on every session; the JDBC driver forwards them from URL query parameters, ADBC from connection options. There are no server-side defaults: every client addresses an explicit tenant and pool.
superuser=true	off	Selects the system realm for the credential check (operator accounts); tenant and pool still drive routing.
x-qod-authorization	fallback	Accepted in place of Authorization for driver builds that strip the standard header but pass custom ones through.
Behind the edge
PROXY_RESUME_HOLD_TIMEOUT_SEC	60	How long the edge holds a statement while a suspended pool wakes before answering a retryable "pool is resuming".

---

Sources

Repository claims were checked against starlake-ai/quack-on-demand at release 0.7.2. External pages are listed for the revision pass to fetch, date and confirm.

• Apache Arrow, "Arrow Flight SQL" format specification: arrow.apache.org/docs/format/FlightSql.html
• Apache Arrow, "Arrow Flight RPC": arrow.apache.org/docs/format/Flight.html
• Apache Arrow, "ADBC: Arrow Database Connectivity": arrow.apache.org/adbc/
• Apache Arrow, "Arrow Flight SQL JDBC driver": arrow.apache.org/docs/java/flight_sql_jdbc_driver.html
• Dremio, Arrow Flight SQL ODBC driver documentation (URL to be pinned at the revision pass)
• Microsoft Learn, "Transition from ODBC to ADBC drivers in Power BI and Microsoft Fabric" (fetched 5 September 2026; the Flight SQL ADBC driver listed as a replacement driver, embedded ODBC removal timeline): learn.microsoft.com/en-us/power-query/transition-to-adbc
• PostHog, Duckgres, a PostgreSQL wire protocol server backed by DuckDB, MIT (fetched 5 September 2026; SQL transpilation, pg_catalog stubs, type fallback to text, Arrow Flight SQL between control plane and workers): github.com/PostHog/duckgres
• DuckDB, "Frequently Asked Questions for Quack": duckdb.org/quack/faq (the client is another DuckDB; no distributed query processing)
• DuckDB, "Securing Quack with a Reverse Proxy": duckdb.org/docs/current/quack/setup/reverse_proxy
• Quack on Demand, Apache-2.0, release 0.7.2: github.com/starlake-ai/quack-on-demand. Client walkthroughs: docs.starlake.ai/qod/connecting/clients; runnable examples in the repository's examples/ directory.