Snowflake App Runtime Data Lineage Integration: An OpenLineage Walkthrough

Why App Runtime Applications Disappear from Metadata

Snowflake App Runtime lets developers build and deploy Next.js applications directly on Snowflake, offering enormous convenience for rapidly shipping internal tools. But once an app goes live, it vanishes from metadata—Snowsight's lineage graph only traces down to the table and view level, and can't answer questions like "which application or API depends on this specific column?"

According to analysis from Bilibili creators and YouTube tech channels, Snowflake's External Lineage API has been officially available for some time now. It accepts lineage events in OpenLineage format and merges them directly into the lineage graph. Meanwhile, Horizon Context, a recently launched Snowflake preview feature, extends Horizon Catalog into a broader, pluggable metadata layer. If App Runtime APIs and pages could appear as regular nodes in the lineage graph—positioned downstream of the tables they read—then much of the guesswork around "what breaks if I change this column" would be eliminated.

Note: Horizon Context itself was still in preview as of October 2026, and related frameworks may change. However, the External Lineage mechanism this article relies on is already GA.

OpenLineage is an open-source data lineage standard under the LF AI & Data Foundation, originally initiated by Datakin (now Astronomer), with the goal of providing a unified lineage event format independent of schedulers and compute engines. Its core model consists of three types of objects: Job (the unit of work that produces lineage, such as an Airflow DAG, a Spark job, or an API endpoint), Run (a specific execution instance of a Job, with a unique runId), and Dataset (the data read from or written to, which can carry a namespace). These three are connected by inputs / outputs edges. Events themselves are sent as JSON payloads, typically containing fields like eventType (START, RUNNING, COMPLETE, ABORT, FAIL), eventTime, producer, plus extensible facets—the most flexible design in OpenLineage: all non-core information (schema, column-level lineage, data quality, ownership) is attached via named facets, and receivers can parse as needed and ignore unknown facets. It's precisely because of this "stable core, free extension" structure that Snowflake can pick just two facets—datasetType and columnLineage—to implement, while silently dropping documentation and ownership without breaking overall protocol compatibility.

Structure of the Sample Application

The author built a small Next.js application using dbt-labs' jaffle_shop sample data, containing an analytics dashboard and a master data editor page. The app follows standard Next.js App Router conventions: API routes live under app/api/, and pages live under app/ and components/.

Five categories of information need to be automatically extracted from the directory structure: API endpoint names, HTTP methods, referenced tables, page node names, and page-to-API mappings. For example, app/api/customers/route.ts maps to /app/api/customers, and a fetch("/api/customers") call in customers-view.tsx establishes the link between the page and the API.

The entire pipeline is divided into three steps: Extract—use regex to pull SQL template literals from TypeScript API route files; Analyze—use sqlglot to parse the SQL, resolve fully qualified table names, and trace column-level lineage; Send—construct OpenLineage COMPLETE events and push them to the Snowflake REST API.

Three-Step Lineage Integration

Step One: Extracting SQL from TypeScript

Each App Runtime API route lives at app/api/<name>/route.ts, following Next.js App Router conventions, with SQL written as template literals. The author uses a regular expression to extract these SQL templates from TypeScript code, supporting two forms: the direct const NAME = \...`and the arrow function formconst NAME = (db, schema) => `...``. The HTTP method is inferred from the exported function names and variable names.

Step Two: Parsing SQL with sqlglot and Tracing Column-Level Lineage

The extracted queries need preprocessing: replace ${db} template placeholders with actual values and convert Snowflake bind variables (:name, ?) into placeholders. sqlglot parses the SQL into an abstract syntax tree, from which all table references are extracted.

Its lineage() API goes further, tracing the exact source table and column for each output column of a SELECT. Passing dialect="snowflake" correctly handles column-level attribution for JOINs, CASE expressions, and aggregate functions.

One key prerequisite: lineage() needs schema information to work, so the author connected to Snowflake and ran DESCRIBE TABLE in advance to obtain the table structures.

Column-level lineage differs from table-level lineage in tracking granularity: table-level only answers "which tables did this query read," while column-level answers "for each output column, which input table column did it come from and what transformations were applied." This matters enormously in practice—when you're about to rename or modify CUSTOMERS.CUSTOMER_NAME, table-level lineage can only tell you that three downstream tasks depend on the CUSTOMERS table, while column-level lineage precisely identifies which tasks and which fields actually use this column, avoiding collateral damage. sqlglot's lineage() works by parsing SQL into an AST, then bottom-up attributing each SELECT output expression to its source: direct column references map to source tables, CASE expressions are split across branches, and aggregate functions map to their argument columns. It handles JOINs, CTEs, subqueries, and UNIONs fairly completely. Its limitation is that it performs static, syntax-level inference—when encountering SELECT *, dynamically concatenated SQL, or logic inside UDFs, it can't fully reconstruct the true dependencies. Therefore schema information (what columns exist, what types they are) is a necessary input for expanding * and validating column attribution.

Step Three: Building and Sending OpenLineage Events

Two types of nodes are registered as lineage: Layer 1 establishes the relationship between tables and API endpoints, including column-level lineage; Layer 2 establishes the relationship between API endpoints and page nodes. Page node names include the application name, e.g., jaffle-app/dashboard.

OpenLineage is an open data lineage format, and Snowflake accepts it at the /api/v2/lineage/external-lineage endpoint and merges it into the lineage graph. Layer 1's read events carry the columnLineage facet, but Snowflake currently only recognizes two facets: datasetType and columnLineage. Other facets like documentation and ownership are silently ignored.

Horizon Catalog is the metadata brand Snowflake formally established around 2024, unifying previously scattered governance capabilities—permissions, objects, lineage, tags—under a single catalog layer. It is both the entry point in Snowsight for browsing tables, columns, and tags, and the underlying support for capabilities like GET_LINEAGE, policy tags, and object tags. Horizon Context is an extension attempt built on top of this: transforming the Catalog from "managing only Snowflake's own objects" into "a pluggable, general-purpose metadata layer," connecting external systems' metadata (such as dbt, Looker, Airflow) via connectors so the lineage graph can span multiple tech stacks. It's specifically mentioned in this article because if App Runtime applications and APIs could one day be registered as first-class citizens in Horizon Context, there would be no need to rely on manually pushing OpenLineage events. But as of the time noted in the article, it remains in preview, and its interfaces and model may change—so the author chose the already-GA External Lineage as a stable implementation path.

Actual Results and Queries

Before integration, Snowflake's native query lineage could trace the CUSTOMERS table back to staging and raw tables, but no further—no applications, no APIs, a blank downstream. After events are registered, the application's APIs appear in the lineage of the tables and views they touch. Nodes registered via OpenLineage are marked as External.

Write-type APIs (such as the PUT-style master data editor) appear upstream of the tables, since they write rather than read.

The most practical scenario is change impact analysis. Suppose you want to modify the definition of CUSTOMERS.CUSTOMER_NAME—Snowflake's built-in GET_LINEAGE function can precisely report the affected objects. A single query reveals that modifying CUSTOMER_NAME affects both the /api/customers and /api/orders endpoints. The same path works through Snowflake's CoCo interface.

GET_LINEAGE is a Snowflake table function for querying the lineage graph programmatically, and internally it uses the same Horizon metadata that powers Snowsight's lineage graph. Its typical invocation is SELECT * FROM TABLE(SNOWFLAKE.CORE.GET_LINEAGE('<object_fqn>', '<direction>', '<object_type>')), where direction can be DOWNSTREAM or UPSTREAM. Compared to the graphical interface, its value lies in composability—the returned results can be further JOINed, filtered, aggregated, and embedded in scripts or CI checks. For example, in a column-level scenario, passing the target column name as a filter condition turns "which downstream objects would break if I change this column" from manual investigation into a reusable SQL query, or even an automated blocking gate before schema changes. Note that this function returns object-level dependencies; drilling down to specific columns still requires combining column-level lineage information.

Reflections and Outlook

Horizon Context's metadata connectors handle simple scenarios—integrating systems that already have connectors. OpenLineage applies to cases where no ready-made connector exists: it lets an application that was previously invisible to Snowflake's lineage graph register itself API by API, page by page, with full column-level detail.

Although Snowflake's support for OpenLineage facets is still fairly limited, columnLineage already covers the core column-level tracking needs. If you've ever had to dig through source code to figure out the impact of a schema change, this approach offers a pragmatic starting point.

For teams building internal tools on Snowflake with an eye on data governance, bringing the application layer into the lineage graph is a worthwhile investment.

One-line Summary

A three-step pipeline using regex, sqlglot, and OpenLineage to add Snowflake App Runtime applications into the lineage graph.

Paragraph Summary

Snowflake App Runtime lets developers deploy Next.js applications directly on Snowflake, but these applications don't appear in the Snowsight lineage graph by default, making it impossible to answer questions like "which application or API breaks if I change a certain column." The article presents a three-step pipeline: first use regex to extract SQL template literals from TypeScript API routes, then use sqlglot to parse the SQL and trace column-level lineage via its lineage() API, and finally construct the results into OpenLineage COMPLETE events pushed to Snowflake's External Lineage endpoint. After integration, read-type APIs appear downstream of tables, write-type APIs appear upstream, and all are marked External. The most practical benefit is change impact analysis—a single GET_LINEAGE query can reveal which endpoints are affected by modifying CUSTOMERS.CUSTOMER_NAME. The author also notes that Snowflake currently only recognizes two facets: datasetType and columnLineage.