Flyte 2 Goes GA: Goodbye DAG, Hello Pure Python-Native Orchestration in a Complete Rewrite

Flyte 2 Goes GA: A Complete Architectural Rewrite

The Union AI team recently announced the general availability (GA) of Flyte 2. As an open-source project under the Apache 2.0 license, Flyte 2 is not an iterative upgrade on the existing codebase—it's a complete rewrite. The team removed the legacy DSL (Domain-Specific Language), eliminated the mandatory DAG (Directed Acyclic Graph) construction requirement, and even stripped out core components like Propeller from the architecture.

It's worth explaining two key concepts here. A DSL (Domain-Specific Language) is a programming language or configuration syntax designed for a specific application domain, as opposed to general-purpose languages like Python or Java. In workflow orchestration, DSLs are typically used to describe dependencies and execution order between tasks. A DAG (Directed Acyclic Graph) is a data structure where nodes represent tasks and directed edges represent dependencies—"acyclic" means no circular dependencies exist. Tools like Apache Airflow require users to express all workflow logic as DAG structures. While conceptually clear, this forces developers to employ numerous workarounds to bypass DAG's static limitations when dealing with dynamic branching, conditional loops, or runtime decisions.

Propeller was the core execution engine in the Flyte 1.x architecture. Running as a Kubernetes Operator within the cluster, it was responsible for monitoring workflow execution state, scheduling task Pods, handling retry logic, and more. It drove workflow progression by continuously polling CRD (Custom Resource Definition) states. While Propeller proved robust in production environments, its deep coupling with the Kubernetes API and CRD-based state management model could become performance bottlenecks at scale, while also increasing overall system complexity. Flyte 2's removal of Propeller signals the adoption of an entirely new execution scheduling mechanism.

There are multiple considerations behind such sweeping changes, but the team explicitly states that the core driving force is developer experience. In traditional workflow orchestration tools, data scientists, ML engineers, and researchers are often forced to decompose their work into DAG-compliant structures and learn an additional DSL. For people already focused on algorithms and experimentation, this represents a significant barrier to entry.

From DAGs to .task Decorators: Lowering the Orchestration Barrier

Orchestration Logic Returns to Pure Python

The most visible change in Flyte 2 is the radical simplification of orchestration. The previously tedious process of explicitly declaring node dependencies and constructing DAGs has been replaced by a simple .task decorator—add the decorator, and the task is incorporated into the orchestration system.

More importantly, since the entire framework is built on pure Python, developers can directly use the language's native control flow. The team specifically emphasizes that try/except, loops, and asyncio async programming all work "out of the box." This means complex branching logic, error handling, and concurrency control no longer need to be expressed through specific DSL abstractions—they return to the programming paradigm engineers know best.

Recovering from OOM Crashes: Fine-Grained Fault Tolerance

This pure Python design enables a highly practical capability: developers can recover from Pod crashes caused by OOM (Out of Memory) at the code level.

OOM is one of the most common causes of task failure in Kubernetes environments. When a container's memory usage exceeds the Pod's resource limits, the Linux kernel's OOM Killer forcefully terminates the process, and Kubernetes marks the Pod as OOMKilled. In traditional orchestration systems, OOM-induced Pod crashes typically can only trigger a full task retry—an extremely costly outcome for large-scale training tasks that have already been running for hours.

Flyte 2 allows catching such failures at the Python code level and executing recovery logic (such as switching to a higher-memory environment or reducing batch size). Essentially, it delegates fault recovery control from the platform layer to the application layer. In traditional DAG-based orchestration systems, task failure often means the entire pipeline is interrupted. Flyte 2 gives developers more fine-grained fault tolerance control—particularly critical for long-running training tasks.

Environments: Declarative Resource Management

Drawing from their own Kubernetes background, the project authors identified a long-standing pain point: the fragmentation and complexity of resource declarations and container image definitions in application manifests.

In standard Kubernetes workflows, developers need to write YAML-formatted Pod Specs or Deployment manifests to declare resource requirements (requests/limits), volume mounts, environment variables, etc., while simultaneously maintaining Dockerfiles to define dependencies in container images. The disconnect between these two is a major pain point in practice: modifying a single Python dependency might require rebuilding the image, pushing to a Registry, and updating the image tag in the manifest—a process that can take anywhere from minutes to tens of minutes.

Flyte 2 introduces "Environments" as the solution. Developers can define any number of environments for any pipeline. At runtime, Pods are automatically provisioned according to the specified configuration, including:

• CPU / RAM / GPU compute resources
• OS-level packages
• Python dependencies

This declarative environment management binds resource configuration more tightly with business logic—similar to combining Heroku-style developer experience with underlying Kubernetes capabilities, where the platform automatically handles image building and Pod configuration. The team believes this can significantly accelerate iteration speed during the experimentation phase—researchers can quickly adjust runtime environments and validate ideas without constantly switching between YAML manifests and code.

Data Lineage and Version Control: Ensuring Reproducibility

For machine learning and data engineering scenarios, reproducibility is paramount. Flyte 2 provides comprehensive support here: all data inputs, outputs, and executed code are captured and version-stored in object storage.

Object storage services (like AWS S3, Google Cloud Storage, or MinIO) store binary objects of any size as flat key-value pairs, making them naturally suited for datasets, model files, and execution snapshots. Data lineage tracks the complete transformation path of data from source to final output, including which code processed the data, input data versions, and environment configurations at execution time.

In ML scenarios, data lineage is the foundation for experiment reproducibility—without it, teams often cannot answer critical questions like "what data was this model trained on" or "why can't last week's experimental results be reproduced." Tools like MLflow and DVC offer similar capabilities, but Flyte 2 builds this into the orchestration layer, eliminating the cost of additional integration.

This means every execution has complete lineage traceability. Teams can clearly identify which version of code produced a given result and under what data inputs. For teams that need auditing, experiment reproduction, or issue debugging, this is an indispensable foundational capability.

Product Positioning: An Alternative to Kubeflow and Airflow

From a product positioning standpoint, Flyte 2 explicitly places itself in the competitive arena of workflow orchestration tools like Kubeflow and Airflow.

Apache Airflow was originally developed by Airbnb, with its core design oriented toward data pipeline scheduling. It uses Python to write DAG definitions but relies on external systems (like Spark or databases) for execution logic. Its Executor model (Celery/Kubernetes/Local) determines how tasks actually run. Kubeflow is a Google-led ML platform project that includes Pipelines (based on Argo Workflows), Katib (hyperparameter tuning), KFServing (model serving), and other components, with deep dependencies on Kubernetes primitives like PVC, Service, and Istio.

This space has long faced several core tensions:

• Expressiveness vs. learning curve: Airflow's DAG model is intuitive but offers limited support for complex dynamic logic (its DAGs are static at parse time; dynamically generating tasks at runtime requires fairly hacky approaches). More flexible solutions often come with steep learning curves.
• Kubernetes-native vs. developer-friendly: Kubeflow is deeply tied to K8s—powerful but burdened with too many components and high operational complexity. Resource and image management isn't friendly to algorithm engineers—a data scientist shouldn't need to understand PersistentVolumeClaim or Istio VirtualService to run a training task.

Flyte 2 attempts to resolve both tensions simultaneously through its "pure Python + decorators + environment abstractions" combination: retaining fine-grained control over Kubernetes resources while hiding complexity behind a clean Python API.

Observations and Reflections: The Evolution of Orchestration Tools

The workflow orchestration space has never lacked tools, but the real competitive moat often lies not in feature lists but in whether developers are willing to use it for their daily work. Flyte 2's strategy of removing DSLs and embracing native Python reflects a clear industry trend: orchestration tools are shifting from "requiring users to adapt to the tool" toward "making the tool adapt to users' habits."

This trend is also evident in the broader infrastructure tooling space—Pulumi replacing Terraform's HCL with general-purpose programming languages, CDK replacing CloudFormation's YAML with TypeScript/Python—all explorations in the same direction. The core logic is: when target users are already proficient in a general-purpose language, designing specialized syntax for them only adds cognitive burden rather than reducing complexity.

Of course, a complete rewrite also carries risks—removing mature components like Propeller and restructuring the core architecture poses new challenges for migration costs of existing Flyte 1.x users, ecosystem compatibility, and production stability. The answers to these questions still require time and real-world large-scale deployments to validate.

For teams currently evaluating MLOps orchestration solutions who find themselves frustrated by Airflow's DAG constraints or Kubeflow's complexity, Flyte 2 at least offers a new option worth serious consideration. The project is available at www.flyte.org for interested developers to explore further.