I chose Prefect 3 to orchestrate a private market-data and strategy platform. The workload mixes fixed schedules with parameterized backfills, event-driven child flows, long-running calculations and manual operations. Most implementation code is Python and many workflows need to be invoked by other tools as well as run as deployments or ordinary functions during testing.

Why Prefect fits this workload

Prefect keeps orchestration close to normal Python. That matters when the same domain functions support a scheduled daily ingestion, a bounded historical backfill and an ad-hoc repair. Parameters, branching and composition remain direct rather than being forced into a static DAG shape.

Dynamic execution: the workflow can derive symbols, dates and child deployments at runtime.
Ad-hoc and scheduled parity: the same flow can run on a schedule or be launched with explicit parameters.
Incremental adoption: Python functions can gain task boundaries, retries and observability without a full rewrite.
Tool integration: APIs and deployment calls let applications, operational tools and AI-assisted workflows launch the exact bounded flow they need.
Self-hosted control: the control plane and workers can remain inside the private operating environment.
Cost and footprint: the platform can run with less infrastructure and fewer dedicated resources at this scale.

Why not automatically choose Airflow?

Airflow is excellent when an organization needs a mature ecosystem, broad operator support, highly visible DAG ownership and a familiar standard across many data teams. Its scheduler-centric model is a strength for large estates of regular batch dependencies.

Those strengths were not the dominant constraint here. This platform is maintained as a Python product, has many parameterized and externally launched workflows, and evolves quickly across ingestion, analytics and strategy logic. Prefect provided faster development, more runtime flexibility and a lower infrastructure burden for the scale of the system.

The trade-offs are real

Choosing Prefect does not remove operational work. The control plane needs resource tuning. Scheduler catch-up and concurrency require deliberate limits. Client and server behavior can change across versions, and Airflow still has the larger ecosystem and talent pool.

If the platform grew into hundreds of teams sharing thousands of mostly static batch DAGs, or depended heavily on existing Airflow operators and governance, I would reassess. Architecture decisions should expire when their assumptions expire.

The orchestrator cannot supply reliability by itself

Market data is imperfect: providers are late, instruments are illiquid, calendars differ and upstream history gets republished. The platform therefore adds explicit policies around the orchestrator:

Simplified resilient flow
@flow(timeout_seconds=policy.deadline)
def market_data_ingestion(target_date=None):
    date = resolve_business_date(target_date)
    symbols = run(fetch_symbols, policy.fetch)
    prices = run_parallel(fetch_prices, symbols)
    coverage = validate(prices, date)
    if not coverage.acceptable:
        raise IncompleteMarketData(coverage)
    finalize_atomically(date, prices, provenance=True)

The decision in one sentence

Prefect fits because the platform is a dynamic Python application with orchestration needs: it accelerates development, integrates with other operational tools and meets the workload with a smaller infrastructure footprint.

Airflow remains a strong alternative. The senior decision is not loyalty to one tool; it is understanding the workload well enough to name the fit, the cost and the condition that would trigger a change.