14. Execution model

14.1 From text to rows

Planning errors are raised before any data is read. Plans are cached: a repeated query skips parsing and planning.

14.2 Filters move toward the source

The planner splits where a and b into where a | where b (same result, same errors, same order) and moves each filter before the stages that leave its fields and the set of rows alone:

It stops at any other stage (limit, skip, first, distinct, fill…) and after another filter, whose order it keeps. A guard then goes on alone: for each comparison of a field with a constant, type(f?) != "string" or f? == "x", which cannot fail and drops only rows the filter would drop. The filter itself stays where it stopped, so no row reaches it that did not before, and no error is lost. explain shows the result.

A filter written after a view therefore reaches the view’s source:

let $end = "2026-03" | on view ledger.statement | where entity == "E1"

reads the ledger with the guard on entity right after the view’s own date filter, instead of classifying every entity first (10.2 s → 0.25 s on a ledger of 1.8 M rows).

14.3 Execution shapes

ogd never falls back to loading everything in memory. A read query fits one of these shapes, or is refused before it runs.

1. Pure streaming. Any mix of where, select, derive, rename, drop, root, near, skip, limit, first, single, unwind, optionally ending with count. Rows flow one by one; a row bound (limit, first) stops the scan early.

2. A streaming prefix, then blocking stages. The blocking stages are sort, distinct, group, bucket, pivot, window, fill, sample, lookup and union. Any number may follow one another, each followed by row-local stages, and the query may end with count.

• The first blocking stage reads the streaming prefix; group | sort and sort | distinct run as one stage.
• Each later blocking stage takes the rows of the stages before it once they are all produced.
• A leading skip k / limit n / first after a blocking stage bounds it, even across select, derive, rename, drop, root; a stage that can drop rows (where, near) stops that: sort k | skip 10 | limit 5 is a bounded Top-15.
• sort, group, pivot, window and fill spill sorted runs to disk when their state outgrows the memory they can get.

Anything else is refused:

query plan contains a blocking stage sequence without a bounded executor;
materialized fallback is forbidden by the memory contract

3. Transactional. Writes (set, delete, load, insert, into) and system collections (_…) take the transactional path, which these shapes do not restrict.

14.4 Memory

Every query is admitted with a small reservation and reserves memory as its blocking stages grow; all queries share the query budget of the memory profile. OGD_MEMORY_LIMIT sets the profile; on _memory and on _query_memory show it live.

A request waits for a query slot when all are busy (query wait, 60 s by default).

14.5 Joins at scale

• An equality or prefix lookup indexes the joined source once per query.
• A deterministic sub-pipeline reading only known outer fields is run once per distinct value of those fields.
• The index and kept results live within the query working set; past it, rows are joined by scan.

14.6 Grouping at scale

• group reads the storage’s per-field kinds to decide the compact form’s roles before the scan.
• A one-key group whose only stage before is a derive of + - * / over number fields is computed on each storage block’s typed columns (37 → 22 ms on 2 M rows).
• group | limit N keeps only the N smallest live keys.

14.7 Tips

• Filter early and on fields that exist everywhere.
• select what you need before a blocking stage: smaller rows, smaller spills.
• Prefer sort … | limit N to sort alone.
• Materialize costly shapes read by many queries (chapter 12).
• Read the plan: explain ….