3. Lexical structure

3.1 Layout

Spaces and tabs are interchangeable. A query may sit on one line or many, and indenting a sub-pipeline is cosmetic. A line break is a space too, except in a query written without |, where it can end a stage (3.2).

on users
| where age >= 18
| select name

is the same query as on users | where age >= 18 | select name.

There are no comments. In this book, -- … marks an annotation, never something to type.

3.2 Stage separators

A query separates its stages in one of two styles. The three queries below are the same query:

on users | where age >= 18 | select name

on users; where age >= 18; select name

on users
where age >= 18
select name

Both styles give the same stages, and every rule of this book is the same in both. The query is kept as written: a view stores its body in the style it was defined in.

The pipe

| separates stages.

• A | outside brackets always ends a stage.
• A | inside ( … ), [ … ] or { … } does not: brackets must balance.
• || is the boolean or, never two separators: where a || b is one stage.
• In a query holding a |, line breaks are spaces, as in 3.1.

; and line breaks

A query that holds no | (outside brackets and strings, || aside) separates its stages with ; and with line breaks. ; fits a query on one line (a request, a command line); line breaks fit a query written by hand.

• A ; outside brackets always ends a stage. A final ; is allowed (on users;); an empty stage (;;) is refused.
• A line break ends a stage when all three hold:
  1. brackets are balanced: insert {…} and values [ … ] may span lines;
  2. the line does not end on a continuation: ,, an operator (=, +, ==, >, &&…), or one of the words and or not in as between by on into with from to step default version of keep index view;
  3. the next line starts with a stage word, or with a directive of the block the line is in (below).
• Otherwise the next line goes on with the stage: a line starting with a field, a value, an operator or and continues the line before.
• end closes a block, at the start of a line or after ;.

on orders
where status == "paid"
  and qty > 1                      -- `and` goes on with the `where`
derive total = price * qty,
  big = total > 100                -- the line before ends with `,`
sort total desc

Inside a block, its directives start lines:

on orders
group
  by region
  sum qty as units
  count as orders
end
sort units desc

A source (on, from) starts a stage only after union, after exec, or after a let header:

let $min = 2
on orders
lookup customers
  on _id == customer into who      -- one-line lookup: `on` goes on with it
where qty >= $min

A one-line form stays on one line, or its next lines start with a continuation. Here the second line starts a stage into, so lookup opens a block that no end closes, and the query is refused with the line it read:

on orders
lookup customers on _id == customer
  into who
-- `lookup` at line 2 opens a block that no `end` closes; line 3 starts a new stage `into`: …

To go on with a line whose next word is a stage word, end the line with an operator or a comma, or put the expression in parentheses.

One style per query

• A query uses one style: on t; where a | sort a is refused (`;` at line 1 in a query separated by `|`).
• Text in parentheses keeps its own style: in let $s = (on settings | limit 1) the let query is read on its own, so the query around it may still be written with lines.
• explain stays in front of the query, on the first line or alone on it.

3.3 Identifiers

• Start with a letter (any Unicode letter) or _; continue with letters, digits or _.
• Case-sensitive: Name and name are two fields.
• Unicode works everywhere: on données | where prénom == "Élodie".

Field paths join identifiers with dots: profile.country, a.b.c. A key holding a space or a hyphen can be stored (insert {"display name": "Zed"}) but not written as a path: read it with get(document, "display name").

Optional paths put ? after a segment: nick?, profile?.country. A missing field then reads as null instead of failing (4.4).

Collection names may be dotted too: on shop.orders. Names starting with _ are system collections.

View names are identifiers joined by dots: ledger.cube, sales.by_month.

3.4 Keywords

Reserved by the pipeline lexer:

on  from  as  where  set  lookup  join  union  load  pivot
into  with  chunk  replace  update  merge  end
and  or  not  true  false  null

Inside expressions in is also reserved. between is recognized only after an operand (age between 18 and 30), so it stays usable as a field name.

Other stage names (select, sort, group, limit, window, …) are ordinary identifiers recognized in stage position, and function names are not reserved: round(x) is a call, round alone a field.

Statement words: explain, let, define view, undefine view, and the view clauses description, version, materialized, exec.

3.5 Literals

3.6 Signs

A + or - written directly before a number literal is part of it: -5, - 5 and +2.5 are constants in every stage. Between two operands, + and - are binary: a - 5 subtracts, a > -5 compares with -5. Before a field (-qty), - negates at run time.

3.7 Operators and punctuation