Specifications / Language

Lexical Structure

Default source is tokenized by tree-sitter-osprey; ML source uses the lexer in src/toolchain/crates/osprey-syntax/src/ml/lexer.rs, which also inserts INDENT, DEDENT, and NEWLINE tokens. Both frontends share string unescaping and interpolation scanning.

This chapter describes Default and ML tokens. The third frontend uses Python's concrete syntax; its marker, indentation, literals and supported forms are specified in Python Flavor Syntax.

Identifiers

Default identifiers use ASCII letters and underscores:

identifier ::= [a-zA-Z_][a-zA-Z0-9_]*

The ML lexer accepts Unicode alphabetic characters in the same leading and continuation positions, plus _; digits remain continuation-only.

Keywords

The Default grammar reserves the words used by its declarations and expressions, including let, mut, fn, extern, type, where, effect, perform, handle, handler, mask, instance, finally, in, resume, spawn, await, yield, send, recv, select, match, if, else, import, namespace, module, signature, export, opaque, state, as, true, and false.

ML reserves the corresponding words its surface uses. It deliberately does not reserve Default-only let, fn, if, or else. handler constructs a callable handler in both flavors; do remains reserved in ML. ML reserves mask and instance; finally is a keyword there only at the head of a handler clause.

in and out are variance markers inside type-parameter declarations (TYPE-VARIANCE-DECL). Outside that position, out is an identifier; in also separates a kernel from its body.

Literals

Numbers

integer ::= [0-9]+
float   ::= [0-9]+ "." [0-9]+

Negative values apply unary -; the sign is not part of the token. Integer literals infer as int and decimal literals as float. Exponent notation, numeric separators, and non-decimal bases are not accepted.

Finite float literals — [FLOAT-LITERAL-RANGE]

A decimal float literal must round to a finite 64-bit floating-point value. Both flavors reject overflow at the literal's source position, including the magnitude of a negative literal. Finite values up to the largest representable float and signed zero remain accepted. This is a literal rule; it does not change the current runtime behavior of float arithmetic.

Booleans and strings

Boolean literals are true and false. Strings are double-quoted. Shared unescaping recognizes \n, \r, \t, \e, \0, \", and \\; an unknown escape is preserved verbatim. Default strings may contain a source newline. The ML lexer rejects a source newline in string text, but permits one inside an interpolation expression.

${ expression } introduces interpolation in either flavor. See String Interpolation.

Lists

list ::= "[" (expression ("," expression)*)? "]"

The list elements must unify to one element type. Map and record literal forms are defined in Type System.

Operators

  • Arithmetic: +, -, *, /, %. Integer +, -, *, % and unary - have type int; / has type float; floating-point +, -, *, and unary - are plain IEEE-754 operations. No arithmetic operator can trap, panic, wrap silently, or yield an unspecified value; an unrepresentable result is discharged by an Arith handler or the program is rejected (ARITH-CHECKED, ARITH-TOTAL).
  • Comparison: ==, !=, <, >, <=, >=.
  • Boolean: &&, ||, !. && and || short-circuit.
  • Calls and data access: (), ., [].
  • Flow and matching: |>, =>, ? :, ?:.
  • Types and effects: :, ->, < >, and ! on an effect set.
  • Namespace qualification: ::.

Default uses = both to initialize a let/mut declaration and to reassign a previously declared mut binding. ML uses = for a binding and := for reassignment. A mut binding is a cell for handler-owned effect state, not a general imperative variable; where its reassignment is valid is defined in Bindings. The checker accepts reassignment only in an effect handler arm.

[LEX-RESULT-DEFAULT-TOKEN] ?: is a single token and is matched before bare ? or :. Its semantics are defined by PATTERN-RESULT-DEFAULT.

Statement Boundaries

Default has no statement terminator: a statement ends where its line ends. The grammar spells that as a zero-width _statement_break token, produced by the external scanner in src/toolchain/tree-sitter-osprey/src/scanner.c.

The rule [LEX-STATEMENT-BREAK]

A statement ends at the first newline, // comment, } closing its block or namespace body, or end of file that follows it. Two statements on one line are a syntax error, because nothing would separate them:

let r = add 2 3     // rejected: Default has no juxtaposition application
print("a") print("b")   // rejected: two statements, one line

Without this rule nothing delimited a statement, and the greedy parse chose the boundary silently. let r = add 2 3 became let r = add followed by the orphan expression-statements 2 and 3; inside a block the orphan was absorbed by the trailing block value, so { let r = double 5 } evaluated to the argument 5 rather than applying double. Both are well-formed trees for source that means a call, which is why the boundary is now a lexical decision rather than an accident of precedence.

A line whose first token can only continue the preceding expression does not begin a statement, so an expression may still be laid out across lines:

let total = buffer
    |> gpuMap(scale)
    |> gpuMap(clamp)

type Json = JNull
    | JBool { value: bool }
    | JNum { value: int }

The continuing tokens are *, %, /, <, >, ?, :, ., |, |>, ||, &&, ==, !=, <=, >=, and ::. +, -, and ! are deliberately absent: each has a prefix reading, so a line opening with one starts a new statement whose discarded value is then reported by BLOCK-DISCARD. Write a continued sum with the operator trailing the first line.

ML needs no such marker. Its lexer emits explicit NEWLINE, INDENT, and DEDENT tokens, so layout already delimits its statements.

Comments

Default uses // line comments and /// documentation comments. ML accepts // line comments, nested (* ... *) block comments, and (** ... *) documentation comments.