The Runtime Theory
mediumcrafting-interpreters#lexing#finite-state-machine

Write a Tokenizer for Arithmetic Expressions

Scan numbers, identifiers, parentheses, and operators while preserving source positions for clear parser errors.

The Runtime Theory Team1 min read
Solve it

Solving happens on the judge — come back and mark it done

Sample cases

intotal = price * 2 + tax

outIDENT, EQUAL, IDENT, STAR, NUMBER, PLUS, IDENT, EOF

in(7 + 2) / 3

outLEFT_PAREN, NUMBER, PLUS, NUMBER, RIGHT_PAREN, SLASH, NUMBER, EOF

in12.5 + @

outRecognize the number, report an error at @, and continue or stop by documented policy

Build a scanner that converts a source string into tokens for identifiers, decimal numbers, + - * /, parentheses, assignment, and end of input. Each token should retain its lexeme or value and the source offset where it began.

Define what characters are allowed in identifiers and how a decimal point is handled. Report invalid input with a position instead of silently dropping it. Add tests for empty input, multi-digit numbers, whitespace, and an unfinished decimal according to your chosen grammar.

After tokenization, predict the syntax tree for one expression with mixed operators. The linked chapter explains a scanner as one stage of a language implementation; parsing comes next.

One dispatch a week

The trace behind each problem, the tradeoff that explains it, and one technical dispatch per week — no noise.

One technical dispatch per week. No noise.

Not started

Sign in to save your learning progress.

Sign in to save