The problem
A new language is easy to describe and difficult to keep coherent. I wanted TeapotLang to be memory-efficiency-oriented, but I also needed a compiler structure that I could understand and change quickly.
The approach
I split the work into explicit stages: source text becomes tokens, tokens become an abstract syntax tree, and later stages can operate on that tree without re-parsing the original input. This kept each decision local and made debugging more concrete.
The lexer owns character-level decisions, while the parser owns structure. That separation means a syntax error can point back to a useful token instead of exposing an implementation detail from a later stage. I also kept the tree deliberately small so the representation reflected the language I actually understood.
How I actually built this
- First pass: the lexer and parser became stable enough to test independently, which exposed mistakes earlier than an end-to-end compiler demo.
- Setback: semantic analysis was broken for a week before I found the bug in how the AST traversal handled top-level nodes.
- Recovery: I added focused semantic-analysis tests and kept CI running ruff, mypy, and pytest so later changes could not quietly undo the fix.
The trade-offs
It would have been tempting to optimise early or add a large feature set. I chose clarity first: straightforward data structures, explicit stages, and tests around the pieces that were already stable. The result is easier to extend because each new language feature has a visible place to belong.
What I learned
Compiler errors are part of the language design, not an afterthought. Giving tokens useful positions and keeping the parser's responsibilities narrow made experiments much easier to evaluate. The biggest lesson was that a compiler is not one enormous transformation: it is a sequence of understandable contracts.