Tokenizing

Turn a byte stream into tokens, tracking position for good error messages.

#[derive(Debug, PartialEq)]
enum Token {
    LBrace, RBrace, LBracket, RBracket, Colon, Comma,
    String(String),
    Number(f64),
    True, False, Null,
}

struct Lexer<'a> {
    input: &'a [u8],
    pos: usize,
}

The <'a> is a lifetime parameter: it says the Lexer borrows its input and cannot outlive it. You don’t write lifetimes often, but a struct holding a reference is exactly the case that requires one.

Track pos even when you don’t need it yet — the difference between “invalid JSON” and “invalid JSON at byte 412” is the difference between a toy and a tool.