[#3] Make tokenizer more generic #8

Merged

zyxw59 merged 4 commits from 3-generic-tokenizer into main

2023-07-04 20:51:08 +00:00

zyxw59 commented

2023-07-04 17:35:17 +00:00

(Migrated from github.com)

This adds a new Tokenizer trait, which allows for other tokenizers to be used. It also adds the CharSetTokenizer (and associated CharSet trait) which allows for tokenizer implementations based on grouping characters into tokens based on some categorization scheme.

(closes #3)

This adds a new `Tokenizer` trait, which allows for other tokenizers to be used. It also adds the `CharSetTokenizer` (and associated `CharSet` trait) which allows for tokenizer implementations based on grouping characters into tokens based on some categorization scheme. (closes #3)

zyxw59 (Migrated from github.com) reviewed

2023-07-04 20:45:58 +00:00

src/token.rs

exponents in floating point numbers are no longer supported by the default tokenizer; they can be handled at the parser/evaluator layer instead, by making e a binary operator.

exponents in floating point numbers are no longer supported by the default tokenizer; they can be handled at the parser/evaluator layer instead, by making `e` a binary operator.