Reduce dependencies #22

New issue

Closed

opened 2025-08-01 21:49:22 +00:00 by zyxw59 · 0 comments

zyxw59 commented

2025-08-01 21:49:22 +00:00

(Migrated from github.com)

Currently, we have 5 direct dependencies:

bstr: used to get the char_indices method on &[u8] in the BufReadSource implementation
bytes: used for the BufReadSource implementation
itertools: actually we just use its dependency, Either, as an error type for the CharSetTokenizer
thiserror: used for deriving Error
unicode-xid: used for the is_xid_start and is_xid_continue functions in the SimpleCharSet tokenizer

Four of these are only used for the default tokenizers provided by this library, which mostly exist as examples (since most users will have their own desired tokenization method), and the last mainly exists to slightly reduce a small amount of boilerplate, while bringing in a proc-macro (and syn) dependency.

Moving the CharSetTokenizer and related structs to a separate library, or at least an optional feature, and replacing the thiserror usage with handwritten Display and Error impls, could make this crate dependency-free, reducing compile time impacts.

Currently, we have 5 direct dependencies: - `bstr`: used to get the `char_indices` method on `&[u8]` in the `BufReadSource` implementation - `bytes`: used for the `BufReadSource` implementation - `itertools`: actually we just use its dependency, `Either`, as an error type for the `CharSetTokenizer` - `thiserror`: used for deriving `Error` - `unicode-xid`: used for the `is_xid_start` and `is_xid_continue` functions in the `SimpleCharSet` tokenizer Four of these are only used for the default tokenizers provided by this library, which mostly exist as examples (since most users will have their own desired tokenization method), and the last mainly exists to slightly reduce a small amount of boilerplate, while bringing in a proc-macro (and `syn`) dependency. Moving the `CharSetTokenizer` and related structs to a separate library, or at least an optional feature, and replacing the `thiserror` usage with handwritten `Display` and `Error` impls, could make this crate dependency-free, reducing compile time impacts.