For a given input, how many tokens will be used for an answer, and how high quality will that answer be?
Measuring the tokenizer is just one input into the cost-benefit tradeoff.
For a given input, how many tokens will be used for an answer, and how high quality will that answer be?
Measuring the tokenizer is just one input into the cost-benefit tradeoff.