Calculate the number of language model tokens for a string

by waldekm • 1 year ago

17 3 1 year ago

sva_• 1 year ago

  import tiktoken
  enc = tiktoken.encoding_for_model('o1-mini')
  len(enc.encode('some string'))

NitpickLawyer• 1 year ago

Does that work on anything other than openai?

topicseed• 1 year ago

No but most models have their own libs you can use

Related Stories