Tiktoken

Problem

How to compute the number of tokens in a prompt with tiktoken ?

Solution

import tiktoken

encoding = tiktoken.encoding_for_model("gpt-4")
prompt_encoded = encoding.encode(prompt)
nb_tokens = len(prompt_encoded)