Tokenize text
POST /v1/tokenize
Tokenizes text with the selected model's tokenizer. The local worker takes content; a remote provider may expose a different tokenization contract, which the gateway forwards. Token IDs are model-specific, so do not hard-code them.
Guide and examplesLanguage and visionAll endpoints
Request body
application/jsonRequired
modelstringRequiredModel whose tokenizer is used. The same text yields different token IDs on different models.contentstringRequiredText to tokenize.Responses
200The token IDs. The array is illustrative; count its length when only an array is returned.tokensinteger[]{
"tokens": [
9707,
1879,
13
]
}400Invalid request. Fix the request before retrying; use error.code for program logic and error.param to locate the input.401Invalid or missing API key.403Host or origin is not allowed, or the license was rejected. Inspect error.code to tell them apart.413Request body exceeds 512 MiB.503A required local model is still being prepared (model_downloading) or the service is busy (service_busy). Honor Retry-After when present.