vodorod/Dorod-Sky
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

cache tiktoken (#4521)

Browse Source 
Co-authored-by: Shuchang Zheng <wintonzheng0325@gmail.com>
This commit is contained in:
LawyZheng
2026-01-23 05:55:45 +08:00
committed by GitHub
parent 51fd789f09
commit 8be0669b04
2 changed files with 7 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
scripts/load_tiktoken.py Normal file
View File

@@ -0,0 +1,4 @@
import tiktoken
for model in tiktoken.model.MODEL_TO_ENCODING.keys():
tiktoken.encoding_for_model(model)