dnrops
/
transformers
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Render chat template tojson filter as unicode (#31041) 

* Render chat template tojson filter as unicode

* ruff--
This commit is contained in:
Sigbjørn Skjæret 2024-05-28 16:02:51 +02:00 committed by GitHub
parent 4f98b14465
commit 22dab246c5
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
src/transformers/tokenization_utils_base.py
View File

@ -1853,6 +1853,7 @@ class PreTrainedTokenizerBase(SpecialTokensMixin, PushToHubMixin):
raise TemplateError(message)
jinja_env = ImmutableSandboxedEnvironment(trim_blocks=True, lstrip_blocks=True)
jinja_env.policies["json.dumps_kwargs"]["ensure_ascii"] = False
jinja_env.globals["raise_exception"] = raise_exception
return jinja_env.from_string(chat_template)