'PreTrainedTokenizerFast' object has no attribute 'tokenizer'

[schwjustin]

When I use this code, I get the error 'PreTrainedTokenizerFast' object has no attribute 'tokenizer':

import tensorrt_llm
from tensorrt_llm.runtime import ModelRunner
from transformers import AutoTokenizer

...

self.tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

...

from lmformatenforcer import JsonSchemaParser
from lmformatenforcer.integrations.trtllm import (
  build_trtllm_logits_processor,
  build_trtlmm_tokenizer_data
)

schema = {
  "type": "object",
  "properties": {
    "answer": {
      "type": "string",
      "title": "Answer"
    },
  },
  "required": ["answer"]
}

parser = JsonSchemaParser(schema)

tokenizer_data = build_trtlmm_tokenizer_data(tokenizer)

logits_processor = build_trtllm_logits_processor(tokenizer_data, parser)

[schwjustin]

Any ideas how to get this working?

[noamgat]

I tried applying a fix for a similar issue from a different integration. Can you try it?

pip install git+https://github.com/noamgat/lm-format-enforcer.git@bugfix/trtllm-types

[schwjustin]

When I add pip install git+https://github.com/noamgat/lm-format-enforcer.git@bugfix/trtllm-types and try to run I get this error and the code doesn't actually get to execute:

`Building image im-yl8Bxa0rQFHd4TicxFGIre

=> Step 0: FROM base

=> Step 1: RUN python -m pip install git+https://github.com/noamgat/lm-format-enforcer.git@bugfix/trtllm-types lm-format-enforcer tensorrt_llm==0.10.0.dev2024042300 --extra-index-url https://pypi.nvidia.com --pre
Looking in indexes: http://pypi-mirror.modal.local:5555/simple, https://pypi.nvidia.com
Collecting git+https://github.com/noamgat/lm-format-enforcer.git@bugfix/trtllm-types
Cloning https://github.com/noamgat/lm-format-enforcer.git (to revision bugfix/trtllm-types) to /tmp/pip-req-build-7lzdakej
Running command git clone --filter=blob:none --quiet https://github.com/noamgat/lm-format-enforcer.git /tmp/pip-req-build-7lzdakej
Running command git checkout -b bugfix/trtllm-types --track origin/bugfix/trtllm-types
Switched to a new branch 'bugfix/trtllm-types'
Branch 'bugfix/trtllm-types' set up to track remote branch 'bugfix/trtllm-types' from 'origin'.
Resolved https://github.com/noamgat/lm-format-enforcer.git to commit ff5cd86
Installing build dependencies: started
Traceback (most recent call last):
File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/pkg/modal/_container_entrypoint.py", line 622, in
main(container_args, client)
File "/pkg/modal/_container_entrypoint.py", line 542, in main
metadata: Message = container_app.object_handle_metadata[object_id]
KeyError: 'im-hmf2kfuwV3nTWZPoMiK6rP'`

I tried only pip installing this and pip installing both this and lm-format-enforcer. I get this error either way

[noamgat]

It looks like something in your build setup.
An alternative way to check the fix is to install the normal package from pip, and add your own copy of trtllm.py with the contents of this:
https://github.com/noamgat/lm-format-enforcer/blob/ff5cd86a9f007340ae73e043c419c23ba1438075/lmformatenforcer/integrations/trtllm.py
and then import from there instead of the from lmformatenforcer.integrations.trtllm import

[schwjustin]

That worked! Thank you