Bug: breaks when buffer temporary lose "buflisted" option

[backdround]

Describe the bug
It breaks when i hide buffer from buffer list. It doesn't restore highlihting even after option restoration.

Works on minimal.lua config from Lazy.

To Reproduce
Open file with color highlights and type:

:set nobuflisted | set buflisted

Expected behavior
It will highlight (after option restoration at least)

Screenshots
before:

after:

Operating System:
Archlinux

Neovim Version:
v0.8.3

Colorizer Version:
dde3084 [master]