Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] identifies UTF16LE for a pair of ascii punctuation characters #509

Closed
GavinHuttley opened this issue Aug 12, 2024 · 1 comment
Closed
Labels
detection Related to the charset detection mechanism, chaos/mess/coherence
Milestone

Comments

@GavinHuttley
Copy link

GavinHuttley commented Aug 12, 2024

Describe the bug
Introducing conventional ascii text returns UTF-16LE encoding

To Reproduce

import chardet, charset_normalizer

charset_normalizer.detect(b");")  # error also happens with b"(;"
# returns  {'encoding': 'utf_16_le', 'language': '', 'confidence': 1.0}
chardet.detect(b");")
# {'encoding': 'ascii', 'confidence': 1.0, 'language': ''}

Expected behavior

These are standard ASCII characters, I expect a UTF-8 encoding

Desktop (please complete the following information):

  • macOS 14.5
  • Python version 3.12.1 (anaconda build)
  • charset_normalizer version 3.3.2

Additional context
Evaluate either b"(", b")", b";" or b"()" produces the expected result. There are other combinations of punctuation characters that produce the same error, e.g. b".;".

I understand this is a very small string but perhaps a default to the minimum character set?

@GavinHuttley GavinHuttley added bug Something isn't working help wanted Extra attention is needed labels Aug 12, 2024
@Ousret Ousret added this to the Next release milestone Sep 21, 2024
@Ousret Ousret added detection Related to the charset detection mechanism, chaos/mess/coherence and removed bug Something isn't working help wanted Extra attention is needed labels Sep 21, 2024
Ousret added a commit that referenced this issue Sep 25, 2024
Ousret added a commit that referenced this issue Sep 25, 2024
@Ousret
Copy link
Member

Ousret commented Sep 25, 2024

We fixed that case. It will be available in the next release.

@Ousret Ousret closed this as completed Sep 25, 2024
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 9, 2024
##### v3.4.0 (`https://github.com/Ousret/charset_normalizer/blob/HEAD/CHANGELOG.md#340-2024-10-08`)

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 9, 2024
##### v3.4.0 (`https://github.com/Ousret/charset_normalizer/blob/HEAD/CHANGELOG.md#340-2024-10-08`)

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 9, 2024
##### v3.4.0 (`https://github.com/Ousret/charset_normalizer/blob/HEAD/CHANGELOG.md#340-2024-10-08`)

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 9, 2024
##### v3.4.0 (`https://github.com/Ousret/charset_normalizer/blob/HEAD/CHANGELOG.md#340-2024-10-08`)

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 9, 2024
##### v3.4.0 (`https://github.com/Ousret/charset_normalizer/blob/HEAD/CHANGELOG.md#340-2024-10-08`)

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 13, 2024
##### v3.4.0 

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 13, 2024
##### v3.4.0 

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 13, 2024
##### v3.4.0 

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 14, 2024
##### v3.4.0 

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 14, 2024
##### v3.4.0 

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 14, 2024
##### v3.4.0 

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 14, 2024
##### v3.4.0 

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
zemnmez-renovate-bot added a commit to zemn-me/monorepo that referenced this issue Oct 14, 2024
##### v3.4.0 

##### Added

-   Argument `--no-preemptive` in the CLI to prevent the detector to search for hints.
-   Support for Python 3.13 ([#512](jawah/charset_normalizer#512))

##### Fixed

-   Relax the TypeError exception thrown when trying to compare a CharsetMatch with anything else than a CharsetMatch.
-   Improved the general reliability of the detector based on user feedbacks. ([#520](jawah/charset_normalizer#520)) ([#509](jawah/charset_normalizer#509)) ([#498](jawah/charset_normalizer#498)) ([#407](jawah/charset_normalizer#407)) ([#537](jawah/charset_normalizer#537))
-   Declared charset in content (preemptive detection) not changed when converting to utf-8 bytes. ([#381](jawah/charset_normalizer#381))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
detection Related to the charset detection mechanism, chaos/mess/coherence
Projects
None yet
Development

No branches or pull requests

2 participants