check template UTF8 validity before parsing

[ggmichaelgo]

What are you trying to solve?

Liquid relies on regex for parsing, and Liquid needs to check the template's String encoding validity.

Currently, Liquid is raising a ArgumentError when a template includes an invalid UTF8 byte sequence.

require 'liquid'
Liquid::Template.parse("{% assign foo = '\xC0' %}") 

# lib/liquid/tokenizer.rb:31:in `split': invalid byte sequence in UTF-8 (ArgumentError)

Instead of throwing the ArgumentError, this PR updates Liquid to raise a Syntax error when a template has a invalid encoding.

With this change, the developers won't have to catch the invalid encoding error like this:

begin
  Liquid::Template.parse("\xC0")
rescue ArugmentError => e
  if e.message == "invalid byte sequence in UTF-8"
     ...
  else
    raise e
  end
end

Instead, the error can be handled like this

begin
  Liquid::Template.parse("\xC0")
rescue Liquid::Error
     ...
end

[samdoiron]

Will the regex always throw the UTF-8 error? If so, then this duplicates the validation work. What if we caught the exception and re-threw instead?

[ggmichaelgo]

This change is also for Liquid-C parser. Since Liquid-C doesn't use regex, it allowed Liquid templates like this:

{% assign foo = "Broken String\x00" %}
{% comment \xC0 %}
  \xC0
{% endcomment \xC0 %}

With this change, we won't have to implement UTF-8 validation check on Liquid-C.

Also, the UTF-8 error would have to be handled by checking error message, and I think it is more cleaner to check the validity by using String#valid_encoding?

rescue ArgumentError => e
  if e.message == "invalid byte sequence in UTF-8"