[testing] Use bem/html-differ to perform tests

[Feder1co5oave]

I never liked the strategy of stripping all whitespace before comparing generated html code against the expected one. Taking into account whitespace when testing is important as it is often meaningful and leads to different results in the browser. For example, #1013 and #1011 fix issues that consist in different whitespace in the output, so even if tests are added they are not checked.

So, I thought we needed a more html-aware comparison algorithm that takes (meaningful) space into account. html-differ seemed to do the trick. It uses parse5 under the hood, which looks stable enough.

Also, the current text comparison algorithm is broken: whenever the expected html is shorter than the generated one, and matches everything up until its end, the test is marked as complete, even though marked outputted something more.

For more details about the proposed comparison algorithm, head over here. As an option, html-differ allows to ignore whitespace differences, html comments, and certain node attributes. I was thinking about taking advantage of this last feature to ignore headings' id attribute. Currently, header tags that do not have an id are "fixed" by the preprocessing phase of the test system, and ids are inserted into the expected html code automatically. This allows to test headings' ids when necessary, by specifying them in the .html file of the test case. (I used this in #456)

As an added benefit, when tests fail, we get a nice colored diff between the expected and the actual output, with space included, which wasn't before. As an example:

I took the opportunity to lint the test/index.js source file and remove unused variables and so on.

Since html-differ treats space and newlines differently (I'm not sure this is in accordance to Html/CSS specs, see my inquiry at bem/html-differ#150), in order to make the main test case pass, I had to replace some simple spaces with newlines, when newlines were to be expected.
The uppercase_hex test case was plain wrong as it lacked some space which should have been there but was not required by the current comparison algorithm.

As addendum, I myself am not enthusiastic about yet another npm package dependency pulled in at startup. Including parse5 and a bunch of other packages for mere testing may seem overkilling. But the fact is that testing is the most important development phase of marked, especially if our goal is to adhere to commonmark as much as possible (their current test suite includes 624 test cases) and I think it needs to be done correctly. Comparing two different html codes and determining if they are equivalent is no easy task. It involves parsing html and checking the AST according to some very verbose specification that we cannot take care of all by ourselves.
Stripping all whitespace was a quick&easy solution to allow some testing, now's the time to step away from it.

/cc to @UziTech who was the kind author of the recent upgrade to the test system and may know it very well.

[UziTech]

I like it. much easier to read. 💯

[KostyaTretyak]

I also did not like stripping all whitespace in HTML. But as for me it is more clearly visible and simpler - just create a html file that have diff between expected and actual. After that, simply point to, for example,

Expected in test/tests/def_blocks.html:3:1
Actual in test/tests/def_blocks-actual.html:3:1

My editor allows me (with the Ctrl key pressed) to go through these links exactly to the place where there is a difference. In our case, the link points to the third line and the first character. I can see very clearly where the differences begin, I do not have to further distinguish between color text etc...

[Feder1co5oave]

@KostyaTretyak so what would you do differently to test for whitespace?

[KostyaTretyak]

In my fork marked-ts I do not stripping whitespaces and do a check expected to actual on each line.

[joshbruce]

Right on! I'm going to flag this for 0.5.0; however, between this and linting - we may want to include both in the next release in general. Still catching up with the action over the last week. :)

[Feder1co5oave]

Sounds good to me, it is no priority, even though this should help us test correctly.

I'd also like to take a look at a simpler solution: just normalizing html code on both sides, if I find the right library to do it.

[joshbruce]

Think it might be worthwhile to find fallbacks to any libraries we include...just in case they end in the same position Marked was in before we all stepped back up, you know? Maybe a DEPENDENCIES.md file??

[joshbruce]

See #1024

[joshbruce]

Closing for now in favor of html-differ inclusion in #1160, merge conflicts, and possible future considerations related to developer operations and whatnot.