Node#line heuristic under JRuby is hacky and should be smarter

[camertron]

With MRI 2.1.5, line numbers are correctly reported. With any recent version of JRuby, they aren't:

$> rbenv local ruby-2.1.5
$> bundle console
irb> require 'nokogiri'
irb> puts Gem.loaded_specs['nokogiri'].version
1.6.5
irb> doc = Nokogiri::XML('<root><foo>bar</foo>\n\n<foo>baz</foo></root>')
irb> root = doc.xpath('/root').first
irb> root.child.first.line
1
irb> root.child.last.line
3

And then:

$> rbenv local jruby-1.7.15
$> bundle console
irb> require 'nokogiri'
irb> puts Gem.loaded_specs['nokogiri'].version
1.6.5
irb> doc = Nokogiri::XML('<root><foo>bar</foo>\n\n<foo>baz</foo></root>')
irb> root = doc.xpath('/root').first
irb> root.child.first.line
1
irb> root.child.last.line
1

As you can see, under JRuby nokogiri incorrectly returns 1 for the line number of the second child under root.

[flavorjones]

Hello @camertron,

Thanks for reporting this! I've captured this in a slightly different repro script:

#! /usr/bin/env ruby

require 'nokogiri'

doc = Nokogiri::XML <<EOXML
<root>
  <foo>
    bar
  </foo>
  <foo>
    baz
  </foo>
</root>
EOXML
puts doc.to_xml
doc.css("foo").each { |node| puts node.line }

MRI outputs:

<?xml version="1.0"?>
<root>
  <foo>
    bar
  </foo>
  <foo>
    baz
  </foo>
</root>
2
5

JRuby outputs:

<?xml version="1.0"?>
<root>
  <foo>
    bar
  </foo>
  <foo>
    baz
  </foo>
</root>
2
4

As you can see, when compared to your script, JRuby appears to use a different heuristic when reporting the line number than MRI does. This is likely due to the underlying implementations being different XML libraries (libxml2 for MRI, and xerces for JRuby).

Indeed, looking at the code, in MRI we call a libxml2 method to get the internal line number from the parser:

https://github.com/sparklemotion/nokogiri/blob/master/ext/nokogiri/xml_node.c#L1247

And in JRuby, we look at a counter maintained in the JRuby extension (not the Xerces parser):

https://github.com/sparklemotion/nokogiri/blob/master/ext/java/nokogiri/XmlNode.java#L1430-L1452

Part of the issue here is that the w3c has not defined for what serialization a Node's "line number" might correlate:

http://www.w3.org/2003/01/dom2-javadoc/org/w3c/dom/Node.html

You can output an XML doc in a couple of different formats (e.g., ignoring whitespace, canonicalized, as-parsed), so which "line number" would we mean?

Libxml2 stores the line number for each node as-parsed:

http://xmlsoft.org/html/libxml-tree.html#xmlNode

that is, "on what line of the original document was this element encountered?". Xerces doesn't have similar functionality, and so you get the kludgey calculation that I linked to above -- it's a "best guess", and in most cases it's wrong, unfortunately.

I think what we might be able to do is implement startDocument() to store off the XMLLocator, then implement startElement() to query the element's line number from the locator and store it in a hash table for later use by the #line method. But I don't have a ton of experience with Xerces, so I might be totally off base, or there might be a more-clever solution.

@jvshahid or @yokolet -- does this strategy seem sound to you?

I'm also going to rename this issue to reflect the actual work that needs to be done.

[camertron]

Hey @flavorjones thanks for the awesome and thorough diagnosis! Glad to know the issue is getting some attention :) Let me know if I can be of any help.

[camertron]

Any update on this, @flavorjones?

[flavorjones]

Started a proof-of-concept branch at flavorjones-1223-jruby-line-numbers

[flavorjones]

Ok, in the interest of closing this out, I dug in today. The approach I started with is:

commit f435981 (origin/flavorjones-1223-jruby-line-numbers)
Author: Mike Dalessio <[email protected]>
Date:   2015-09-30 12:20:04 -0400

    WIP for properly tracking line numbers in JRuby.
    
    Related to #1223.

diff --git a/ext/java/nokogiri/internals/NokogiriDomParser.java b/ext/java/nokogiri/internals/NokogiriDomParser.java
index cefae5e..550ab93 100644
--- a/ext/java/nokogiri/internals/NokogiriDomParser.java
+++ b/ext/java/nokogiri/internals/NokogiriDomParser.java
@@ -39,6 +39,12 @@ import nokogiri.XmlDocument;
 import org.apache.xerces.parsers.DOMParser;
 import org.apache.xerces.parsers.XIncludeParserConfiguration;
 import org.apache.xerces.xni.parser.XMLParserConfiguration;
+import org.apache.xerces.xni.XMLLocator;
+import org.apache.xerces.xni.NamespaceContext;
+import org.apache.xerces.xni.Augmentations;
+import org.apache.xerces.xni.QName;
+import org.apache.xerces.xni.XMLAttributes;
+import org.apache.xerces.xni.Augmentations;
 import org.cyberneko.dtd.DTDConfiguration;
 import org.w3c.dom.Document;
 import org.xml.sax.Attributes;
@@ -57,6 +63,7 @@ public class NokogiriDomParser extends DOMParser {
     protected DOMParser dtd;
     protected boolean xInclude;
     protected XMLParserConfiguration config;
+    protected XMLLocator locator;
 
     public NokogiriDomParser(XMLParserConfiguration config) {
         super(config);
@@ -113,4 +120,21 @@ public class NokogiriDomParser extends DOMParser {
             return source;
         }
     }
+
+    @Override
+    public void startDocument(XMLLocator locator,
+                              String encoding,
+                              NamespaceContext namespaceContext,
+                              Augmentations augs) {
+        this.locator = locator;
+        super.startDocument(locator, encoding, namespaceContext, augs);
+    }
+
+    @Override
+    public void startElement(QName element,
+                             XMLAttributes attributes,
+                             Augmentations augs) {
+        System.out.println(element + ": " + locator.getLineNumber());
+        super.startElement(element, attributes, augs);
+    }
 }

but here I got stuck, because I don't know of any way to link the locator metadata with the final Node in the parsed Document. If someone can help me out, I'm all ears.

So I'm falling back on simply tweaking the existing algorithm to handle newlines better.

[flavorjones]

Closed by #2177.