[bug] v1.11.0 native Windows gem causes nokogumbo installation to fail

[flavorjones]

Please describe the bug

Nokogumbo cannot build against the Nokogiri v1.11.0 native Windows gem.

It is able to build against

• the v1.10.10 native Windows gem
• the v1.11.0 native Linux and Darwin gems.

Help us reproduce what you're seeing

See the Nokogumbo test suite's failing windows jobs when built against v1.11.0:

MAKE='make -j2 V=1' bundle exec rake -- --with-libxml2
mkdir -p tmp/x64-mingw32/nokogumbo/2.7.2
cd tmp/x64-mingw32/nokogumbo/2.7.2
C:/hostedtoolcache/windows/Ruby/2.7.2/x64/bin/ruby.exe -I. ../../../../ext/nokogumbo/extconf.rb -- --with-libxml2
checking for whether -ID:/a/nokogumbo/nokogumbo/vendor/bundle/ruby/2.7.0/gems/nokogiri-1.11.0-x64-mingw32/ext/nokogiri is accepted as CFLAGS... yes
checking for whether -ID:/a/nokogumbo/nokogumbo/vendor/bundle/ruby/2.7.0/gems/nokogiri-1.11.0-x64-mingw32/ext/nokogiri/include is accepted as CFLAGS... yes
checking for whether -ID:/a/nokogumbo/nokogumbo/vendor/bundle/ruby/2.7.0/gems/nokogiri-1.11.0-x64-mingw32/ext/nokogiri/include/libxml2 is accepted as CFLAGS... yes
checking for libxml/tree.h... yes
checking for nokogiri.h... yes
creating Makefile
cd -
cd tmp/x64-mingw32/nokogumbo/2.7.2
make -j2 V=1
(echo EXPORTS && echo Init_nokogumbo) > nokogumbo-x64-mingw32.def
...
x86_64-w64-mingw32-gcc -shared -o nokogumbo.so nokogumbo.o ascii.o attribute.o char_ref.o error.o foreign_attrs.o parser.o string_buffer.o string_piece.o svg_attrs.o svg_tags.o tag.o tag_lookup.o tokenizer.o token_buffer.o utf8.o util.o vector.o -L. -L/C/hostedtoolcache/windows/Ruby/2.7.2/x64/lib -L. -pipe -s -fstack-protector-strong -pipe -s -fstack-protector-strong -Wl,--enable-auto-image-base,--enable-auto-import nokogumbo-x64-mingw32.def    -lx64-msvcrt-ruby270  -lshell32 -lws2_32 -liphlpapi -limagehlp -lshlwapi  
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x34): undefined reference to `xmlFreeDoc'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x10a8): undefined reference to `xmlNewDocText'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x10c4): undefined reference to `xmlAddChild'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x10dd): undefined reference to `xmlNewDocNode'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x111e): undefined reference to `xmlAddChild'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x1153): undefined reference to `xmlNewNsProp'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x1191): undefined reference to `xmlSearchNs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x11af): undefined reference to `xmlNewNs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x11c8): undefined reference to `xmlNewDocComment'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x120d): undefined reference to `xmlNewCDataBlock'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x123e): undefined reference to `xmlSearchNs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x1260): undefined reference to `xmlNewNs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x127e): undefined reference to `xmlSearchNs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x12a0): undefined reference to `xmlNewNs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x12e6): undefined reference to `xmlSearchNs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x12f6): undefined reference to `xmlSetNs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x130e): undefined reference to `xmlSearchNs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x132c): undefined reference to `xmlNewNs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x1352): undefined reference to `xmlNewNs'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x1440): undefined reference to `htmlNewDocNoDtD'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x1462): undefined reference to `xmlCreateIntSubset'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x1484): undefined reference to `Nokogiri_wrap_xml_document'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.text+0x14c5): undefined reference to `htmlNewDocNoDtD'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.rdata$.refptr.mNokogiri[.refptr.mNokogiri]+0x0): undefined reference to `mNokogiri'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: nokogumbo.o:nokogumbo.c:(.rdata$.refptr.cNokogiriXmlSyntaxError[.refptr.cNokogiriXmlSyntaxError]+0x0): undefined reference to `cNokogiriXmlSyntaxError'
collect2.exe: error: ld returned 1 exit status
make: *** [Makefile:261: nokogumbo.so] Error 1

And the same suite's passing status when built against v1.10.10.

Expected behavior

Nokogumbo should be able to build identically against v1.11.0 as it does against v1.10.10.

Environment

From Nokogumbo's ci.yml config:

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu, macos, windows]
        ruby: [2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7]
        system_libraries: [true, false]
        exclude:
        - os: macos
          ruby: 2.1
        - os: macos
          ruby: 2.2
        - os: macos
          ruby: 2.3
        - os: windows
          ruby: 2.1
        - os: windows
          ruby: 2.2
        - os: windows
          ruby: 2.3
    runs-on: ${{ matrix.os }}-latest

    steps:
    - uses: actions/checkout@v2
    - name: Set up Ruby
      uses: ruby/setup-ruby@v1
      with:
        ruby-version: ${{ matrix.ruby }}
        bundler-cache: true
        
    - name: install html5lib tests
      run: git clone --depth 1 --branch all-error-fixes --single-branch https://github.com/stevecheckoway/html5lib-tests.git test/html5lib-tests

    - name: Run tests
      env:
        USE_SYSTEM_LIBRARIES: ${{ matrix.system_libraries }}
      shell: bash
      run: ./scripts/ci-test.sh

Additional context

Note that Linux and Darwin native gems aren't a problem.

[flavorjones]

@larskanis I'm not sure if you have any ideas here. I probably won't have time to look into it for another day or two.

[flavorjones]

Goal for today is to set up a Windows virtual machine so I can bisect when this started happening.

[larskanis]

From first reading I would say that the linker command is missing -lnokogiri and possibly libxml and libxslt. Windows doesn't allow missing references at build time, while Linux and Macos do.

[flavorjones]

I think this is related to rubys/nokogumbo#165 and might be fixed by rubys/nokogumbo#166. We'll see if the Windows builds are all green and I'll come back here with more info.

[flavorjones]

Still failing. I have a Windows dev env set up now, will investigate this weekend.

[flavorjones]

@larskanis I think I need help with this. I can easily reproduce on a Windows dev environment running:

• Ruby 3.0 and devkit (via RubyInstaller)
• Nokogiri 1.11.1 x64-mingw32
• Nokogumbo source at https://github.com/flavorjones/nokogumbo/tree/flavorjones-add-support-for-precompiled-native-nokogiri-gems

Output matches what's seen at https://github.com/rubys/nokogumbo/pull/163/checks?check_run_id=1990881540

I've tried adding (in nokogumbo's extconf.rb) ldflags of:

• -LC:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/nokogiri-1.11.1-x64-mingw32/ext/nokogiri/3.0
• and -l:nokogiri.so

and the mkmf.log output indicates that the library is found; but I still get these unresolved references.

Anything else you can think of to try?

[larskanis]

No idea for now. I'll try to dig deeper tomorrow.

[flavorjones]

OK, I think I've got this figured out. It's a combination of:

• libxml2 and nokogiri not properly exporting symbols (note that libxslt/libexslt do this properly, though!)
• ensuring that nokogumbo has linker flags to link against the installed nokogiri.so

The first can be solved by passing -DIN_LIBXML -ULIBXML_STATIC as cflags in the libxml2 recipe (in Nokogiri's extconf.rb) and by properly exporting things in nokogiri.h.

The second needs some more work to figure out, but for now the combination of -LC:/path/to/installation/dir/lib/nokogiri/3.0 and -l:nokogiri.so seems to be working.