Python test suite is failing on 32 bit builds (MinGW32, Cygwin32, Linux)

[clanmills]

I suspect this is because the reference output reports 64bit integers.

======================================================================
FAIL: test_run (bugfixes.github.test_CVE_2017_14859.TestCvePoC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rmills/gnu/github/exiv2/exiv2/tests/system_tests.py", line 632, in test_run
    self.compare_stderr(i, command, processed_stderr, stderr)
  File "/home/rmills/gnu/github/exiv2/exiv2/tests/bugfixes/github/test_CVE_2017_14859.py", line 20, in compare_stderr
    self.assertIn(expected_stderr, got_stderr)
AssertionError: 'Exiv2 exception in print action for file /home/rmills/gnu/github/exiv2/exiv2/test/data/005-invalid-mem:\ncorrupted image metadata\n' not found in 'Warning: Directory Image, entry 0x011a has unknown Exif (TIFF) type 64772; setting type size 1.\nError: Upper boundary of data for directory Image, entry 0x011b is out of bounds: Offset = 0x00000030, size = 1073741832, exceeds buffer size by 1073734073 Bytes; truncating the entry\nError: Upper boundary of data for directory Photo, entry 0x9003 is out of bounds: Offset = 0x000001f8, size = 3538992, exceeds buffer size by 3531689 Bytes; truncating the entry\nWarning: Directory Nikon3 has an unexpected next pointer; ignored.\nExiv2 exception in print action for file /home/rmills/gnu/github/exiv2/exiv2/test/data/005-invalid-mem:\nArithmetic operation overflow\n'

======================================================================
FAIL: test_run (bugfixes.github.test_CVE_2017_14862.TestCvePoC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rmills/gnu/github/exiv2/exiv2/tests/system_tests.py", line 632, in test_run
    self.compare_stderr(i, command, processed_stderr, stderr)
  File "/home/rmills/gnu/github/exiv2/exiv2/tests/bugfixes/github/test_CVE_2017_14862.py", line 23, in compare_stderr
    self.assertIn(expected_stderr, got_stderr)
AssertionError: 'Exiv2 exception in print action for file /home/rmills/gnu/github/exiv2/exiv2/test/data/008-invalid-mem:\ncorrupted image metadata\n' not found in 'Warning: Directory Image, entry 0xff13 has unknown Exif (TIFF) type 65535; setting type size 1.\nError: Offset of directory Image, entry 0xff13 is out of bounds: Offset = 0x30303030; truncating the entry\nWarning: Directory Photo has an unexpected next pointer; ignored.\nError: Offset of directory Photo, entry 0x8827 is out of bounds: Offset = 0x30303030; truncating the entry\nError: Directory Photo, entry 0x9204 has invalid size 4286513153*8; skipping entry.\nWarning: Directory Nikon3 has an unexpected next pointer; ignored.\nError: Upper boundary of data for directory Nikon3, entry 0x0004 is out of bounds: Offset = 0x00000170, size = 1376264, exceeds buffer size by 1369403 Bytes; truncating the entry\nError: Offset of directory Nikon3, entry 0x0006 is out of bounds: Offset = 0x0000e803; truncating the entry\nError: Directory NikonPreview with 12336 entries considered invalid; not read.\nWarning: Directory Nikon3, entry 0x0095 has unknown Exif (TIFF) type 2562; setting type size 1.\nError: Offset of directory Nikon3, entry 0x009c is out of bounds: Offset = 0x000ffff8; truncating the entry\nExiv2 exception in print action for file /home/rmills/gnu/github/exiv2/exiv2/test/data/008-invalid-mem:\nArithmetic operation overflow\n'

======================================================================
FAIL: test_run (bugfixes.github.test_CVE_2017_14864.TestCvePoC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rmills/gnu/github/exiv2/exiv2/tests/system_tests.py", line 632, in test_run
    self.compare_stderr(i, command, processed_stderr, stderr)
  File "/home/rmills/gnu/github/exiv2/exiv2/tests/system_tests.py", line 755, in compare_stderr
    msg="Standard error does not match"
  File "/home/rmills/gnu/github/exiv2/exiv2/tests/system_tests.py", line 726, in _compare_output
    expected, got, msg=msg
AssertionError: 'Exiv[73 chars]test/data/02-Invalid-mem-def:\nArithmetic operation overflow\n' != 'Exiv[73 chars]test/data/02-Invalid-mem-def:\ncorrupted image metadata\n'
  Exiv2 exception in print action for file /home/rmills/gnu/github/exiv2/exiv2/test/data/02-Invalid-mem-def:
- Arithmetic operation overflow
+ corrupted image metadata
 : Standard error does not match

======================================================================
FAIL: test_run (bugfixes.github.test_issue_262.DivByZeroInPrintIFD)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rmills/gnu/github/exiv2/exiv2/tests/system_tests.py", line 632, in test_run
    self.compare_stderr(i, command, processed_stderr, stderr)
  File "/home/rmills/gnu/github/exiv2/exiv2/tests/system_tests.py", line 755, in compare_stderr
    msg="Standard error does not match"
  File "/home/rmills/gnu/github/exiv2/exiv2/tests/system_tests.py", line 726, in _compare_output
    expected, got, msg=msg
AssertionError: '' != 'Exiv2 exception in print action for file [93 chars]ta\n'
+ Exiv2 exception in print action for file /home/rmills/gnu/github/exiv2/exiv2/test/data/7-printIFD-divbyzero-1:
+ corrupted image metadata
 : Standard error does not match

----------------------------------------------------------------------
Ran 135 tests in 2.877s

FAILED (failures=4, skipped=5)
Makefile:115: recipe for target 'newtests' failed
make[5]: *** [newtests] Error 1
Makefile:102: recipe for target 'test' failed
make[4]: *** [test] Error 2
CMakeFiles/tests.dir/build.make:57: recipe for target 'CMakeFiles/tests' failed
make[3]: *** [CMakeFiles/tests] Error 2
CMakeFiles/Makefile2:136: recipe for target 'CMakeFiles/tests.dir/all' failed
make[2]: *** [CMakeFiles/tests.dir/all] Error 2
CMakeFiles/Makefile2:143: recipe for target 'CMakeFiles/tests.dir/rule' failed
make[1]: *** [CMakeFiles/tests.dir/rule] Error 2
Makefile:175: recipe for target 'tests' failed
make: *** [tests] Error 2
rmills@ubuntu:~/gnu/github/exiv2/exiv2/build$ 
rmills@ubuntu:~/gnu/github/exiv2/exiv2/build$ 
rmills@ubuntu:~/gnu/github/exiv2/exiv2/build$ 
rmills@ubuntu:~/gnu/github/exiv2/exiv2/build$ 
rmills@ubuntu:~/gnu/github/exiv2/exiv2/build$ 

[clanmills]

Making steady progress. Solved 1. 3 more to solve:

CVE	file	Ours	Expected	32 bit	Reason
don't know	-pX 7-*	#262	corrupted	SegFault	stringFormat()
14859	005*	#74	corrupted	Arithmetic operation overflow	Don't know yet
14862	008*	#75	corrupted	Arithmetic operation overflow	Don't know yet
14864	02*	#73	corrupted	Arithmetic operation overflow	Don't know yet

[clanmills]

I've been over #262 very carefully on 32 bit linux. I think there's a crashing issue in vsnprintf on 32 bit systems. I've written the following test program:

#include <iostream>
#include <string>
#include <cstring>
#include <vector>
#include <stdarg.h>
#include <stdio.h>
#include <assert.h>

#include <exiv2/exiv2.hpp>


std::string stringFormat(const char* format, ...)
{
    std::string result;
    std::vector<char> buffer;
    size_t need = std::strlen(format)*8;  // initial guess
    int rc = -1;

    // vsnprintf writes at most size (2nd parameter) bytes (including \0)
    //           returns the number of bytes required for the formatted string excluding \0
    // the following loop goes through:
    // one iteration (if 'need' was large enough for the for formatted string)
    // or two iterations (after the first call to vsnprintf we know the required length)
    do {
        buffer.resize(need + 1);
        va_list args;            // variable arg list
        va_start(args, format);  // args start after format
        rc = vsnprintf(&buffer[0], buffer.size(), format, args);
        va_end(args);            // free the args
        assert(rc >= 0);         // rc < 0 => we have made an error in the format string
        if (rc > 0)
            need = static_cast<size_t>(rc);
    } while (buffer.size() <= need);

    if (rc > 0)
        result = std::string(&buffer[0], need);
    return result;
}
        
const char* typeName1(uint16_t tag)
{
    //! List of TIFF image tags
    const char* result = NULL;
    switch (tag ) {
        case Exiv2::unsignedByte     : result = "BYTE"      ; break;
        case Exiv2::asciiString      : result = "ASCII"     ; break;
        case Exiv2::unsignedShort    : result = "SHORT"     ; break;
        case Exiv2::unsignedLong     : result = "LONG"      ; break;
        case Exiv2::unsignedRational : result = "RATIONAL"  ; break;
        case Exiv2::signedByte       : result = "SBYTE"     ; break;
        case Exiv2::undefined        : result = "UNDEFINED" ; break;
        case Exiv2::signedShort      : result = "SSHORT"    ; break;
        case Exiv2::signedLong       : result = "SLONG"     ; break;
        case Exiv2::signedRational   : result = "SRATIONAL" ; break;
        case Exiv2::tiffFloat        : result = "FLOAT"     ; break;
        case Exiv2::tiffDouble       : result = "DOUBLE"    ; break;
        case Exiv2::tiffIfd          : result = "IFD"       ; break;
        default                      : result = "unknown"   ; break;
    }
    return result;
}

int main(int /*argc*/, const char** /* argv */)
{
    const uint64_t address    = 54321     ;
    const uint16_t tag        = 0x1234    ;
    std::string    tagName    = "tag name you know";
    const char*    typeName   = typeName1(Exiv2::asciiString);
    uint32_t       count      = 999       ;
    const uint64_t offset     = 333444    ;
    bool           usePointer = true      ;
    std::string    offsetString(usePointer?stringFormat("%10u", (size_t)offset):"");

    std::cout  << stringFormat("%8u | "    ,address)
               << stringFormat("%#06x %-25s |" ,tag,tagName.c_str())
               << stringFormat("%10s |"        ,typeName)
               << stringFormat("%9u |"         ,count)
               << stringFormat("%10s | "       ,offsetString.c_str())
               << std::endl
               ;

    std::cout  << stringFormat("%8u | %#06x %-25s |%10s |%9u |%10s | ",
                    (size_t)address, tag, tagName.c_str(), typeName, count, offsetString.c_str())
               << std::endl
               ;
               
    std::cout  << stringFormat("STRUCTURE OF BIGTIFF FILE ") << std::endl;

    return 0;
}

It's odd. We don't need the (size_t) cast the first time. This issue has something to do with 64 bit integers on the stack going to va_start.

(gdb) run
Starting program: /home/rmills/temp/hello 
   54321 | 0x1234 tag name you know         |     ASCII |      999 |    333444 | 

Program received signal SIGSEGV, Segmentation fault.
0xb7d55163 in _IO_vfprintf_internal (s=s@entry=0xbfffed90, format=<optimized out>, format@entry=0x804a130 "%8u | %#06x %-25s |%10s |%9u |%10s | ", ap=0xbfffeee4 "\024\320\004\bW\240\004\b\347\003", 
    ap@entry=0xbfffeed8 <incomplete sequence \324>) at vfprintf.c:1661
1661    vfprintf.c: No such file or directory.
(gdb) backtrace
#0  0xb7d55163 in _IO_vfprintf_internal (s=s@entry=0xbfffed90, format=<optimized out>, format@entry=0x804a130 "%8u | %#06x %-25s |%10s |%9u |%10s | ", ap=0xbfffeee4 "\024\320\004\bW\240\004\b\347\003", 
    ap@entry=0xbfffeed8 <incomplete sequence \324>) at vfprintf.c:1661
#1  0xb7d74fa3 in _IO_vsnprintf (string=0x804d180 "   54321 | 000000 ", maxlen=<optimized out>, format=0x804a130 "%8u | %#06x %-25s |%10s |%9u |%10s | ", args=0xbfffeed8 <incomplete sequence \324>) at vsnprintf.c:119
#2  0x08048d10 in stringFormat(char const*, ...) ()
#3  0x08049155 in main ()
(gdb) quit

I've been over every call to stringFormat() and added (size_t) casts if there is any possibility of the 64bit integer being passed.

I still have to investigate #73 #74 and #75. I think we're throwing an arithmetic exception on a 32 bit processor and never reach our kerCorruptedMetadata. I'll find that tomorrow. Meanwhile, I've submitted a fix for #266 on 32 bit systems.

[clanmills]

I suspected those three issues were instances of the same fault on 32 bit systems. Here's the fix:

--- a/src/tiffvisitor_int.cpp
+++ b/src/tiffvisitor_int.cpp
@@ -1523,7 +1523,7 @@ namespace Exiv2 {
             if ((static_cast<uintptr_t>(baseOffset()) > std::numeric_limits<uintptr_t>::max() - static_cast<uintptr_t>(offset))
              || (static_cast<uintptr_t>(baseOffset() + offset) > std::numeric_limits<uintptr_t>::max() - reinterpret_cast<uintptr_t>(pData_)))
             {
-                throw Error(kerArithmeticOverflow);
+                throw Error(kerCorruptedMetadata); // #562 don't throw kerArithmeticOverflow
             }
             if (pData_ + static_cast<uintptr_t>(baseOffset()) + static_cast<uintptr_t>(offset) > pLast_) {
                 throw Error(kerCorruptedMetadata);

We're processing a fuzzed file and we detect a 32 bit overflow on a 32 bit system and throw kerArithmeticOverflow. 64 bit systems will throw kerCorruptedMetadata.

The test harness expects kerCorruptedMetadata . I could put a fix into the test harness, however it feels better to be consistent on all platforms.

Result: 32 and 64 bit systems behave the same.