timecode problem with FCP XML converter

[msheby]

Bug Report

Incorrect Functionality and General Questions

We used otioconvert on a FCP file with content described as:

<rate>
  <timebase>24</timebase>
  <ntsc>TRUE</ntsc>
</rate>

and got an output OTIO file with a bunch of

"rate": 23.976023976023979

entries. So far so good. However, otiostat is unhappy with the resulting file:

parsed: True
top level object: Timeline.1
number of tracks: 3
Tracks are the same length: True
deepest nesting: 4
number of clips: 6
total duration: RationalTime(33820, 23.976)
There was a system error: invalid timecode rate
top level rate: 23.97602397602398
clips with cdl data: 0
Tracks with non standard types: 0

To Reproduce

1. CentOS Linux release 7.2.1511 (Core)
2. Python 3.6.8
3. n/a
4. OpenTimelineIO 0.13.0 (via pip install)

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-initfini-array --disable-libgcj --with-isl=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/isl-install --with-cloog=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/cloog-install --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) 

Expected Behavior

Well, I would expect that otiostat would be happy with otioconvert's output. That said, I don't know if otioconvert emits an incorrect decimal place, if otiostat needs to change its _top_level_rate behavior (note both the 23.976 and 23.97602397602398 strings in its output), or even if there need to be additional entries in the valid_timecode_rates array.

Screenshots

n/a

Logs

n/a

Additional Context

Interestingly enough, I get this on the same machine:

Python 3.6.8 (default, Apr  2 2020, 13:34:55) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> float(24) * (1000.0 / 1001)
23.976023976023978
>>> str(float(24) * (1000.0 / 1001))
'23.976023976023978'

As such, I'm not sure why otioconvert writes a least significant digit of 9 instead of 8.

[cds1503]

It doesn't seem to be a simple NTSC eqauls True problem.
I also have the same issue. The problem occurs when there is an Effect:Time Remap.

[reinecke]

I think we'd need an example FCP xml file to validate what's happening with the time warp, but I do have evidence of the wrong start offset coming through when using NTSC rates.
For example, I have a clipitem with these fields (only relevant timing fields included):

<duration>767</duration>
<rate>
    <ntsc>TRUE</ntsc>
    <timebase>24</timebase>
</rate>
<in>447</in>
<out>477</out>

And the file element under that clipitem has these fields (only relevant timing fields included):

<rate>
    <timebase>24</timebase>
    <ntsc>TRUE</ntsc>
</rate>
<duration>767</duration>
<timecode>
    <rate>
        <timebase>24</timebase>
    </rate>
    <string>14:11:44:09</string>
    <frame>1226505</frame>
</timecode>

source_range on that clip is resolving to:

"source_range": {
  "OTIO_SCHEMA": "TimeRange.1",
  "duration": {
    "OTIO_SCHEMA": "RationalTime.1",
    "rate": 23.976023976023978,
    "value": 30
  },
  "start_time": {
    "OTIO_SCHEMA": "RationalTime.1",
    "rate": 23.976023976023978,
    "value": 1225726.7202797204
  }
}

And, in the FCP 7 UI the timecode range for the clip is: 14:12:03:00 - 14:12:04:05

So, it would appear that perhaps what's happening is that the match to calculate source range is coming out to:

clip_relative_start_time = RationalTime(clipitem.in, (clipitem.rate.timebase * (1000/1001 if clipitem.rate.ntsc else 1)))
clip_start_offset = RationalTime(file.timecode.frame, (file.timecode.rate.timebase * (1000/1001 if file.timecode.rate.ntsc else 1)))
clip_start_time = (clip_start_offset + clip_relative_start_time)

Since the timecode of 23.976 is counted as 24 (the timecode time drifts from "wall clock" time), this is adding the timecode time to wall clock time. I believe the correct thing to do is use file.timecode.frame with the file.rate fields to generate the start offset.

If we generate the start time using:
(opentime.RationalTime(447, 24000/1001) + opentime.RationalTime(1226505, 24000/1001))

Then start_time.to_timecode(24000/1001) resolves to the correct 14:12:03:00

[reinecke]

One other interesting observation:
I took the same EDL I got the fields above from, imported to premiere, then re-exported to FCP XML. The exported version includes the <ntsc>TRUE</ntsc> element in the timecode rate.

@msheby Which app did your original XML file come from? FCP 7 or Premiere?

[reinecke]

@msheby If you get a chance, could you pull down #843 and see if your issue is addressed?

[meshula]

Root cause has been possibly identified by @msheby as stemming from exact frame rate tests against inaccurate values here: https://github.com/PixarAnimationStudios/OpenTimelineIO/blob/466a9b51aeaab2aae20b46620eff6ec49c9106ba/src/opentime/rationalTime.cpp#L41

Proposed fix is one of: add an epsilon to the function signature, or to change the comparison loop such that the result returns true if the supplied value is no worse than the values in the table, or to hard code a tolerance of 1ms from an exact value.

[jminor]

I think the intent behind those similar, but numerically wrong, rates (e.g. 29.97 vs 30000/1001) was to identify rates coming from "pragmatic" workflows where users had typed the convenient truncated rate, and to coerce them into the correct NTSC rates. Maybe that should be factored out into separate functions with clear intent, like coerce_fuzzy_rate_into_commonly_used_rate(...)

The code as written seems to be using those lists of "valid" rates just when formatting or parsing timecode strings, which should be the only place in OTIO's base library where NTSC matters. The only other place would be in some of the adapters where timecode is used. The core of OTIO was designed to not care what the rate is, as long as it is positive. In this case, otiostat's output doesn't seem to be honoring that intent.

[meshula]

Ah, so the issue @msheby references here #190 (comment) is another rate-rounding issue, not this one per se.

[vade]

Hi friends.

Im having a similar issue in OTIO 0.16-dev (im running off of main / master) - and am looking for pragmatic solutions to getting an OTIO -> FCP7 XML for Premiere Pro workflows in the wild.

Realistically, not all video used in our system will adhere to video broadcast standards and rates, as our software can ingest arbitrary sources.

Our workflow is to export an OTIO file from our in progress AFVoundation OTIO wrapper, and then leverage otioconvert via the python embedded environment to export an XML.

Im unaware of adaptor flags or work arounds - im curious if there's any workarounds.