SVG I/O error: unsorted input lines detected

[rachtsingh]

Thank you for your help and quick response earlier! I had one more error I'm running into (besides the MKL error which I've raised a bug report for in threadpoolctl):

When I write the svg file, I get the following:

=fil-profile= Preparing to write to new-code/
=fil-profile= Wrote memory usage flamegraph to new-code/peak-memory.svg
=fil-profile= Error writing SVG: I/O error: unsorted input lines detected

Is that a known problem?

Also, if I can suggest something, I find the attempts to open the profiler output kind of annoying. I access my cluster using SSH (as presumably a lot of people doing memory profiling are doing) and it opens a bunch of odd things; I think at one point I saw an X session with Konqueror pop up, which is definitely not how I want to browse an SVG.

That being said, if you / others find that useful, it's totally fine. A CLI flag to disable that might be helpful, but I can definitely deal.

[itamarst]

Hi,

Sorry it's not working. Assuming you're using the latest version of Fil—

You should have a peak-memory-source.prof file in the output directory that I'd need you to upload. And there's two problems:

1. It gets deleted right now once processing is done. I will add a flag that makes it stick around for debugging purposes.
2. When you upload it I will be able to see some of your source code, which I can imagine is not a thing you want.

If the second part is a problem let me know, and I will try to come up with a debugging method that won't leak proprietary information.

Meanwhile, as a workaround: in the output folder there is a peak-memory.prof file. You can run it through either flamegraph.pl (https://github.com/brendangregg/FlameGraph) or inferno-flamegraph (https://github.com/jonhoo/inferno#as-a-binary) to get a flamegraph. Output won't be as nice (no source tracebacks, visually won't be as nice), but it will be something.

[rachtsingh]

Thanks for the response! You're correct, unfortunately I can't share the debugging output because of reason 2. So here's the thing: this error shows up but there's still a peak-memory.prof, peak-memory.svg and peak-memory-reversed.svg (as well as index.html) written to the folder. So unless you think the output is missing something somehow, I think the warning is extraneous?

In one case my job was killed while the output was being written, leaving an empty peak-memory.svg, though there was now a peak-memory-source.prof and peak-memory.prof file. I ran flamegraph.pl on the latter and got an output, which seemed mostly correct but possibly might have been missing a bunch of allocations (though it's unclear to me about this particular job).

For the latter scenario, would it be possible expose a python -m filprofile create_report <peak-memory-source.prof> -o output.svg -type interface? I was trying to hack one together but realized that the SVG output code seems to be in Rust, and I don't think I have the time to write a Rust CLI as much I'd love to help.

Thank you again!

[itamarst]

If you get the "unsorted" error, you get an SVG, but it'll be empty. So it's not good. Just to verify, what version of Fil are you using? Also, do you use emoji or unicode characters in your source code?

[rachtsingh]

No emojis or other unicode characters in source code; I just upgraded to the latest version that you released with the patch for my RHEL problem (thank you!).

I think I am getting non-empty SVGs with this, though it's totally possible that these SVGs are from the first dump, which doesn't have this problem (i.e. when I first run the program it runs correctly, then gives the warning afterwards).

[itamarst]

It's possible only the second SVG (the reversed one) is having the error, I've had more problems with it. When you open the HTML file, is the top one OK but the bottom one broken? Both broken?

[rachtsingh]

Top one is fine, bottom one is missing (also peak-memory-reversed.svg has size 0 bytes). Seems to confirm your suspicion. If there's a way to run this with verbose output, I can send you that after censoring some lines (sorry, debugging a tool that outputs stack traces is difficult in proprietary environments).

[itamarst]

The bottom one is the same information rearranged in a different way, so you should be able to just look at top one and do OK. But this is helpful in narrowing down the problem, I'll hopefully get some time to coding work on Fil today.

[itamarst]

OK have some theories now... Do you happen to use tabs in your source code?

[itamarst]

Maybe should leave some notes: my theory is that the trim() in the reversing code (https://github.com/jonhoo/inferno/blob/master/src/flamegraph/mod.rs#L412) results in https://github.com/jonhoo/inferno/blob/master/src/flamegraph/merge.rs#L127 skipping lines, thus resulting in weird unsorted results.

However, apriori one would expose source lines that cause allocations to either be completely empty (in which case https://github.com/pythonspeed/filprofiler/blob/master/memapi/src/memorytracking.rs#L175 should fix that), or to have some code in them, in which case they won't get skipped. So maybe not?

[itamarst]

Ah, don't need an empty line. Imagine a line with tabs at the end or beginning. Even if lines are sorted, the trimming of whitespace (https://github.com/jonhoo/inferno/blob/master/src/flamegraph/merge.rs#L126) can break the sort order.

Fil already deals with spaces to prevent this scenario, but other whitespace (e.g. tabs) won't be.

One solution would be to prefix and suffix each entry with a non-whitespace character, which then gets stripped in post-processing.

[rachtsingh]

Do you happen to use tabs in your source code?
I'm sure there is a file that uses tabs, and probably files that mix spaces and tabs. Unfortunately these are large codebases and I can't guarantee anything.

Your explanation makes sense to me. I'm not sure I can help propose a solution without reading more of the context, but it sounds like it'll work.

[itamarst]

I know how to fix this particular problem, so hopefully can get you some code to try this week.

[itamarst]

OK, new release should be up on PyPI within the hour, and on Conda sometime in the next few hours, or tomorrow at the latest.

Let me know if this fixed it for you!