-
Notifications
You must be signed in to change notification settings - Fork 160
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Need help evaluating "one weird trick" to improve AVBR ratecontrol #265
Comments
If it's something that would be recommended for a user (even if it's only an advanced user) to use (if you see X, do Y), then I would side with the clever option name. And documentation in the README (if you see X and you're OCD, use this option) and Wiki (here are some examples of X, what might be causing X, what effect Y should have, and how things can go wrong). |
Hi Don
Do you know why it didn’t work for Lock Stock, as that’s just important to
document?
Your postNoiseBoost option sounds to me like it just needs a article in a
wiki rather than a new option name from my view. There are already so many
options it’s tough to keep on top of what they all do and how they interact
with each other sometimes.
I tend to transcode a whole series at once and don’t get a proper chance to
play each file back to check it fully, so rely a lot on defaults and gut
feel - and this option seems like it’ll need a specific part of a movie to
tackle that you’re aware of before hand (or if you’re watching a movie and
notice the drop off in bit rate and make a not to re-transcode it again
later).
On Sun, 3 Mar 2019 at 04:55, vr8hub ***@***.***> wrote:
If it's something that would be recommended for a user (even if it's only
an advanced user) to use (if you see X, do Y), then I would side with the
clever option name. *And* documentation in the README (if you see X and
you're OCD, use this option) *and* Wiki (here are some examples of X,
what might be causing X, what effect Y should have, and how things can go
wrong).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#265 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAYJDUteZrsZVIFKGNLQ3T53ELgSPZzPks5vS1XMgaJpZM4baxhO>
.
--
Kind regards,
Dave
|
Completely agree with @vr8hub and @damorrison - make it a more memorable option name if you feel you must, but it really needs to be a part of the doco. I really like the notion of a section in the wiki being "If I see this -> This is what I should do to compensate -> here's a link to the relevant explanation of what I'm actually changing". In that way, we have the opportunity educate the user while fixing the immediate problem, if they are inclined to learn. |
Also, since our unofficial mission seems to be "finding the perfect balance between size, portability, and fidelity" we can approach the documentation from that point of view. Adding "and here is why this setting works in service of rule 1" to the relevant explanation to clarify the philosophical as well as the technical seems a good idea. |
@donmelton To me this feels just like the dpl2 conversation in #262; better most of the time but occasionally really drops the ball. In that issue the decision was leave the option in, but choose a safer default knowing that not everyone reads all of the documentation and better to default to the safe path. Re-reading your post I see that you aren't proposing it as default behaviour, I thought when you mentioned it was forthcoming in one of the other issues you were toying with making it either the default or the default Mostly I have the same question as @damorrison, why doesn't Lock Stock work, what is special about it that completely breaks? Any idea? I only have it on DVD, but I am curious to dust it off and see if the DVD has the same problem. OK, now I am going to the basement to dig out old DVDs... |
I agree with @klogg416, and have concerns about making a short option name for a setting that sounds good with a potential of producing misunderstood results. It would be easy to skim the documentation and get the wrong impression that the |
@donmelton @ttyS0
Edit: got it, triple code flag. |
OK, on the DVD copy adding @donmelton Assuming you are using a BD source, is that similar to what you are seeing? |
@vr8hub ...
Good way to put it. I'm not sure I would recommend using this trick to an advanced user. After all, AVBR looks great without it. There are just a few narrow cases where it could look better. So, agreed, this needs documentation either way. @damorrison ...
Good question! Yeah, I didn't explain that. Partially because I'm not sure exactly why it happens. But, agreed, that will need to documented as well.
You make a good point! Yeah, I'm really worried about confusing users with yet another option. @khaosx ...
Agreed, I should probably use the original text of this issue as the basis for that Wiki page. Perhaps even a section in the Wiki detailing workarounds for possible problems with each of my four ratecontrol systems? The good news is that it's possible to tweak The sad thing is that when you encounter a problem (and you will) with my special, or default, ratecontrol system, the best recourse is often just to use one of my other ratecontrol systems. That's why I developed the other three, because I couldn't workaround those problems I was seeing.
That's an excellent way to put it! Yeah, that's exactly the lens to view any recommendation we put in the Wiki. (That line is going in my Wiki meta issue that I'm working on now. :) ) @klogg416 ...
Actually, I don't think adding But yeah, better to default to the safe path.
Oh yeah, I would never make this the default!
I suspect it's due to "Lock, Stock and Two Smoking Barrels (1998)" being so noisy from high film grain and crappy disc mastering. So adding @ttyS0 ...
Agreed.
Yep! This is an excellent summary of why I fear adding any option at all for this. I mean, even I barely understand it. @klogg416 ...
While I appreciate the zaniness of something like
Interesting! OK, I haven't seen that before. Now I'll have try and find a DVD copy of it. And, yeah, this film has grain on top of its grain. It's like the poster child for grain. :)
Yeah, mine is the Blu-ray. And what I see when adding But it looks fantastic with just plain |
Interesting. I was looking at chapter 3 (09:55) because it encompasses a variety of lighting and motion changes, in a window at 200% zoom as I tried to find a balance of something big enough to look at without stretching 480p over a 4k screen. The messy grain/noise definitely persists in the background, while the "smoothing" effect is limited to flesh toned areas on the actors. In summary, DVD is too low resolution.
I hadn't picked up in your initial description that you didn't think |
@klogg416 Now I am really gonna have to find a DVD version to see what that looks like. But, yeah, the resolution is probably exacerbating other problems. You know, I had forgotten about that Of course, the future Wiki is the best place for this kind of thing. And I am typing on that as fast as I can. Really. :) |
@donmelton hey, was trying to find a good way to message you directly with this link, but twitter DMs require follow back, github doesn't seem to have a messaging feature, and suddenly I am over complicating it. This is, after all, expressly so you can see what I struggled to describe. https://www.dropbox.com/sh/v9kuj29n5fo9jxk/AADSwZ7_4CCsvMF0FqW4SjPGa?dl=0 That dropbox link has 3 two and a half minute clips of the encode I was talking about. File names are self explanatory.
|
@klogg416 Thanks! What's your Twitter handle and I can fix that in the future. :) But your filenames are confusing me. I don't understand which settings you used for which files. Can you spell it out for each of them to a knucklehead like me? |
@klogg416 BTW, you had an absolute path, i.e. starting at "/", when you created that .zip file. Thankfully, |
@klogg416 Now inspecting the downloaded files with MediaInfo and it doesn't appear that the two transcoded ones were created using a software encoder, i.e. x264 from HandBrake, since encoder settings are not embedded in the metadata. Are you sure you didn't accidentally use a hardware-based encoder? |
@klogg416 OK, I understand which file is which now based on the names. Sorry about that. |
@klogg416 OK, I made transcodings myself using |
@donmelton I sneak off to watch one little movie and I miss all this action! Twitter handle, @kloggnation Two lessons from this; I should have tested that dropbox link before assuming it would be easy, and I actually transcoded the entire DVD and then carved out only chapter 3 after the fact which seems to have had the side effect of purging some of the meta data. Comparing pre-and post in mediainfo I see why you are confused. SO! Why mess around, now that I have a raw copy of just chapter 3, I did it again. New files in dropbox, dated with today's date (added post transcode), and transcode details follow. Handbrake built from source on March 1st, 2019 so I have access to the FDK-AAC engine while running Windows. Version reports as There are 3 files (again). None of which were messed with after the fact except to add the dates to file names. Lock, Stock, and Two Smoking Barrels (1998) DVD ch3.mkv
Lock, Stock_avbr_ch3 2019-03-08.mkv
Lock, Stock_qcomp_ch3 2019-03-08.mkv
I didn't create any archive, just dropped in the files, that must be a DropBox feature. :-) |
@klogg416 Apologies if you already know this, but you can always add |
And I recognise that I built handbrake to be able to use the FDK-AAC and then did not specify it in these examples, this wasn't for keeping, just to see what you were describing in the original issue.. @khaosx Thanks man. I definitely use the Anyway, I backed my way into doing it badly instead of just redoing it the right way; redo with just the |
@donmelton sweet, good to know that it can get way more funky. I am going to skip the drama and take your good advice and just stick to the |
I do indeed dig it. Wise choice :)
And that's why I made it a team mission in my office that every Friday is "Read-Only" Friday! |
@khaosx I will suggest this to my boss for next week. I have no doubt that I will have much more free time to report back. :-) |
@klogg416 ...
And I sneak off to set up a replacement Roku and I miss all the new action! :)
Followed! (So you can DM me now.) But, Kyle, you've never tweeted anything!? Ever? Dude! :)
Thanks! I will look at those shortly.
Impressive that you build your own version! How do you think FDK-AAC compares to the new and improved AAC encoder in FFmpeg's LibAV? Also, enabling all the hardware-based encoders is a damn good idea!
OK, much clearer for a dope like me. :) Thanks! @khaosx ...
Yes, you can, but keep in mind that you won't get the same results if you transcode a portion from the middle of a video when using Those all use a variation of ABR ratecontrol which takes into account the duration of the transcode and happened previously. My default, or special, ratecontrol system and @klogg416 ...
No worries, I do that kind of thing all the time. It's actually better since it simplifies things.
Like I said, just be careful of side effects when using the
Whatever gets you to beer thirty faster, I always say. :)
Definitely stick with |
I knew I was going to get called out on that! Twitter isn't my cup of tea, but every now and again I need to try to get the @askatp guys to answer a question. Or monitor the iMore Black Friday deal tracking. We all have our crosses to bare. :-)
I humbly reject all praise. I have read for years that the default AAC encoder was crap in Windows, this was the accepted solution, I googled instructions and just rinse and repeat. I seized on it without empirical rigor and have no idea if it is actually better, the audio world taught me that elitism doesn't need proof, and besides, this whole game is an exercise in making something a little more difficult than it needs to be because the process is the fun part. Truth be told, I spend more time designing the solution than benefiting from it. My kid is watching a 4gig version of Toy Story and she isn't old enough to know that there is a plot. This is meta fun. I hadn't read that FFmpeg has a new encoder, thanks for the new reading assignment!
Huh, interesting, that explains the small file size variation. Makes sense, but hadn't considered it. When I use the
Beer thirty arrived... some time ago. Success! :-) |
@klogg416 ...
You are forgiven. :) (BTW, any other "usual suspects" (you know who you are) who are reading this and want me to follow you on Twitter so you can DM me, just let me know.)
Indeed. Making it harder is exactly why we do this. :) And your daughter has excellent taste! Even if she doesn't understand the hero's journey yet. :)
Yes, and it's much improved! The original AAC encoder in HandBrake for Windows and Linux, FAAC, was possibly even worse than the original FFmpeg encoder. And thank god Plex uses the new FFmpeg encoder as well now for dynamic re-transcoding. The new FFmpeg encoder is not as good as Apple's AAC encoder (the gold standard) but it's really hard to tell the difference at 160 Kbps, and pretty much impossible at 256 Kbps which is what Plex uses.
Indeed. Why wait?
Congrats! My wife tells me that Wine thirty will arrive any moment now with Instant Pot Korean-style beef. :) |
That is a savage step up, enjoy it!
This is pretty exciting news, I am definitely going to check it out tomorrow. Frankly it would be exciting to just use a package manager for handbrake and not have to remember to re-roll it. One less occasional own-goal.
I know everyone, myself included, starts their first issue responses with a standard and sincere “thank you”, but this really is a cool community and project that you have cultivated, and I am happy to have stumbled across it all those years ago. Thank you for being awesome @donmelton and for being the catalyst to make this hobby so engaging. And I suppose also for that fucking WestWorld podcast that made me do a victory lap through Seasons 1 and 2. :-) |
@klogg416 ...
It was very tasty, sir! Thank you!
This is exactly why I don't build my own HandBrake anymore. :)
You are all very welcome! I am so fortunate to have so many engaged, thoughtful and smart contributors and commenters here. It is both humbling and a great honor. I will try to live up to it, sir. In the end, I suppose my goal is to make you all as crazed about transcoding as I am. :) And thanks for listening to all the "Westworld" podcasts!! Kelly and I (because she's the boss and I do whatever she says) will be starting a re-watch of Season 2 sometime after the final season of "Game of Thrones." Because, right now, GoT is about all nerds want to talk about. :) |
Well shit, I guess I will clear my schedule. :-) |
@donmelton ...
I'm feeling brazen, so I'll consider myself a "usual suspect", and point at my Twitter handle => @ttys0.
Yay!! With any luck that will give me time to get caught up on my podcast backlog. I think I'm caught up to February now. 😭
I wholeheartedly agree, though my desktop's CPU probably wishes I'd never heard of |
@ttyS0 Sorry I took so long to reply but I was waylaid by evil companions with tasty wine... :)
You are a "usual suspect" so, followed! And I love your self-designation as "Recovering Sysadmin."
I'm probably years behind on my podcasting queue. At this point, I will never catch up. :)
I feel your pain. I've worn out at least one hard drive on my current iMac with this filthy habit. :) |
Need help evaluating "one weird trick" to improve AVBR ratecontrol
I need help evaluating an enhancement to my average variable bitrate (AVBR) ratecontrol system, i.e. the new
--avbr
option.Considering how much some of you really like AVBR and think it's on par with or even better than my special, or default, ratecontrol system, why does it even need enhancement?
Well, there is one situation where I think AVBR can sometimes choose too low of a bitrate to render a portion of video. This is after a prolonged spike in bitrate due to exceptionally noisy content.
To be fair to AVBR, this problem is not even perceptible most of the time to most users. You really have to be looking up close to see it. For me, my I need to be less than two feet away from my Retina 5K iMac. And it's not really an issue at all when I'm watching my 80-inch HDTV from across the room.
But I'm a perfectionist. :)
A good way to demonstrate the problem is by transcoding an episode of "Game of Thrones" or any other TV show prefaced with that annoying animated HBO "static" logo.
While AVBR will render the logo sequence perfectly, the scene immediately following it can seem... less than perfect. Mind you, it won't exhibit blockiness, color banding or artifacts but it might not look quite as sharp as the rest of the video.
And this lack of sharpness only lasts for a few seconds before everything is back to normal.
For a "Game of Thrones" episode, this isn't much of an issue because usually the logo is followed my the main title sequence and any reduction in bitrate is not apparent.
But for those episodes with a "cold open," this might be more noticeable. For example, the first episodes of seasons 1 and 7, "Winter Is Coming" and "Dragonstone."
You can also see this problem in the scene immediately following the digital dream sequence from chapter 3 of "Prometheus (2012)" that @samhutchins loves to test. Again, that noisy dream sequence is rendered perfectly. But afterwards things are not as sharp for a few seconds.
There might be other instances of this issue but I haven't been able to find any after transcoding over 700 videos using AVBR.
So, why does this happen at all?
AVBR is a variable bitrate system. That's what the "V" in the abbreviation indicates. :) But it's also an average bitrate system. And while variability allows it to render those extremely noisy scenes well, sticking to the average can cause the bitrate in quiet scenes following that extreme noise to dip below desired levels.
To be clear, the magic sauce that I already added to AVBR, disabling Macroblock-tree ratecontrol, significantly reduces wild swings in bitrates. But it might not always be enough as these examples I've described demonstrate.
Fortunately, I've discovered "one weird trick" which can fix this problem. :)
However, I can't just include this new trick in AVBR by default. :( Because there's one video I've found where applying it looks much worse. And there are probably others.
Here's how you can apply the "one weird trick" yourself:
So, what the Sheol is that
--encoder-option qcomp=0
voodoo? Simple. It's just manipulating the quantizer curve compression factor. :) Let me explain...Ordinarily the
qcomp
setting within the x264 encoder doesn't have much affect on output because Macroblock-tree ratecontrol is used instead ofqcomp
for weighting frame quality based on complexity. But because I've already disabled the MB-tree in AVBR,qcomp
has a big impact.The default value of
qcomp
is0.6
(or 60 percent) and its valid value range is between0
and1
(or 100 percent).Setting
qcomp
to0
effectively means that x264 shouldn't evaluate complexity to assess frame quality. This further reduces bitrate variability.And for those "Game of Thrones" episodes, everything looks great with
qcomp=0
. Both the noisy HBO logo and any video immediately after or even later. It's all good.And that dark scene in "Prometheus (2012)" after the noisy dream sequence looks perfect with
qcomp=0
. However, the dream sequence itself is doesn't looks quite as good. You can't have everything.What about that one video that looks much worse? Apply
qcomp=0
to the movie "Lock, Stock and Two Smoking Barrels (1998)" and... Jiminy! It really looks awful.And while
qcomp=0.4
(a setting of 40 percent) still looks OK with "Lock, Stock and Two Smoking Barrels (1998)," it's not enough of an improvement with the other cases to be worth including as a default in AVBR.So, why am I telling you all this?
Mostly so you can know about the trick yourselves if you want to use it.
But also I would like advice on whether I should wrap the
--encoder-option qcomp=0
voodoo in some clever (and much shorter) option name for others to use. Or should I just document this stuff in the "README" file or our eventually arriving Wiki?I would love to hear from @samhutchins, @elliotclowes, @khaosx and anyone else who commented on the original AVBR issue #248, as well as our usual gang of commenters like @JMoVS, @klogg416, @vr8hub, @damorrison, @rhapsodians and anyone else who has an opinion.
Thanks in advance!
The text was updated successfully, but these errors were encountered: