An obvious tip which might be easily overlooked when approaching this for the first time is to use a short sample for testing.
Instead of trialling these options with your final 120+min render, just create a 5min or so sample file of your project. This will allow you to trial various Handbrake settings quickly, getting various encodings which you can directly compare for processing time, visual quality, and file size.
Handbrake allows you to queue things so you can just create a stack of encodings with different settings (eg CQ 18,19,20,21 etc or Speed setting), then it will run through the queue and in a few hours you can have many sample files to look at. Just make sure that you title each file appropriately so you can easily see what settings were used for it (eg "SampleFile_h265_Slower_CQ20.mp4". Look in 'settings/preferences' for auto-naming options). The queue window gives details of settings used and lists the encoding time, so you don't have to sit there with a stopwatch observing it. All the data is listed there. Just be aware that the queue will be cleared when quitting Handbrake so you'll lose the list. (I don't know if Handbrake keeps a permanent text file log which you can look up any time.)
This will avoid the problem had by someone such as Prism Skywalker who seems to be jumping in with both feet and then saying "omg I can't spend 12 hours on this". Everyone's computer setup and project specs will differ, so there's no absolute guide to how long it will take. Using a small sample which is visually representative of the typical kind of imagery/detail in your project will allow you to get a good idea of what settings to use.
It's probably also worth getting your head round the difference between h264 and h265. For the same visual quality, h264 will give faster encoding times but a larger filesize. h265 is a more efficient way of encoding, meaning you can get the same visual quality in a smaller filesize, but it takes longer to encode. In my experience/testing, using slower speed settings with h265 can dramatically increase encoding times as compared to using slower speeds for h264.
For ease, make your sample file length a round number which you can easily multiply to accurately estimate the ultimate per hour/project encoding time and filesize. For example, a 2min, 5min, or 10min sample can be easily multiplied by 30, 12, or 6 respectively to give you a good idea of the encoding time and filesize for 1 hour of video.
All the basic settings (speed, bitrate, CQ/RF 264/265 etc) make a significant difference in the three things you care about: picture quality, processing time, and file size. But what you'll find is that for all settings there comes a point where increasing the bitrate, or slowing the encoding speed etc just gives diminishing returns.
To this point, I'll urge caution regarding one thing Tremault said about encoding speed: "ideally want to make this 'encoder preset' go as slow as you can bear." I've done extensive testing across three variables (264/265, CQ number, Encoder Speed) to compare the numbers and found encoding speed to be the biggest potential pitfall, in practical terms. What I mean is that when adjusting the Encoder Speed, there really is such a thing as too slow. Using a mid-slow speed gives longer encoding times for some gain in quality, but when you get to the slowest speeds (particularly with h265) it can drastically increase encoding times (like 10-20x baseline) while giving marginal differences in resulting filesize/quality. Rule of thumb for me is that it's more beneficial to use a higher quality setting (lower CQ number) with a mid-slow speed, compared to a lower quality setting with a very-slow speed. But that's just me on my rig (M1 Mac Mini). It will vary for others which is why I'd recommend everyone doing some of their own testing to see what works for them.
forums.fanedit.org
