Forum rules
An Activity Log is required for support requests. Please read How-to get an activity log? for details on how and why this should be provided.
When I'm adding subtitles to a video I usually use the ssa format. I set Handbrake to the MKV mode and use the preset Fast 1080p30. I use a Plex server to cast the file to a flat screen TV via a Roku app.
What would be the best preset to use, Roku 1080p30 surround, or Matroska H264 or H265 or the standard Fast 1080p30?
Steps to reproduce the problem (If Applicable):
Not Applicable.
HandBrake version (e.g., 1.0.0):
1.3.3
Operating system and version (e.g., Ubuntu 16.04 LTS, macOS 10.13 High Sierra, Windows 10 Creators Update):
Win 10 64 bit
That depends a lot on your Plex server hardware and settings.
If you're targeting a high-specification playback most of the time, you want to store it at best quality, and let Plex down-convert it as needed for other devices.
Given the choice of downloading the same video in different formats, what's the best choice for best quality? Should I choose file size? And after the subtitle is set to ssa, what shall I choose in Handbrake as a preset?
Now you're into subjective stuff, and I can only tell you what I would do... And I make my judgements differently than others. I can live with "DVD quality on a big screen" as long as what is being shown isn't boring or annoying. Interlace combing is annoying to me, so I chose Bluray sources when available. Boring is rather hard to fix, though.
My two standard presets (I don't use Plex) are customized versions of "HQ 1080p30 Surround" (modified for MKV output and preselect all audio/subtitle tracks in English and Japanese) and "Android 720p30" (modified to add AC3 audio and one subtitle track). My targets are similar to yours, but my 4K monitor isn't as capable as newer ones, and my other devices are not super-high-resolution.
I'm coding the same video with two different presets, the fast 1080 and the HQ1080, in the hope I will see some difference when they are served by Plex. One difference already is the time for the encoding, 2 -3times longer in HQ. I'll let you know what happens.
I've just compared the two versions on my TV and there is not a huge difference. When I make a DVD for a friend, using AVStoDVD, from here on in I'll use the HQ version. I've also changed the display type on ROKU from autodetect to 1080p. I wish there weren't so many settings!
If the source is a DVD, the "480" and "576" (NTSC vs PAL) presets include removing interlacing effects. I change these to disable comb detect, use Variable frame rate, and "Same as Source". It is slower to de-interlace ALL frames, but gives better results.
Locking the Roku to the monitor resolution also gets rid of a lot of annoying flashes as it switches monitor modes for each source.