Ai upscaling - 1080p to 8k to 1080p

General questions or discussion about HandBrake, Video and/or audio transcoding, trends etc.
Post Reply
YaBoyShredderson
Posts: 48
Joined: Fri May 29, 2020 5:17 pm

Ai upscaling - 1080p to 8k to 1080p

Post by YaBoyShredderson »

Anyone experimented with ai upscaling? Using topaz video enhancer or some other similar technology to convert a 1080p bluray up to 8k, to be reencoded back to 1080p? This kind of upscaling seems quite promising from i have seen. Mabe even keeping a higher res like 4k?
rollin_eng
Veteran User
Posts: 4840
Joined: Wed May 04, 2011 11:06 pm

Re: Ai upscaling - 1080p to 8k to 1080p

Post by rollin_eng »

Why would you want to upscale 1080 to 8k, then convert to 1080?
Deleted User 13735

Re: Ai upscaling - 1080p to 8k to 1080p

Post by Deleted User 13735 »

Tanstaafl
YaBoyShredderson
Posts: 48
Joined: Fri May 29, 2020 5:17 pm

Re: Ai upscaling - 1080p to 8k to 1080p

Post by YaBoyShredderson »

rollin_eng wrote: Sun Aug 23, 2020 3:51 pm Why would you want to upscale 1080 to 8k, then convert to 1080?
Upscale to 8k to improve quality, downsample to 1080p to save space. Downsampled content looks better than native content. So like 4k video, downsampled to 1080p will look better than if it was shot natively at 1080p. Obviously its not going to retain the same sharpness as if it stayed at 4/8k but it should, at least in theory look better than before.
rollin_eng
Veteran User
Posts: 4840
Joined: Wed May 04, 2011 11:06 pm

Re: Ai upscaling - 1080p to 8k to 1080p

Post by rollin_eng »

Upscaling wont improve quality.
YaBoyShredderson
Posts: 48
Joined: Fri May 29, 2020 5:17 pm

Re: Ai upscaling - 1080p to 8k to 1080p

Post by YaBoyShredderson »

rollin_eng wrote: Sun Aug 23, 2020 5:30 pm Upscaling wont improve quality.
From what ive seen topaz video enhancer ai works pretty well, it takes a while, sometimes a few seconds for each frame, but they do look very good. Its not the kind of upscaling your tv does, where its just mapping a lower res image to more pixels. I dunno what it does but it really does work well.
rollin_eng
Veteran User
Posts: 4840
Joined: Wed May 04, 2011 11:06 pm

Re: Ai upscaling - 1080p to 8k to 1080p

Post by rollin_eng »

If thats the case just buy the DVD and upscale to 32k :)
Deleted User 13735

Re: Ai upscaling - 1080p to 8k to 1080p

Post by Deleted User 13735 »

Upscale to 8k to improve quality,
Wishful thinking, otherwise called "alchemy."
Deleted User 11865

Re: Ai upscaling - 1080p to 8k to 1080p

Post by Deleted User 11865 »

I understand the idea of "enhanced" upscaling for playback on a high-resolution display, but upscaling then downscaling sounds a lot like just applying a sharpening filter. I wonder whether the results would be any better than using a dedicated sharpening filter; I suspect not consistently so, but maybe sometimes, who knows? The optimal amount of sharpening will also depend on the size of your display and viewing distance, so not something I'd recommend applying generically to all your encodes.
Woodstock
Veteran User
Posts: 4614
Joined: Tue Aug 27, 2013 6:39 am

Re: Ai upscaling - 1080p to 8k to 1080p

Post by Woodstock »

Despite what you see in crime dramas, you really cannot get better quality by upscaling. You can make educated guesses as to what a single pixel converted to 16 might look like, but it is just a guess. It is influenced not by what was really recorded, but what the algorithms used can create that "mostly doesn't disagree" with the original.

An interesting experiment would be to take an actual 8K source, shrink it to 1080p, then compare an AI-upscaled-to-8k picture looks like. But, you will probably find that whatever quality the original 8K image had, it's suffered badly from the journey.
YaBoyShredderson
Posts: 48
Joined: Fri May 29, 2020 5:17 pm

Re: Ai upscaling - 1080p to 8k to 1080p

Post by YaBoyShredderson »

Woodstock wrote: Sun Aug 23, 2020 6:54 pm Despite what you see in crime dramas, you really cannot get better quality by upscaling. You can make educated guesses as to what a single pixel converted to 16 might look like, but it is just a guess. It is influenced not by what was really recorded, but what the algorithms used can create that "mostly doesn't disagree" with the original.

An interesting experiment would be to take an actual 8K source, shrink it to 1080p, then compare an AI-upscaled-to-8k picture looks like. But, you will probably find that whatever quality the original 8K image had, it's suffered badly from the journey.
No doubt an upscaled image is worse than an image at that res natively, but the program i used works very well. Its not a simple upscaler to map pixels to larger display, but actually "adds detail". Obviously it cannot "enhance" the image and extract true detail that isnt present in the image in the first place, but its pretty good at taking, for example, a few darkened pixels that are supposed to be a pore, and actually making look correct.
YaBoyShredderson
Posts: 48
Joined: Fri May 29, 2020 5:17 pm

Re: Ai upscaling - 1080p to 8k to 1080p

Post by YaBoyShredderson »

Rodeo wrote: Sun Aug 23, 2020 6:41 pm I understand the idea of "enhanced" upscaling for playback on a high-resolution display, but upscaling then downscaling sounds a lot like just applying a sharpening filter. I wonder whether the results would be any better than using a dedicated sharpening filter; I suspect not consistently so, but maybe sometimes, who knows? The optimal amount of sharpening will also depend on the size of your display and viewing distance, so not something I'd recommend applying generically to all your encodes.
That was kind of what i was getting at. Not sure tbh. I did try and the image does look noticably sharper, and i cant see any artifacts that are associates with sharpening. So maybe its a good way to go about it.
rollin_eng
Veteran User
Posts: 4840
Joined: Wed May 04, 2011 11:06 pm

Re: Ai upscaling - 1080p to 8k to 1080p

Post by rollin_eng »

Woodstock wrote: Sun Aug 23, 2020 6:54 pm An interesting experiment would be to take an actual 8K source, shrink it to 1080p, then compare an AI-upscaled-to-8k picture looks like. But, you will probably find that whatever quality the original 8K image had, it's suffered badly from the journey.
If anyone wants to try this...

https://images.nasa.gov/details-First-8 ... Space.html
oRBIT2002
Experienced
Posts: 90
Joined: Sun Nov 01, 2009 11:22 am

Re: Ai upscaling - 1080p to 8k to 1080p

Post by oRBIT2002 »

I've experimented a bit with "TopazLabs Video Enhance AI". It's pretty promising but I think it works best on animated/CGI material at the moment. I upscaled one of my animated DVD's to 1080p for a test and it looks really good. It has pretty good sharpening/noise-removing-stuff implemented that works very well for animations.
Rorchach91
Posts: 17
Joined: Fri Aug 14, 2020 4:13 am

Re: Ai upscaling - 1080p to 8k to 1080p

Post by Rorchach91 »

rollin_eng wrote: Sun Aug 23, 2020 5:30 pm Upscaling wont improve quality.
I've heard the AI upscaling do enhance the graphic quality, it using a certain algorithm to blur the pixel and add noise to the graphic, perhaps do dithering too. No doubt that the AI upscaled video does no longer represent its original object which been filmed, it is somehow distorted, but make the graphic looks pretty & enhanced for sure.
mduell
Veteran User
Posts: 8187
Joined: Sat Apr 21, 2007 8:54 pm

Re: Ai upscaling - 1080p to 8k to 1080p

Post by mduell »

Much like the x264 psychovisual optimizations, you can do things that look better subjectively, but objectively have reduced the quality of the video.
Deleted User 13735

Re: Ai upscaling - 1080p to 8k to 1080p

Post by Deleted User 13735 »

TopazSmall.jpg
TopazSmall.jpg (873.82 KiB) Viewed 1094 times
Test Setup: I took a 2712x2400 original, reduced it to 50% (no sharpening), and then upscaled to original size using Bicubic, Lanczos 4, and Topaz AI.

Full Size Image for download here:
https://drive.google.com/open?id=1kdUk- ... Gw458VQvD4
Deleted User 13735

Re: Ai upscaling - 1080p to 8k to 1080p

Post by Deleted User 13735 »

TopazDiffSmall.jpg
TopazDiffSmall.jpg (725.56 KiB) Viewed 1088 times
Now that you're as impressed as I was by the sharpening, Here are the enhanced Difference Composites. You can see that Topaz is definitely an enhancement, and not an accurate representation.
forart.eu
Posts: 17
Joined: Tue Feb 27, 2007 4:05 pm

Re: Ai upscaling - 1080p to 8k to 1080p

Post by forart.eu »

Hi everyone, dunno if can help in any way, but I collected some open source resources about "AI upscaling":
- Video2X: Machine learning video/GIF/image upscaling
- Waifu2x-Extension-GUI: Image, GIF and Video enlarger/upscaler(super-resolution[/url] achieved with Waifu2x, SRMD, RealSR, Anime4K and ACNet.
- Video Super Resolution: A collection of state-of-the-art video or single-image super-resolution architectures, reimplemented in tensorflow.
- AviSynth AiUpscale: An AviSynth+ implementation of some Super-Resolution Convolutional Neural Networks.
- Fast-SRGAN: enable real time super resolution for upsampling low resolution videos.
- Anime4K: a set of open-source, high-quality real-time anime upscaling/denoising algorithms that can be implemented in any programming language.
- Zooming-Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution
- Video-Super-Resolution list: collection of papers

Of course I also opened a "feature request" @ Intel Media SDK's github...

Hope that inspires !
coffeelover
Posts: 2
Joined: Wed Jan 20, 2021 8:23 am

Re: Ai upscaling - 1080p to 8k to 1080p

Post by coffeelover »

Have you tried just using one of the AI models at 100% (Denoise/DeBlock).

I did a small test using the Artemis HQv9 model and it did a great job of maintaining the fine details while stripping the noise from my source material.

My only issue was it took hours to process a short scene on my system so it's not worth my time to do a full movie.
forart.eu
Posts: 17
Joined: Tue Feb 27, 2007 4:05 pm

Re: Ai upscaling - 1080p to 8k to 1080p

Post by forart.eu »

Here's the result of my 1st Video 2X upscale test on SHELLSHOCK AMV:

Author introduction (from this page):

By far this is my most austere video, a work of intense significance - tumultuously political and passionately opinionated. Like drugs and alcohol, war can infect even the most righteous of souls, and as seen through the eyes of Shiro Amada, chaos sides with neither friend nor foe.

Every decision in making this video, from song to scene selection, was totally deliberate. My intention was to emulate old war footage (WWII, Vietnam, etc.), which the raw quality of my MJPEG codec aptly provides. For research, I began watching The History Channel as well as every piece of film from Apocalypse Now to Saving Private Ryan. That direction proved more of a challenge than I realized. Countless revisions were done to many parts before settling on a sequence that worked.

There is a seemingly random dimension to the video, amplifying the nightmare of war. What I enforce upon the viewer is combat, terror, fear, blood, contempt, and death – juxtaposed with a message, a warning, and a lesson to learn.


Source file (download):

Code: Select all

Generale
Nome completo                           : Shellshock.mpg
Formato                                 : MPEG-PS
Dimensione                              : 67,1MiB
Durata                                  : 5 min 29s
Modo bitrate generale                   : Variabile
Bitrate totale                          : 1.708 kb/s
Compressore                             : encoded by TMPGEnc (ver. 2.02.31.119)

Video
ID                                      : 224 (0xE0)
Formato                                 : MPEG Video
Versione formato                        : Version 1
Impostazioni formato                    : CustomMatrix / BVOP
Impostazioni formato, BVOP              : Si
Impostazioni formato, Matrix            : Personalizzato
Impostazioni formato, GOP               : M=3, N=18
Durata                                  : 5 min 29s
Modalità bitrate                        : Variabile
Bitrate                                 : 1.482 kb/s
Larghezza                               : 352 pixel
Altezza                                 : 240 pixel
Rapporto aspetto visualizzazione        : 4:3
Frame rate                              : 29,970 (30000/1001) FPS
Standard                                : NTSC
Spazio colore                           : YUV
Croma subsampling                       : 4:2:0
Profondità bit                          : 8 bit
Tipo scansione                          : Progressivo
Modo compressione                       : Con perdita
Bit/(pixel*frame)                       : 0.585
Timecode del primo frame                : 00:00:00:00
Timecode sorgente                       : Group of pictures header
GOP, Open/Closed                        : Open
GOP, Open/Closed of first frame         : Closed
Dimensione della traccia                : 58,2MiB (87%)
Compressore                             : TMPGEnc 2.02.31.119

Audio
ID                                      : 192 (0xC0)
Formato                                 : MPEG Audio
Versione formato                        : Version 1
Profilo formato                         : Layer 2
Durata                                  : 5 min 29s
Modalità bitrate                        : Costante
Bitrate                                 : 192 kb/s
Canali                                  : 2 canali
Frequenza campionamento                 : 44,1 kHz
Frame rate                              : 38,281 FPS (1152 SPF)
Modo compressione                       : Con perdita
Dimensione della traccia                : 7,54MiB (11%)
Upscaled result (download):

Code: Select all

Generale
ID univoco                              : 113452538163437116898826089180663258847 (0x555A29FF489FA9ECBE61E603C0821EDF)
Nome completo                           : Shellshock_2X.mkv
Formato                                 : Matroska
Versione formato                        : Version 4
Dimensione                              : 168MiB
Durata                                  : 5 min 32s
Bitrate totale                          : 4.238 kb/s
Data codifica                           : UTC 2021-01-16 15:55:24
Creato con                              : mkvmerge v52.0.0 ('Secret For The Mad') 64-bit
Compressore                             : libebml v1.4.1 + libmatroska v1.6.2

Video
ID                                      : 1
Formato                                 : AVC
Formato/Informazioni                    : Advanced Video Codec
Profilo formato                         : High@L4
Impostazioni formato                    : CABAC / 6 Ref Frames
Impostazioni formato, CABAC             : Si
Impostazioni formato, RefFrames         : 6 frame
ID codec                                : V_MPEG4/ISO/AVC
Durata                                  : 5 min 29s
Bitrate                                 : 4.147 kb/s
Larghezza                               : 1.408 pixel
Altezza                                 : 960 pixel
Rapporto aspetto visualizzazione        : 3:2
Modalità frame rate                     : Costante
Frame rate                              : 29,970 (30000/1001) FPS
Spazio colore                           : YUV
Croma subsampling                       : 4:2:0
Profondità bit                          : 8 bit
Tipo scansione                          : Progressivo
Bit/(pixel*frame)                       : 0.102
Dimensione della traccia                : 163MiB (97%)
Compressore                             : x264 core 152 r2854 e9a5903
Impostazioni compressione               : cabac=1 / ref=6 / deblock=1:1:1 / analyse=0x3:0x113 / me=hex / subme=7 / psy=1 / psy_rd=0.40:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=3 / lookahead_threads=1 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=5 / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=25 / scenecut=40 / intra_refresh=0 / rc_lookahead=40 / rc=crf / mbtree=1 / crf=17.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40 / aq=1:0.60
Default                                 : Si
Forced                                  : No

Audio
ID                                      : 2
Formato                                 : AAC LC
Formato/Informazioni                    : Advanced Audio Codec Low Complexity
ID codec                                : A_AAC-2
Durata                                  : 5 min 24s
Bitrate                                 : 127 kb/s
Canali                                  : 2 canali
Channel layout                          : L R
Frequenza campionamento                 : 44,1 kHz
Frame rate                              : 43,066 FPS (1024 SPF)
Modo compressione                       : Con perdita
Ritardo video                           : 7s 954 ms
Dimensione della traccia                : 4,92MiB (3%)
Default                                 : Si
Forced                                  : No
"Side-to-side" (original vs upscaled):

Image
Image

Image
Image

Image
Image

Image
Image

Image
Image

Image
Image

Image
Image

Image
Image

Image
Image

Image
Image

Image
Image

Approx. - Google Collab - processing time (waifu2x ncnn Vulkan algorithm) 1 hour and 30 minutes
Post Reply