How to avoid interpolating scene changes?

[hsilva664]

Hello, I noticed that sometimes the model attempts to interpolate scene changes, which leads to bad results. This is very noticeable when watching the videos (see the example below). Simply not doing the interpolation in these cases would be enough to greatly improve results. Is there any parameter to avoid interpolation in these cases?

[TNTwise]

I'm pretty sure this is only a feature in vapoursynth-rife-ncnn-vulkan , or you could just use a frame interpolation GUI that uses rife-ncnn-vulkan and supports scene change detection.

If you want something to extract the transition frames, and you move them back into the frame directory, you can use ffmpeg to extract those frames (it wont extract the frame number though, you would have to multiply the timestamp extracted by the framerate and multiply that by the num_frame multiplier)

ffmpeg -i "/path/to/input/file" -filter_complex "select='gt(scene,0.3)',metadata=print" -vsync vfr "path/to/transition/%08d.(image_type)"

I am sure flowframes and enhancr both support scene change detection, and if you are looking for something on linux you could use my project: REAL-Video-Enhancer

[hsilva664]

Hello, thanks for sharing these resources. I'm currently running things on headless remote servers, so I'm not really using GUI's. I'll take a better look at vapoursynth-rife-ncnn-vulkan.

The manual approach seems interesting, but it looks like it might be hard to get the exact frames I want. I assume you mean to store the transition frames temporarily and then reinsert them in the right positions in the final result (overwriting the blurred frames). However, the above transition, for example, has two blurred frames instead of one. I think this might be because I set the output number of frames explicitly (such as two get from 23.97fps to 60fps) instead of, say, adding one new frame for each 2 existing ones.

[TNTwise]

Sorry for the late response!
vapoursynth-rife-ncnn-vulkan is a little complex to use, and if you want to try this, i have created a manual script to detect scene changes, and can adapt to variable timesteps(i tested from 24-60 and it did it), please tell me if you encounter any issues. This script assumes ffmpeg and ffprobe are installed by default, otherwise no pip dependencies from my knowledge.

(EDIT: it also removes and creates a folder called transitions in the directory the video inputed is in, so make sure you dont have an important folder called transitions there lol)
save this script as script.py and run it with python3 script.py
EDIT2: havent tested this with other than 2.5x timestep, may not work great with others. im going to fix this later, just dont have the time right now.

import os
import subprocess
import re
import math
def extract(Image_Type,input_file,SceneChangeDetection,times,amount_of_zeros=8):
    os.chdir(os.path.dirname(file))
    os.system('rm -rf transitions')
    os.mkdir('transitions/') 
    if Image_Type != '.webp':
                    ffmpeg_cmd = f'ffmpeg -i "{input_file}" -filter_complex "select=\'gt(scene\,{SceneChangeDetection})\',metadata=print" -vsync vfr -q:v 1 "transitions/%07d.{Image_Type}"' 
    else:
                    ffmpeg_cmd = f'ffmpeg -i "{input_file}" -filter_complex "select=\'gt(scene\,{SceneChangeDetection})\',metadata=print" -vsync vfr -q:v 100 "transitions/%07d.png"'

    output = subprocess.check_output(ffmpeg_cmd, shell=True, stderr=subprocess.STDOUT)
    
    output_lines = output.decode("utf-8").split("\n")
    timestamps = []

    
    ffprobe_cmd = f"ffprobe -v error -select_streams v -of default=noprint_wrappers=1:nokey=1 -show_entries stream=r_frame_rate {input_file}"
    result = subprocess.check_output(ffprobe_cmd, shell=True).decode("utf-8")

    match = re.match(r'(\d+)/(\d+)', result)
    
    numerator, denominator = match.groups()
    fps = int(numerator) / int(denominator)
        

    fps_value = int(numerator) / int(denominator) if match else None
    for line in output_lines:
                if "pts_time" in line:
                    timestamp = str(line.split("_")[3])
                    timestamp = str(timestamp.split(':')[1])
                    timestamps.append(math.ceil(round(float(timestamp)*float(fps_value))*times))
    transitions = os.listdir('transitions/')
    for iteration,i in enumerate(transitions):
        
        if Image_Type != '.webp':
                    os.system(f'mv transitions/{str(str(iteration+1).zfill(7))}.{Image_Type} transitions/{timestamps[iteration]}.{Image_Type}')
        else:
                    os.system(f'mv transitions/{str(str(iteration+1).zfill(7))}.png transitions/{timestamps[iteration]}.{Image_Type}')
    for i in timestamps:
            for j in range(math.ceil(times)):
                    os.system(f'cp transitions/{i}.{Image_Type} transitions/{str(int(i)-j).zfill(amount_of_zeros)}.{Image_Type}' )
            os.remove(f'transitions/{i}.{Image_Type}')
image = input('Please pick an image type:\n1:PNG\n2:JPG\n3:WEBP\n(Please pick 1,2 or 3): ')
if image == '1':
        image = 'png'
elif image == '2':
        image = 'jpg'
elif image == '3':
        image = 'webp'
else:
    print('Invalid answer!')
    exit()

file = input('\nPlease paste input file location here: ')
if os.path.isfile(file) == False:
        print('Not a file!')
        exit()

sensativity=input('\nPlease enter the sensitivity (0-9)\n(0 means most sensitive, meaning it will detect the most frames as scene changes)\n(9 being the least sensitive, meaning it will detect the least amount of frames as scene changes.)\n: ')
try:
    if int(sensativity) > 9 or int(sensativity) < 0:
            print('invalid sensitivity')
            exit()
except:
        print('Not an integer!')
        exit()
try:
    timestep = input('\nPlease enter a timestep (not the number of frames, just the multiplier): ')
except:
        print('Not a float!')
        exit()
try:
    if len(timestep) > 1:
        timestep = float(timestep)
    else:
        timestep=int(timestep)
except:
        print('invalid!')
        exit()
try: 
    amount_of_zeros = int(input('\nPlease enter the num you put in for the amount of zeros per frame, default = %08d\nif you changed this value in extraction of frames, please put that value here\nif not, just skip this. '))
except:
        amount_of_zeros=8
extract(image,file,f'0.{sensativity}',timestep,amount_of_zeros)

[hsilva664]

I did some preliminary testing. I think you meant to subtract frames in the last loop, rather than add, since the frame returned by the complex filtergraph is the one after the transition, not before. Other than that, it seems to be working. Thanks!

Edit: it seems to be correct, I had counted wrong before my previous comment because I was correcting the index.