Page 1 of 1

Restoration project - fix choppy footage with duplicate frames

Posted: Thu Feb 13, 2020 1:30 pm
by derkork
I'm trying myself on a little restoration project. I have some home video footage which was made with some abysmal settings a long time ago. The input is a 320x240 MPEG-2 file. It has 25fps in the name, however the camera only recorded 12.5fps so in these 25fps the frames are duplicated. The sequence is that each image is duplicated for 3 frames for 8 times (total of 24frames) and then the next frame is only duplicated for 2 frames. Then again 8 triplets and one pair and so on. This makes for some very choppy footage. Now my idea was to fix this using some optical flow magic, but for this to work I figure I would need to remove the duplicate images so the optical flow algorithm sees the real sequence of images without duplicates.

So first and foremost - is this assumption correct, or can the optical flow algorithm handle duplicate frames? If it cannot, is there some clever way I can remove those duplicate frames with Fusion (e.g. some magic with the retime-controls)? Worst case i can still export the whole sequence to images and manually remove the duplicates but maybe there is a way that would involve less manual labor :)

Re: Restoration project - fix choppy footage with duplicate frames

Posted: Thu Feb 13, 2020 2:02 pm
by Midgardsormr
As it happens, I just did this exact thing with some old footage I'd been given. Start with a TimeStretcher that you'll use to drop out the duplicate frames. If the cadence is as steady as you say, then you should be able to set up the first 25 frames, compressing it down to... 13? 12? Anyway, get that done, then you should be able to use the Set Relative button in the Spline View such that the pattern repeats until the end. You just want to make sure you wind up with a unique image at every frame. Then add an OpticalFlow, followed by another TimeStretcher pulling it back out to the original length, with the Interpolation Mode set to Flow.

The issue you're going to see is that the interpolated frames will be a little softer than the key frames. You could shift all the keyframe values by 0.25 to force them all to be interpolated, which will reduce the sharpness of the footage—you will no longer have any 'real' frames—but it will also eliminate flicker.

Re: Restoration project - fix choppy footage with duplicate frames

Posted: Thu Feb 13, 2020 2:10 pm
by Kristof
I hope you have your sequencing correct. If you paste these in a comp that starts at 1 (and not 0 or anything else), this will get you back the original frames without the dupes. Check the final node in the buffer and you'll see it displays the correct frame number.
  1. {
  2.     Tools = ordered() {
  3.         Text1 = TextPlus {
  4.             Inputs = {
  5.                 GlobalIn = Input { Value = 1, },
  6.                 GlobalOut = Input { Value = 100, },
  7.                 Width = Input { Value = 1920, },
  8.                 Height = Input { Value = 1080, },
  9.                 ["Gamut.SLogVersion"] = Input { Value = FuID { "SLog2" }, },
  10.                 Font = Input { Value = "Open Sans", },
  11.                 StyledText = Input {
  12.                     SourceOp = "TimeCode1",
  13.                     Source = "Time",
  14.                 },
  15.                 Style = Input { Value = "Bold", },
  16.                 ManualFontKerningPlacement = Input {
  17.                     Value = StyledText {
  18.                         Array = {
  19.                         },
  20.                         Value = ""
  21.                     },
  22.                 },
  23.             },
  24.             ViewInfo = OperatorInfo { Pos = { 275, 247.5 } },
  25.         },
  26.         TimeCode1 = TimeCode {
  27.             CtrlWZoom = false,
  28.             Inputs = {
  29.                 Hrs = Input { Value = 0, },
  30.                 Mins = Input { Value = 0, },
  31.                 Secs = Input { Value = 0, },
  32.                 PadDigits = Input { Value = 4, },
  33.             },
  34.         },
  35.         TimeStretcher2 = TimeStretcher {
  36.             Inputs = {
  37.                 SourceTime = Input {
  38.                     SourceOp = "TimeStretcher2SourceTime",
  39.                     Source = "Value",
  40.                 },
  41.                 InterpolateBetweenFrames = Input { Value = 0, },
  42.                 SampleSpread = Input { Disabled = true, },
  43.                 Input = Input {
  44.                     SourceOp = "Text1",
  45.                     Source = "Output",
  46.                 },
  47.             },
  48.             ViewInfo = OperatorInfo { Pos = { 440, 247.5 } },
  49.         },
  50.         TimeStretcher2SourceTime = BezierSpline {
  51.             SplineColor = { Red = 233, Green = 217, Blue = 243 },
  52.             NameSet = true,
  53.             KeyFrames = {
  54.                 [1] = { 1, RH = { 2, 1 }, Flags = { Linear = true, Loop = true, LoopRel = true, PreLoop = true, PreLoopRel = true } },
  55.                 [4] = { 2, LH = { 4, 1.66666666666667 }, RH = { 5, 2 }, Flags = { StepIn = true } },
  56.                 [7] = { 3, LH = { 7, 2.66666666666667 }, RH = { 8, 3 }, Flags = { StepIn = true } },
  57.                 [10] = { 4, LH = { 10, 3.66666666666667 }, RH = { 11, 4 }, Flags = { StepIn = true } },
  58.                 [13] = { 5, LH = { 13, 4.66666666666667 }, RH = { 14, 5 }, Flags = { StepIn = true } },
  59.                 [16] = { 6, LH = { 16, 5.66666666666667 }, RH = { 17, 6 }, Flags = { StepIn = true } },
  60.                 [19] = { 7, LH = { 19, 6.66666666666667 }, RH = { 20, 7 }, Flags = { StepIn = true } },
  61.                 [22] = { 8, LH = { 22, 7.66666666666667 }, RH = { 23, 8 }, Flags = { StepIn = true } },
  62.                 [25] = { 9, LH = { 25, 8.66666666666667 }, RH = { 25.6666666666667, 9 }, Flags = { StepIn = true } },
  63.                 [27] = { 10, LH = { 27, 9.66666666666667 }, Flags = { StepIn = true, Loop = true, LoopRel = true, PreLoop = true, PreLoopRel = true } }
  64.             }
  65.         },
  66.         TimeStretcher1 = TimeStretcher {
  67.             CtrlWZoom = false,
  68.             Inputs = {
  69.                 SourceTime = Input {
  70.                     SourceOp = "TimeStretcher1SourceTime",
  71.                     Source = "Value",
  72.                 },
  73.                 InterpolateBetweenFrames = Input { Value = 0, },
  74.                 SampleSpread = Input { Disabled = true, },
  75.                 Input = Input {
  76.                     SourceOp = "TimeStretcher2",
  77.                     Source = "Output",
  78.                 },
  79.             },
  80.             ViewInfo = OperatorInfo { Pos = { 605, 247.5 } },
  81.         },
  82.         TimeStretcher1SourceTime = BezierSpline {
  83.             SplineColor = { Red = 233, Green = 217, Blue = 243 },
  84.             NameSet = true,
  85.             KeyFrames = {
  86.                 [1] = { 1, RH = { 1.33333333333333, 1 }, Flags = { Linear = true, Loop = true, LoopRel = true, PreLoop = true, PreLoopRel = true } },
  87.                 [2] = { 4, LH = { 2, 3 }, RH = { 2.33333333333333, 4 }, Flags = { StepIn = true } },
  88.                 [3] = { 7, LH = { 3, 6 }, RH = { 3.33333333333333, 7 }, Flags = { StepIn = true } },
  89.                 [4] = { 10, LH = { 4, 9 }, RH = { 4.33333333333333, 10 }, Flags = { StepIn = true } },
  90.                 [5] = { 13, LH = { 5, 12 }, RH = { 5.33333333333333, 13 }, Flags = { StepIn = true } },
  91.                 [6] = { 16, LH = { 6, 15 }, RH = { 6.33333333333333, 16 }, Flags = { StepIn = true } },
  92.                 [7] = { 19, LH = { 7, 18 }, RH = { 7.33333333333333, 19 }, Flags = { StepIn = true } },
  93.                 [8] = { 22, LH = { 8, 21 }, RH = { 8.33333333333333, 22 }, Flags = { StepIn = true } },
  94.                 [9] = { 25, LH = { 9, 24 }, RH = { 9.33333333333333, 25 }, Flags = { StepIn = true } },
  95.                 [10] = { 27, LH = { 10, 26.3333333333333 }, Flags = { StepIn = true, Loop = true, LoopRel = true, PreLoop = true, PreLoopRel = true } }
  96.             }
  97.         }
  98.     }
  99. }

Re: Restoration project - fix choppy footage with duplicate frames

Posted: Thu Feb 13, 2020 2:52 pm
by derkork
Thank you very much, this will help me to get going. I'm still not very familiar with the time stretcher, but now I have something I can play around with :)

Re: Restoration project - fix choppy footage with duplicate frames

Posted: Fri Feb 14, 2020 6:21 am
by derkork
So I gave it a shot and in general it worked pretty well. However it seems that the optical flow has a very hard time producing some convincing in-between frames. Actually the frame rate was only 8fps, so if I wanted to bring this to some half-way-fluid 24fps the optical flow would have to guess 66% of the frames. Since the video is also fast moving, this seems to be a dead end for now (unless we get some AI magic that is).