Restoration project - fix choppy footage with duplicate frames

Moderator: SecondMan

User avatar
derkork
Posts: 12
Joined: Fri Aug 10, 2018 2:39 am

Restoration project - fix choppy footage with duplicate frames

#1

Post by derkork » Thu Feb 13, 2020 1:30 pm

I'm trying myself on a little restoration project. I have some home video footage which was made with some abysmal settings a long time ago. The input is a 320x240 MPEG-2 file. It has 25fps in the name, however the camera only recorded 12.5fps so in these 25fps the frames are duplicated. The sequence is that each image is duplicated for 3 frames for 8 times (total of 24frames) and then the next frame is only duplicated for 2 frames. Then again 8 triplets and one pair and so on. This makes for some very choppy footage. Now my idea was to fix this using some optical flow magic, but for this to work I figure I would need to remove the duplicate images so the optical flow algorithm sees the real sequence of images without duplicates.

So first and foremost - is this assumption correct, or can the optical flow algorithm handle duplicate frames? If it cannot, is there some clever way I can remove those duplicate frames with Fusion (e.g. some magic with the retime-controls)? Worst case i can still export the whole sequence to images and manually remove the duplicates but maybe there is a way that would involve less manual labor :)

User avatar
Midgardsormr
Fusionator
Posts: 1487
Joined: Wed Nov 26, 2014 8:04 pm
Answers: 8
Location: Los Angeles, CA, USA
Been thanked: 34 times
Contact:

Re: Restoration project - fix choppy footage with duplicate frames

#2

Post by Midgardsormr » Thu Feb 13, 2020 2:02 pm

As it happens, I just did this exact thing with some old footage I'd been given. Start with a TimeStretcher that you'll use to drop out the duplicate frames. If the cadence is as steady as you say, then you should be able to set up the first 25 frames, compressing it down to... 13? 12? Anyway, get that done, then you should be able to use the Set Relative button in the Spline View such that the pattern repeats until the end. You just want to make sure you wind up with a unique image at every frame. Then add an OpticalFlow, followed by another TimeStretcher pulling it back out to the original length, with the Interpolation Mode set to Flow.

The issue you're going to see is that the interpolated frames will be a little softer than the key frames. You could shift all the keyframe values by 0.25 to force them all to be interpolated, which will reduce the sharpness of the footage—you will no longer have any 'real' frames—but it will also eliminate flicker.

User avatar
Kristof
Fusionista
Posts: 683
Joined: Thu Aug 07, 2014 1:30 pm
Answers: 1
Been thanked: 5 times

Re: Restoration project - fix choppy footage with duplicate frames

#3

Post by Kristof » Thu Feb 13, 2020 2:10 pm

I hope you have your sequencing correct. If you paste these in a comp that starts at 1 (and not 0 or anything else), this will get you back the original frames without the dupes. Check the final node in the buffer and you'll see it displays the correct frame number.
  1. {
  2.     Tools = ordered() {
  3.         Text1 = TextPlus {
  4.             Inputs = {
  5.                 GlobalIn = Input { Value = 1, },
  6.                 GlobalOut = Input { Value = 100, },
  7.                 Width = Input { Value = 1920, },
  8.                 Height = Input { Value = 1080, },
  9.                 ["Gamut.SLogVersion"] = Input { Value = FuID { "SLog2" }, },
  10.                 Font = Input { Value = "Open Sans", },
  11.                 StyledText = Input {
  12.                     SourceOp = "TimeCode1",
  13.                     Source = "Time",
  14.                 },
  15.                 Style = Input { Value = "Bold", },
  16.                 ManualFontKerningPlacement = Input {
  17.                     Value = StyledText {
  18.                         Array = {
  19.                         },
  20.                         Value = ""
  21.                     },
  22.                 },
  23.             },
  24.             ViewInfo = OperatorInfo { Pos = { 275, 247.5 } },
  25.         },
  26.         TimeCode1 = TimeCode {
  27.             CtrlWZoom = false,
  28.             Inputs = {
  29.                 Hrs = Input { Value = 0, },
  30.                 Mins = Input { Value = 0, },
  31.                 Secs = Input { Value = 0, },
  32.                 PadDigits = Input { Value = 4, },
  33.             },
  34.         },
  35.         TimeStretcher2 = TimeStretcher {
  36.             Inputs = {
  37.                 SourceTime = Input {
  38.                     SourceOp = "TimeStretcher2SourceTime",
  39.                     Source = "Value",
  40.                 },
  41.                 InterpolateBetweenFrames = Input { Value = 0, },
  42.                 SampleSpread = Input { Disabled = true, },
  43.                 Input = Input {
  44.                     SourceOp = "Text1",
  45.                     Source = "Output",
  46.                 },
  47.             },
  48.             ViewInfo = OperatorInfo { Pos = { 440, 247.5 } },
  49.         },
  50.         TimeStretcher2SourceTime = BezierSpline {
  51.             SplineColor = { Red = 233, Green = 217, Blue = 243 },
  52.             NameSet = true,
  53.             KeyFrames = {
  54.                 [1] = { 1, RH = { 2, 1 }, Flags = { Linear = true, Loop = true, LoopRel = true, PreLoop = true, PreLoopRel = true } },
  55.                 [4] = { 2, LH = { 4, 1.66666666666667 }, RH = { 5, 2 }, Flags = { StepIn = true } },
  56.                 [7] = { 3, LH = { 7, 2.66666666666667 }, RH = { 8, 3 }, Flags = { StepIn = true } },
  57.                 [10] = { 4, LH = { 10, 3.66666666666667 }, RH = { 11, 4 }, Flags = { StepIn = true } },
  58.                 [13] = { 5, LH = { 13, 4.66666666666667 }, RH = { 14, 5 }, Flags = { StepIn = true } },
  59.                 [16] = { 6, LH = { 16, 5.66666666666667 }, RH = { 17, 6 }, Flags = { StepIn = true } },
  60.                 [19] = { 7, LH = { 19, 6.66666666666667 }, RH = { 20, 7 }, Flags = { StepIn = true } },
  61.                 [22] = { 8, LH = { 22, 7.66666666666667 }, RH = { 23, 8 }, Flags = { StepIn = true } },
  62.                 [25] = { 9, LH = { 25, 8.66666666666667 }, RH = { 25.6666666666667, 9 }, Flags = { StepIn = true } },
  63.                 [27] = { 10, LH = { 27, 9.66666666666667 }, Flags = { StepIn = true, Loop = true, LoopRel = true, PreLoop = true, PreLoopRel = true } }
  64.             }
  65.         },
  66.         TimeStretcher1 = TimeStretcher {
  67.             CtrlWZoom = false,
  68.             Inputs = {
  69.                 SourceTime = Input {
  70.                     SourceOp = "TimeStretcher1SourceTime",
  71.                     Source = "Value",
  72.                 },
  73.                 InterpolateBetweenFrames = Input { Value = 0, },
  74.                 SampleSpread = Input { Disabled = true, },
  75.                 Input = Input {
  76.                     SourceOp = "TimeStretcher2",
  77.                     Source = "Output",
  78.                 },
  79.             },
  80.             ViewInfo = OperatorInfo { Pos = { 605, 247.5 } },
  81.         },
  82.         TimeStretcher1SourceTime = BezierSpline {
  83.             SplineColor = { Red = 233, Green = 217, Blue = 243 },
  84.             NameSet = true,
  85.             KeyFrames = {
  86.                 [1] = { 1, RH = { 1.33333333333333, 1 }, Flags = { Linear = true, Loop = true, LoopRel = true, PreLoop = true, PreLoopRel = true } },
  87.                 [2] = { 4, LH = { 2, 3 }, RH = { 2.33333333333333, 4 }, Flags = { StepIn = true } },
  88.                 [3] = { 7, LH = { 3, 6 }, RH = { 3.33333333333333, 7 }, Flags = { StepIn = true } },
  89.                 [4] = { 10, LH = { 4, 9 }, RH = { 4.33333333333333, 10 }, Flags = { StepIn = true } },
  90.                 [5] = { 13, LH = { 5, 12 }, RH = { 5.33333333333333, 13 }, Flags = { StepIn = true } },
  91.                 [6] = { 16, LH = { 6, 15 }, RH = { 6.33333333333333, 16 }, Flags = { StepIn = true } },
  92.                 [7] = { 19, LH = { 7, 18 }, RH = { 7.33333333333333, 19 }, Flags = { StepIn = true } },
  93.                 [8] = { 22, LH = { 8, 21 }, RH = { 8.33333333333333, 22 }, Flags = { StepIn = true } },
  94.                 [9] = { 25, LH = { 9, 24 }, RH = { 9.33333333333333, 25 }, Flags = { StepIn = true } },
  95.                 [10] = { 27, LH = { 10, 26.3333333333333 }, Flags = { StepIn = true, Loop = true, LoopRel = true, PreLoop = true, PreLoopRel = true } }
  96.             }
  97.         }
  98.     }
  99. }
Last edited by Kristof on Thu Feb 13, 2020 3:12 pm, edited 1 time in total.

User avatar
derkork
Posts: 12
Joined: Fri Aug 10, 2018 2:39 am

Re: Restoration project - fix choppy footage with duplicate frames

#4

Post by derkork » Thu Feb 13, 2020 2:52 pm

Thank you very much, this will help me to get going. I'm still not very familiar with the time stretcher, but now I have something I can play around with :)

User avatar
derkork
Posts: 12
Joined: Fri Aug 10, 2018 2:39 am

Re: Restoration project - fix choppy footage with duplicate frames

#5

Post by derkork » Fri Feb 14, 2020 6:21 am

So I gave it a shot and in general it worked pretty well. However it seems that the optical flow has a very hard time producing some convincing in-between frames. Actually the frame rate was only 8fps, so if I wanted to bring this to some half-way-fluid 24fps the optical flow would have to guess 66% of the frames. Since the video is also fast moving, this seems to be a dead end for now (unless we get some AI magic that is).