Coding Period Week 5(5th July — 12th July)

Meeting Minutes-:

• During the meeting, we discussed issues related to the robust face swapper logic (flicker.py file). Daniel highlighted two main problems encountered with the videos. Firstly, when using FSGAN, there were instances where certain frames were left unanonymized despite applying the flicker removal algorithm. Secondly, there was an issue of audio-video synchronization, particularly noticeable in longer-duration videos.
• In the flicker.py file, we have implemented a threshold value to determine whether a frame should be anonymized or not. This threshold is calculated based on the Mean Squared Error (MSE) between corresponding frames of the anonymized and unanonymized videos. By adjusting this threshold value, we may be able to address the first issue.
• Second problem is probably due to the fact that the videos have some corrupted frames which are dropped when we are reading the video using ffmpeg software. As the audio is continuous in nature the effective length of video decreases while the length of audio remains same. This may cause some audio-video async in the final video. This issue is observed in the videos of longer duration.

Goal of this week-:

• The goal of this week is to remove the bugs present in the flicker.py file, test the RHA on videos that Daniel will be sharing and compile features and limitations of RHA that will be be updated on the Techne site. The secondary goal is to finalize the technology that we will be using to build the webpage.

Progress-:

• This week began with debugging the flicker.py file. Initially I suspected that the problem was with the threshold value as Daniel observed unanoymized frames in only two videos. However on closer examination videos, I found out that the problem was occurring specifically in videos where anonymized frames were followed by unanonymized frames.
  I realized that I had made a minor logical mistake in the code. I was considering the MSE value of next frame to decide whether the current frame should be anonymized using hider or left unchanged. This was the root casue of the problem. Finally the problem was solved.
• Complied the features of RHA and shared it with Daniel. I will update the finalized version of the content on the Techne website once it is reviewed.
• Started testing on new dataset shared by Daniel. The videos are of higher resolution and are recorded in good lighting condition. But the videos are in .MTS format which is not supported by RHA. So I converted the video to mp4 format using this command.

ffmpeg -i input.m2ts -c:v copy -c:a aac -strict experimental -b:a 128k output.mp4

• Code improvement-: In flicker.py file, I was converting the video into frames and was storing those frames in a list and then in another iteration I was processing them. This is not the optimal approach. As the length of video becomes longer, the number of frames increases and there is a chance that you may run out of memory. Better alternative to process them frame by frame in the first iteration itself and not store them in a list.
• Another problem-: I noticed that while testing on one of the videos(Sujeto1.MTS) that daniel shared(stored under the “Sujeto Videos” directory), the rha was working fine till swapper stage, after that in flicker removal stage, it effectively removes flickers till some frames. After which it suddenly starts taking very high time to process each frame(around 10⁴ times ideal time) which is not feasible.
  I am not sure what might be the issue. I have to check if this problem persists for other videos(“Sujeto Videos”) also.