Incident Report – Some new recordings showing under moderation message

The incident happened on 8th of April this year (2022).

Incident Effects

The incident effects were seen as videos that were showing 'Under Moderation' message, however in dashboard they would show READY status. This has affected only a small portion of the new videos that were coming in our system. This incident did not affect the existing videos nor their availability in any manner.

Who was affected

This has affected a very small portion of videos that came in, so customers that within the timeframe had more videos coming in, had a greater chance of this error to pop up and affect their workflow.
This means that all videos that were coming in before or after were not affected. It also means that most videos during this timeframe would also have been properly processed.

What happened

We have released a small update to our processing system to the production part of our system. This is something that is known to happen and generally OK. Before it is done, we do several things. We run many different tests and validations before it can be signed off and pushed to production. All our pre-release tests were passed with flying colors.This update was made to make the transcoding better and more reliable. That said, it revealed that in some specific cases, it would cause more trouble than it would help solve.

The detection and the fix

As videos were passing through as READY it was a bit hard to notice that something was off. Thanks to a very quick report by some of you, we were able to detect and connect where the issue is coming from. By that time just a bit shy of 3 hours have passed since the update.
Since the videos were in READY state, it means that simply looking into logs or statuses for failed videos was not an option. Checking what would be the best way to approach this, our team had made a decision that we believe to have been the best one to do. That was to re-process all of the videos that came in during that 3 hour window.
Because we are running highly scalable system with many different servers operational at the time, we have decided to be safe and run the transcoding on all videos that came in since the moment we have released the update and up to 4 hours since. This covered the 3 hour window and also made sure that it covers any instances that were still running the older version of code.

Aftermath

Knowing what code caused the issue and seeing the videos that had failed, our team has prepared for analyzing everything. It is found that all videos should be fixed. We do have a note to attach to this. While it looks like all videos are processed, we are aware that there might still be few videos here and there that will show the under moderation message. We will explain about this near the end of this report.

Plans for the future

We are very happy with the way we have rolled out this update. It did result in some videos to fail, however we followed all of our own key concepts and made sure that everything works right - as far as we can tell. We have also released the update earlier on that day. Day, when we knew that we will be available for quite a lot of time if something did go wrong.
So steps wise, we do not think that there is anything that we could have done differently. What we do plan to do is to extend our tests a bit more to help further minimize the effects of what we create if they were to go down the road we do not want it to go.

Final Thoughts

As we said, a very small number of videos was affected by this. The timeframe is between 1649424600 and 1649435400 ( 1:30PM GMT+0 to 4:30PM GMT+0 ). The "healing process" script was made to run for videos between 1:30PM and 5:30PM GMT+0.
The process was automatically run and everything should be updated for you. If you do have any videos that seem like they are under moderation (and you think they should not be), you can follow the next steps to see if you can re-transcode them:

1. Log into your Dashboard
2. Go to Videos page
3. Find and click on the video you want to check. Under its preview on the right, click on the link that says "View Single Streams"
4. Right of the first stream (top most) you will see "re-transcode" option. This will need few moments and will then recreate the video for you

What would be good to keep in mind is that sometimes videos do fail to be transcoded for different reasons. This is also why it is possible that even after the fix there are still some videos that show "Under Moderation" message or are showing with FAILED status. If the re-transcode option did not work for you, please feel free to let us know of such video to check it out.
It is possible that the video was not finalized or is otherwise corrupt, so it might appear to be working, however is not. This would not be related to this incident, and we might be able to release a small update that addresses such videos.