Incident Report – Autopush failed for some customers

The incident happened starting with 16th and was fixed on 19th of January 2022.

Incident Effects

The incident effects were only possible to be seen by inspecting your push location to not find the media that was supposed to be there.

The incident did not affect any other aspect of our service. It did not otherwise affect the media that was recorded or uploaded at the time neither.

Who was affected

The issue was specific to auto push feature. The manual push was not affected by this.
Also not everyone was affected by this.

What happened

As you know, Ziggeo was for a long time associated with video. Currently we are happy to say that Ziggeo is a service that offers all round support for video, audio as well as images.

To make that switch, we had to make a lot of backend changes allowing us to support different media types. As some of the types of media require different processing and yet to some extent also touch on same points, wording was also something we had to work on.

Generally wording might not seem that important, however our goals are to write simple and clear codes. Well as much as we can! Wording can make it easier to create, extend and debug something, so doing it earlier is definitely better.

Due to how our system is running a change in wording caused our system to run the auto push in a way that cased no errors. However some of our customers did not really get the video pushed over.

This happened over a time as our update was slowly going through and a mismatch was presented in doing so.

The detection and the fix

As everything was running smoothly on our side this was under the radar for us at first. Once You started writing to us, we started paying more attention to this specific part of our system.

Wouldn't you tell, there it was. Otherwise OK and looking proper yet actually a failing push.

Once we found this (thank you all again for reporting it), our engineers started checking different parts of push service to locate the actual cause. As this was located, the fix was introduced making it all work as normal once again.

Aftermath

During the process of locating the issue, our engineers also found the small traces this specific issue was leaving behind. What happened next was to see if the implemented fix solved the issue for everyone. This was followed by a confirmation that everything else was still working properly.

Once that was OKed, all videos that failed to be delivered were set up to be pushed once again.

Plans for the future

Well we have added various detection helpers and we utilize different tools on premises to help detect issues as well. It is clear to us that sometimes something can happen that we can not really know about on our own. That is, not know about it until you report to us.

We will be on a better lookout for anything that is reported by our customers. For example, where same issue might be reported and experienced on 2 or more accounts. Likewise, we are already adding many more automated tests. These will help us know exactly when something bad is about to happen, before it leaves our development cycles.

We are also considering a bit different way of pushing updates that require all code to be "on same page". Generally we prefer a slow rollout. We recognize though that in some cases that might not be possible and we might need to flush everything with new updates.

Final Thoughts

While we are not too happy having anything undesirable happen, we are happy to let you know that all your videos will be automatically pushed to your push locations. We are also grateful for all of our customers that reached out to us with valuable information on this. Your emails have helped track and fix this, and that is always appreciated.

We do also want to remind you of our status page. While it was not possible for it to detect this specific issue, it is a great place to check first to see if something might not work as planned.

To reach out to us at any time, just send an email to our support.