Over the past few days we have been making the largest changes to Kiwi IRC to date. Greatly improved mobile support, lower memory usage, far more consistent plugin and server API, better styled themes, just to name a few changes.
Unfortunately, we hit some issues involving some of the new features and our updating methods which effected a lot of people and networks. We now have all the issues fixed and returned Kiwi IRC back to its normal stable self. I apologise for the issues and disconnections people and networks had noticed.
Details
Previous updates have been much more subtle and easier to roll out, however the updates over the past few days required changes right across the Kiwi IRC platform. Unfortunately this didn't quite go to plan and as smoothly as hoped. A lot of people were getting disconnected and in some cases could not ever connect at all.
What went wrong
There were two main issues that caused a lot of the pain, both causing similar symptoms which made things more difficult to diagnose.
1) A new feature that reconnected users to the IRC network if the IRC server had disconnected them.
It turns out that basic reconnections don't work so well with large numbers of people. We were hitting an issue in which a popular IRC server would loose network connectivity causing hundreds of disconnects, which would then cause high CPU + network traffic when they reconnect. This in turn would knock other IRC connections offline causing more reconnects, repeat, repeat, repeat. This has now been disabled until a better, more sophisticated method is in place.
2) As always with new Kiwi IRC updates we attempt to clear and bypass all browser caches to make sure people are updating correctly and browsers are downloading the latest versions. This time we had to update both the browser and Kiwi servers at the same time due to the large changes make to improve Kiwi. However places such as some proxy servers have much stricter caching policies that wouldn't budge with our usual attempts. People that were hitting a cached version were unable to connect without any warning as to why they were having the issue.
Changes for the near future
With these main issues and other minor problems now under control and for the most part resolved, we can now be making some changes to be sure that future updates do not cause the same interruptions to IRC networks and communities that use the hosted version on kiwiirc.com.
1) Most, if not all problems would have been found earlier had we gradually updated people over time instead of having a single large switch over. This means extra resources to be running multiple versions of Kiwiirc.com but it is required. Luckily we live in the time of the cloud to make this easier.
2) The methods used to get around caches have already been made and are in place. A noticeable drop in people using older version has been seen which is a good sign it is already working. Hopefully this follows on in the future.
3) People using Kiwi IRC whether it be embedded into a website or using it directly need to know what's happening when large changes are happening. Being left in the dark while getting disconnected all the time is worse than knowing why it's happening. Broadcasting messages to all users will be implemented to solve this.
4) Find better timing. Kiwi IRC is an open source project and is run purely by volunteers. All the server maintenance, abuse and system monitoring, development and support are done in spare time by a very small team. Unfortunately, I had chosen to make such a large update while under work load from other work commitments which mixing the two together is not very fun for stress levels - or anyone else using kiwiirc.com.
And now
Now that we are mostly settled and people can start to see the updates made, we are now looking forward to hearing feedback!
Was switching the channel tabs to simpler text a good move? Does the client work better on your phones / tablets? People that use the embedded widget on your websites – does it resize any better, especially the nick list auto hiding on smaller widgets?
There are areas for improvement as always, we want to know which areas should be focussed on by the people that use the client. The Kiwi IRC mailing list can be found at https://groups.google.com/d/forum/kiwiirc while our IRC channel is on irc.freenode.net, #kiwiirc. (Before people go nuts: IRC persistence or IRCCloud like functionality is upcoming, we already know this is a number one wanted feature :)
Again, I apologise the Kiwi IRC update has been rough. Kiwiirc.com should be stable despite being a free and open source service and that is my continued aim.