So as you may have noticed recently, Codex has been experiencing some downtime over the last few days. Yesterday, I finally went all in and turned a bunch of things off so that we could get a handle on what exactly was going on. Usually when something like this happens, the fix is quite obvious and I can take care of it in a couple of minutes. This time however, it required getting the old team back together and digging in real deep. Yeah, that's how bad things got - I had to call @Hydrogen (3 times - which brought back memories of the hydro-zone). @ShadowLink64 showed up to help us, and even @Waser Lave made a slack account so that we could talk and figure out what was going on. All very nostalgic and brought a tear to our eyes... However that tear was quickly wiped away by our frustration of the server unresponsiveness. No matter how many times we would bang on it, things just wouldn't shake loose.
At the root of the problem, something was taking all of the resources - and by "all of the resources" I mean that we would idle at around 160% CPU useage, so it was a little difficult to do testing while letting members onto the site. You might have noticed that there was a "403 Forbidden" message that appeared for you on the site earlier yesterday afternoon. This was in no way to imply that you were banned, we did receive an email inquiring about this and I wanted to clarify why it was that this was done at all. Basically, on the server you have the original contact on the server, then you go to the managing application and then you connect to a database of some sort. The usage spikes came in from the bottom two layers, so we cut off access for everyone at the server layer so that we could actually log into the server ourselves and figure out what was happening. I apologize if this caused anyone confusion yesterday, I am sure you eventually noticed that I replaced that message with one that had 'a little' more detail.
At the end of the day we still do not know what the root cause was, we have a couple theories but after 3 days of debugging we just decided to split up some of the server resources and have our database separate from the core of the site. I cannot promise we won't have another downtime like this, but if it does happen again, we will be able to track down the source a little easier.
With that in mind, I realize that some of you lost out on your subscriptions and would like some compensation for that. We are going to be extending all existing subscriptions by 2 weeks. We are also going to enable all advanced features (within the program manager only) for non-advanced members for a period of 1 week. Hopefully this makes up for some of the downtime that we have had.
Hope this explanation made a bit of sense to you.
- Neocodex Administration