35 replies to this topic

[Pyro699]

Hey Everyone,
 
So as you may have noticed recently, Codex has been experiencing some downtime over the last few days. Yesterday, I finally went all in and turned a bunch of things off so that we could get a handle on what exactly was going on. Usually when something like this happens, the fix is quite obvious and I can take care of it in a couple of minutes. This time however, it required getting the old team back together and digging in real deep. Yeah, that's how bad things got - I had to call @Hydrogen (3 times - which brought back memories of the hydro-zone). @ShadowLink64 showed up to help us, and even @Waser Lave made a slack account so that we could talk and figure out what was going on. All very nostalgic and brought a tear to our eyes... However that tear was quickly wiped away by our frustration of the server unresponsiveness. No matter how many times we would bang on it, things just wouldn't shake loose.
 
At the root of the problem, something was taking all of the resources - and by "all of the resources" I mean that we would idle at around 160% CPU useage, so it was a little difficult to do testing while letting members onto the site. You might have noticed that there was a "403 Forbidden" message that appeared for you on the site earlier yesterday afternoon. This was in no way to imply that you were banned, we did receive an email inquiring about this and I wanted to clarify why it was that this was done at all. Basically, on the server you have the original contact on the server, then you go to the managing application and then you connect to a database of some sort. The usage spikes came in from the bottom two layers, so we cut off access for everyone at the server layer so that we could actually log into the server ourselves and figure out what was happening. I apologize if this caused anyone confusion yesterday, I am sure you eventually noticed that I replaced that message with one that had 'a little' more detail.
 
At the end of the day we still do not know what the root cause was, we have a couple theories but after 3 days of debugging we just decided to split up some of the server resources and have our database separate from the core of the site. I cannot promise we won't have another downtime like this, but if it does happen again, we will be able to track down the source a little easier.
 
With that in mind, I realize that some of you lost out on your subscriptions and would like some compensation for that. We are going to be extending all existing subscriptions by 2 weeks. We are also going to enable all advanced features (within the program manager only) for non-advanced members for a period of 1 week. Hopefully this makes up for some of the downtime that we have had.
 
Hope this explanation made a bit of sense to you.
 
Welcome back!
- Neocodex Administration

[Strategist]

Great to see you guys are still able to get on top of it all in times of need Cheers for working hard to get the site back up and running, somewhat, more smoothly Let's hope the issue stays away. 

 

Also, enjoy your free advanced guys!!! Such an awesome thing to do.

[Kaddict]

For a minute I was thinking "Cool, I get to try out some advanced shit" and then I realized... "oh, I have had that advanced shit for a year or two now"

 

Anyway. you guys are awesome for working to figure this out, and for hooking everyone up for what was really just a minor inconvenience. 

[Pryanka]

You guys are definitely going above and beyond to compensate for the downtime!

I'm glad everything's pretty much up and running for now

[Pyro699]

We are not here to make money. The only reason we have any kind of monetary typed system going on is to help fund the uptime of the server and it has currently let us down. We didn't feel it was right to charge when the service was not providing what we had originally promised. It is nothing out of our pocket to give back to the community that has helped this place become what it is today.

 

If you guys can think of other things to try and make this place better, please let us know and we will see what we can do

 

And before you ask Drak - No, we will not bring back the negative rep system.

[Swar]

We are not here to make money. The only reason we have any kind of monetary typed system going on is to help fund the uptime of the server and it has currently let us down. We didn't feel it was right to charge when the service was not providing what we had originally promised. It is nothing out of our pocket to give back to the community that has helped this place become what it is today.

 

[Romy]

Is it possible someone specifically targeted Neocodex's databse @Pyro699?

 

Be it for their own use (another autobuyer) or maliciously?

[ShadowLink64]

We didn't see any indication of that happening. What we did see a lot of were bots visiting our site to try and harvest content from the forums, itemdb, etc. We banned quite a few yesterday, which should reduce the overall traffic and put less of a burden on the server.

[Trapezeo]

Are the premium features for the program manager automatically started or do I need to do anything to get them?

[KaibaSama]

We didn't see any indication of that happening. What we did see a lot of were bots visiting our site to try and harvest content from the forums, itemdb, etc. We banned quite a few yesterday, which should reduce the overall traffic and put less of a burden on the server.

How exactly would harvesting that kind of data/content help them (as in the bot controllers/owners/whoever needed the data/ect)? 

 

We are not here to make money. The only reason we have any kind of monetary typed system going on is to help fund the uptime of the server and it has currently let us down. We didn't feel it was right to charge when the service was not providing what we had originally promised. It is nothing out of our pocket to give back to the community that has helped this place become what it is today.

 

If you guys can think of other things to try and make this place better, please let us know and we will see what we can do

 

And before you ask Drak - No, we will not bring back the negative rep system.

What about neutral rep? Where you don't give a plus or a minus, it's sort of like giving an "Eh" in rep. 

[Romy]

How exactly would harvesting that kind of data/content help them (as in the bot controllers/owners/whoever needed the data/ect)? 

Despite how slow neo is, I think out DB is still the most maintained.

 

As far as forum content, I have no idea.

[Jozie]

You guys are awesome!!!! Thank you for all of your hard work!!

[Pyro699]

Are the premium features for the program manager automatically started or do I need to do anything to get them?

It was as simple as me flipping a variable on the server. There is nothing you need to do for the manager to adapt the new "settings" other than to log out and log back in.

 

The only difference should be that there are inputs that were greyed out before - that are now enabled.

[Verge]

Has there been any recent development upgrade on the site? Maybe it's a case of you something got fixed and broke something else scenario. 

Another thing is you might want to check for the logs. That should give you an idea on whats going on. 

And also might wanna check if the server had some automatic updates done. Sometimes those screw things up to and would require some hotfixes.

 

You guys are awesome! Keep up the good work. I know you'll be able to fix it.

[Pyro699]

Has there been any recent development upgrade on the site? Maybe it's a case of you something got fixed and broke something else scenario. 

Another thing is you might want to check for the logs. That should give you an idea on whats going on. 

And also might wanna check if the server had some automatic updates done. Sometimes those screw things up to and would require some hotfixes.

 

You guys are awesome! Keep up the good work. I know you'll be able to fix it.

 

a) No we have not done any upgrades or changes to the software.

b) We are knowledgeable enough to know what logs to look in. This included us paying close attention to the server access logs to see who was connecting from what ip and with what useragent. It was must more complicated than we originally would have though.
c) with where we are running our servers, we are in control of everything - there are no automatic updates that could have been applied. But I do agree with you that autoupdates suck and break a lot... Most recently windows 10 autoupdated and broke my virtualbox setups - great times.

 

We are planning some server fixes and updates that will hopefully reduce the possibility of this happening again.

[KaibaSama]

So, not giving any rep at all?

Exactly. You feel good because there's a 1 by your post, then you realize it's grey so it means nothing and no one actually gave you any rep at all, so you feel a bit sad. 

[KaibaSama]

That's beautifully evil. 

And passive aggressive as fuck.

I love it. 

It's who I am. It's who I am. 

Basically it's a way to make someone feel bad that their post wasn't good enough for a positive rep, without needing to bring back negative rep. 

[Junjie]

It's who I am. It's who I am. 
Basically it's a way to make someone feel bad that their post wasn't good enough for a positive rep, without needing to bring back negative rep.

Oh my god

[renegade]

Is this still an ongoing issue? I can't get the program manager to start. The log says this:

 

 

Traceback (most recent call last):

  File "pre_loader.py", line 54, in <module>

  File "urllib2.pyc", line 126, in urlopen

  File "urllib2.pyc", line 394, in open

  File "urllib2.pyc", line 412, in _open

  File "urllib2.pyc", line 372, in _call_chain

  File "urllib2.pyc", line 1199, in http_open

  File "urllib2.pyc", line 1174, in do_open

urllib2.URLError: <urlopen error [Errno 11001] getaddrinfo failed>

 

I've tried uninstalling and reinstalling to no avail. 

[Pyro699]

Is this still an ongoing issue? I can't get the program manager to start. The log says this:

 

 

I've tried uninstalling and reinstalling to no avail. 

 

The downtime happened several weeks ago... This is something else.

 

Is it still happening? Are you behind a firewall? Can you access neocodex.us at the same time you get the error?

[renegade]

The downtime happened several weeks ago... This is something else.

 

Is it still happening? Are you behind a firewall? Can you access neocodex.us at the same time you get the error?

 

Yeah, it's still happening and it just started happening today (I use program manager regularly, and I used it last night). I haven't made any changes to my computer and I have no issues accessing neocodex.us. The error is almost instantaneous after the "downloading updates from neocodex.us" window pops up. Also, @Liesa said she is experiencing the same issue when I posted about the error here: http://www.neocodex....s/#entry1993983