Quantcast

Jump to content


Photo

Recent Downtime


  • Please log in to reply
36 replies to this topic

#1 Pyro699

Pyro699
  • 1,395 posts


Users Awards

Posted 10 September 2016 - 02:44 PM

*
POPULAR POST!

Hey Everyone,
 
So as you may have noticed recently, Codex has been experiencing some downtime over the last few days. Yesterday, I finally went all in and turned a bunch of things off so that we could get a handle on what exactly was going on. Usually when something like this happens, the fix is quite obvious and I can take care of it in a couple of minutes. This time however, it required getting the old team back together and digging in real deep. Yeah, that's how bad things got - I had to call @Hydrogen (3 times - which brought back memories of the hydro-zone). @ShadowLink64 showed up to help us, and even @Waser Lave made a slack account so that we could talk and figure out what was going on. All very nostalgic and brought a tear to our eyes... However that tear was quickly wiped away by our frustration of the server unresponsiveness. No matter how many times we would bang on it, things just wouldn't shake loose.
 
At the root of the problem, something was taking all of the resources - and by "all of the resources" I mean that we would idle at around 160% CPU useage, so it was a little difficult to do testing while letting members onto the site. You might have noticed that there was a "403 Forbidden" message that appeared for you on the site earlier yesterday afternoon. This was in no way to imply that you were banned, we did receive an email inquiring about this and I wanted to clarify why it was that this was done at all. Basically, on the server you have the original contact on the server, then you go to the managing application and then you connect to a database of some sort. The usage spikes came in from the bottom two layers, so we cut off access for everyone at the server layer so that we could actually log into the server ourselves and figure out what was happening. I apologize if this caused anyone confusion yesterday, I am sure you eventually noticed that I replaced that message with one that had 'a little' more detail.
 
At the end of the day we still do not know what the root cause was, we have a couple theories but after 3 days of debugging we just decided to split up some of the server resources and have our database separate from the core of the site. I cannot promise we won't have another downtime like this, but if it does happen again, we will be able to track down the source a little easier.
 
With that in mind, I realize that some of you lost out on your subscriptions and would like some compensation for that. We are going to be extending all existing subscriptions by 2 weeks. We are also going to enable all advanced features (within the program manager only) for non-advanced members for a period of 1 week. Hopefully this makes up for some of the downtime that we have had.
 
Hope this explanation made a bit of sense to you.
 
Welcome back!
- Neocodex Administration

#2 Amethyst

Amethyst
  • I swallowed a rock


  • 2,716 posts


Users Awards

Posted 10 September 2016 - 06:37 PM

"With that in mind, I realize that some of you lost out on your subscriptions and would like some compensation for that. We are going to be extending all existing subscriptions by 2 weeks. We are also going to enable all advanced features (within the program manager only) for non-advanced members for a period of 1 week. Hopefully this makes up for some of the downtime that we have had."

Codex staff best staff omg. That's incredibly kind. 



#3 Strategist

Strategist
  • Sadmin


  • 9,489 posts


Users Awards

Posted 10 September 2016 - 07:24 PM

Great to see you guys are still able to get on top of it all in times of need :D Cheers for working hard to get the site back up and running, somewhat, more smoothly :) Let's hope the issue stays away. 

 

Also, enjoy your free advanced guys!!! Such an awesome thing to do.



#4 Kaddict

Kaddict
  • 1,654 posts


Users Awards

Posted 10 September 2016 - 07:44 PM

For a minute I was thinking "Cool, I get to try out some advanced shit" and then I realized... "oh, I have had that advanced shit for a year or two now"

 

Anyway. you guys are awesome for working to figure this out, and for hooking everyone up for what was really just a minor inconvenience. 



#5 Amethyst

Amethyst
  • I swallowed a rock


  • 2,716 posts


Users Awards

Posted 10 September 2016 - 08:44 PM

For a minute I was thinking "Cool, I get to try out some advanced shit" and then I realized... "oh, I have had that advanced shit for a year or two now"

 

Anyway. you guys are awesome for working to figure this out, and for hooking everyone up for what was really just a minor inconvenience. 

 

I actually forgot I had access to that stuff too 

 

oops



#6 Pryanka

Pryanka
  • 232 posts


Users Awards

Posted 10 September 2016 - 08:48 PM

You guys are definitely going above and beyond to compensate for the downtime!

I'm glad everything's pretty much up and running for now :D



#7 Pyro699

Pyro699
  • 1,395 posts


Users Awards

Posted 10 September 2016 - 09:21 PM

We are not here to make money. The only reason we have any kind of monetary typed system going on is to help fund the uptime of the server and it has currently let us down. We didn't feel it was right to charge when the service was not providing what we had originally promised. It is nothing out of our pocket to give back to the community that has helped this place become what it is today.

 

If you guys can think of other things to try and make this place better, please let us know and we will see what we can do :)

 

And before you ask Drak - No, we will not bring back the negative rep system.



#8 Swar

Swar
  • com muito orgulho, com muito amor <3


  • 9,066 posts


Users Awards

Posted 10 September 2016 - 09:23 PM

We are not here to make money. The only reason we have any kind of monetary typed system going on is to help fund the uptime of the server and it has currently let us down. We didn't feel it was right to charge when the service was not providing what we had originally promised. It is nothing out of our pocket to give back to the community that has helped this place become what it is today.

 

:heart:



#9 Romy

Romy
  • Ableist Neocodex Elite Four Member



  • 4,738 posts


Users Awards

Posted 11 September 2016 - 12:04 AM

Is it possible someone specifically targeted Neocodex's databse @Pyro699?

 

Be it for their own use (another autobuyer) or maliciously?



#10 ShadowLink64

ShadowLink64
  • 16,713 posts


Users Awards

Posted 11 September 2016 - 08:27 AM

We didn't see any indication of that happening. What we did see a lot of were bots visiting our site to try and harvest content from the forums, itemdb, etc. We banned quite a few yesterday, which should reduce the overall traffic and put less of a burden on the server.

#11 Trapezeo

Trapezeo

  • 728 posts


Users Awards

Posted 11 September 2016 - 08:52 AM

Are the premium features for the program manager automatically started or do I need to do anything to get them?



#12 KyloRen

KyloRen
  • Snoke says I'm special.



  • 5,103 posts


Users Awards

Posted 11 September 2016 - 09:04 AM

We didn't see any indication of that happening. What we did see a lot of were bots visiting our site to try and harvest content from the forums, itemdb, etc. We banned quite a few yesterday, which should reduce the overall traffic and put less of a burden on the server.

How exactly would harvesting that kind of data/content help them (as in the bot controllers/owners/whoever needed the data/ect)? 

 

We are not here to make money. The only reason we have any kind of monetary typed system going on is to help fund the uptime of the server and it has currently let us down. We didn't feel it was right to charge when the service was not providing what we had originally promised. It is nothing out of our pocket to give back to the community that has helped this place become what it is today.

 

If you guys can think of other things to try and make this place better, please let us know and we will see what we can do :)

 

And before you ask Drak - No, we will not bring back the negative rep system.

What about neutral rep? Where you don't give a plus or a minus, it's sort of like giving an "Eh" in rep. 



#13 Romy

Romy
  • Ableist Neocodex Elite Four Member



  • 4,738 posts


Users Awards

Posted 11 September 2016 - 10:28 AM

How exactly would harvesting that kind of data/content help them (as in the bot controllers/owners/whoever needed the data/ect)? 

Despite how slow neo is, I think out DB is still the most maintained.

 

As far as forum content, I have no idea.



#14 UnicornSoul

UnicornSoul
  • I ♥ U Kyra Jade


  • 1,745 posts


Users Awards

Posted 11 September 2016 - 11:02 AM

You guys are awesome!!!! Thank you for all of your hard work!!

#15 Pyro699

Pyro699
  • 1,395 posts


Users Awards

Posted 11 September 2016 - 12:40 PM

Are the premium features for the program manager automatically started or do I need to do anything to get them?

It was as simple as me flipping a variable on the server. There is nothing you need to do for the manager to adapt the new "settings" other than to log out and log back in.

 

The only difference should be that there are inputs that were greyed out before - that are now enabled.



#16 Verge

Verge
  • 41 posts


Users Awards

Posted 11 September 2016 - 02:55 PM

Has there been any recent development upgrade on the site? Maybe it's a case of you something got fixed and broke something else scenario. 

Another thing is you might want to check for the logs. That should give you an idea on whats going on. 

And also might wanna check if the server had some automatic updates done. Sometimes those screw things up to and would require some hotfixes.

 

You guys are awesome! Keep up the good work. I know you'll be able to fix it.



#17 Pyro699

Pyro699
  • 1,395 posts


Users Awards

Posted 12 September 2016 - 01:47 PM

Has there been any recent development upgrade on the site? Maybe it's a case of you something got fixed and broke something else scenario. 

Another thing is you might want to check for the logs. That should give you an idea on whats going on. 

And also might wanna check if the server had some automatic updates done. Sometimes those screw things up to and would require some hotfixes.

 

You guys are awesome! Keep up the good work. I know you'll be able to fix it.

 

a) No we have not done any upgrades or changes to the software.

b) We are knowledgeable enough to know what logs to look in. This included us paying close attention to the server access logs to see who was connecting from what ip and with what useragent. It was must more complicated than we originally would have though.
c) with where we are running our servers, we are in control of everything - there are no automatic updates that could have been applied. But I do agree with you that autoupdates suck and break a lot... Most recently windows 10 autoupdated and broke my virtualbox setups - great times.

 

We are planning some server fixes and updates that will hopefully reduce the possibility of this happening again.



#18 ortin

ortin
  • I'm so l33k

  • 5,908 posts


Users Awards

Posted 12 September 2016 - 06:10 PM

What about neutral rep? Where you don't give a plus or a minus, it's sort of like giving an "Eh" in rep. 

So, not giving any rep at all?



#19 KyloRen

KyloRen
  • Snoke says I'm special.



  • 5,103 posts


Users Awards

Posted 12 September 2016 - 06:23 PM

So, not giving any rep at all?

Exactly. You feel good because there's a 1 by your post, then you realize it's grey so it means nothing and no one actually gave you any rep at all, so you feel a bit sad. 



#20 Amethyst

Amethyst
  • I swallowed a rock


  • 2,716 posts


Users Awards

Posted 12 September 2016 - 07:03 PM

Exactly. You feel good because there's a 1 by your post, then you realize it's grey so it means nothing and no one actually gave you any rep at all, so you feel a bit sad. 

 

That's beautifully evil. 

And passive aggressive as fuck.

I love it. 



#21 KyloRen

KyloRen
  • Snoke says I'm special.



  • 5,103 posts


Users Awards

Posted 12 September 2016 - 07:38 PM

That's beautifully evil. 

And passive aggressive as fuck.

I love it. 

It's who I am. It's who I am.  :devil:

Basically it's a way to make someone feel bad that their post wasn't good enough for a positive rep, without needing to bring back negative rep. 



#22 Junjie

Junjie
  • Hi there!


  • 2,169 posts


Users Awards

Posted 12 September 2016 - 11:44 PM

It's who I am. It's who I am.  :devil:
Basically it's a way to make someone feel bad that their post wasn't good enough for a positive rep, without needing to bring back negative rep.


Oh my god

#23 Amethyst

Amethyst
  • I swallowed a rock


  • 2,716 posts


Users Awards

Posted 13 September 2016 - 05:43 AM

It's who I am. It's who I am.  :devil:

Basically it's a way to make someone feel bad that their post wasn't good enough for a positive rep, without needing to bring back negative rep. 

 

I need this feature in my life.



#24 renegade

renegade
  • 55 posts


Users Awards

Posted 03 October 2016 - 12:13 PM

Is this still an ongoing issue? I can't get the program manager to start. The log says this:

 

 

Traceback (most recent call last):

  File "pre_loader.py", line 54, in <module>
  File "urllib2.pyc", line 126, in urlopen
  File "urllib2.pyc", line 394, in open
  File "urllib2.pyc", line 412, in _open
  File "urllib2.pyc", line 372, in _call_chain
  File "urllib2.pyc", line 1199, in http_open
  File "urllib2.pyc", line 1174, in do_open
urllib2.URLError: <urlopen error [Errno 11001] getaddrinfo failed>

 

I've tried uninstalling and reinstalling to no avail. 



#25 Pyro699

Pyro699
  • 1,395 posts


Users Awards

Posted 03 October 2016 - 02:26 PM

Is this still an ongoing issue? I can't get the program manager to start. The log says this:

 

 

I've tried uninstalling and reinstalling to no avail. 

 

The downtime happened several weeks ago... This is something else.

 

Is it still happening? Are you behind a firewall? Can you access neocodex.us at the same time you get the error?




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users