• Nicky
  • Administrator
  • Offline
  • From: Troy, NY, US
  • Registered: 2009-08-21
  • Posts: 167
  • User Karma: 11

Topic: Massive Server Issues :(

After my 9pm nap tonight, I tried to log it. The log didn't seem to work. I refreshed and the whole site was down.

Somehow, we lost all the files for just TryPolyphasic. The directory structure was still there, files gone.

Fortunately, our host (Linode) runs backups daily.

Or so I though; the new backup service has been down for 12 days and is still not back online.

Thank god, we had a backup of most of the actual site that was fairly recent. (And none of the blog/forum data was touched)

What we don't have is the custom code I'd written for the forums and the blogs.

I'll do my best to get things back to 100% asap.

My rough plan is as follows:

  • Forum functioning - done

  • Blog posting / commenting - done

  • Try Polyphasic bar

  • Blog theme - done

  • New blog registration

This should bring us back to where we were before.

Please let me know if you come across any problems I miss as I go and share any thoughts/suggestions/concerns.

I will be looking for improved backup solutions in the future to be sure this does not happen again.

-Nicky

Challenge is the Opportunity for Greatness
tumbledesign.com

Thumbs up Thumbs down

  • From: Borlange, SE
  • Registered: 2009-09-06
  • Posts: 219
  • User Karma: -4

Re: Massive Server Issues :(

Ouch, that must hurt, losing all your hard work =p Well, at least you got something to do during the nights to keep you awake now wink

Thumbs up Thumbs down

  • From: Wellington, NZ
  • Registered: 2009-08-30
  • Posts: 68
  • User Karma: 4

Re: Massive Server Issues :(

Nicky wrote:

After my 9pm nap tonight, I tried to log it. The log didn't seem to work. I refreshed and the whole site was down.

Somehow, we lost all the files for just TryPolyphasic. The directory structure was still there, files gone.

Fortunately, our host (Linode) runs backups daily.

Or so I though; the new backup service has been down for 12 days and is still not back online.

Crikey, that isn't at all good! Perhaps it might be an idea to consider another backup solution?

How big is everything all up, and how much does it change? Provided it's not too bandwidth-intensive, I'd be quite happy to keep nightly backups for the entire site & databases on one of my servers.

Thank god, we had a backup of most of the actual site that was fairly recent. (And none of the blog/forum data was touched)

smile

What we don't have is the custom code I'd written for the forums and the blogs.

hmm ... what happened to it? Did you not have a local copy?

I'll do my best to get things back to 100% asap.

My rough plan is as follows:

  • Forum functioning - done

  • Blog posting / commenting - done

  • Try Polyphasic bar

  • Blog theme - done

  • New blog registration

This should bring us back to where we were before.

Please let me know if you come across any problems I miss as I go and share any thoughts/suggestions/concerns.

/search/all on the forums is depressingly missing - that's the feature I use the most!


I will be looking for improved backup solutions in the future to be sure this does not happen again.

See above - I am happy to provide a backup service.

And finally, a huge thanks to Nicky for making this site what it is - without all your efforts this wouldn't exist. I'm sure I speak for everyone when I say it's much appreciated.

"...people will sleep extra if they can, just like they'll have a second helping of pie if you let them. The existence of a second piece of pie on your plate is hardly an indicator that you're starving!" --PD

Erayd's Website

Thumbs up +1 Thumbs down

  • From: Borlange, SE
  • Registered: 2009-09-06
  • Posts: 219
  • User Karma: -4

Re: Massive Server Issues :(

+1 to last post, I got a 24/7 server too, although it's in europe so I dunno if it would be that useful =p

I see you changed the forum theme, too bad it's the wrong color xP

Thumbs up Thumbs down

  • From: Arlington Heights, IL, US
  • Registered: 2009-08-26
  • Posts: 228
  • User Karma: 2

Re: Massive Server Issues :(

Wow, that really sucks. Lately I've been using Dropbox heavily both to synchronize files between my computers and to keep them backed up...

Lavalamp's Official Adaptation Guide

Thumbs up Thumbs down

  • Nicky
  • Administrator
  • Offline
  • From: Troy, NY, US
  • Registered: 2009-08-21
  • Posts: 167
  • User Karma: 11

Re: Massive Server Issues :(

hmm ... what happened to it? Did you not have a local copy?

Yep, didn't feel like installing MU, Forums, etc locally so I did the connecting bits remotely. Rookie mistake.

How big is everything all up, and how much does it change? Provided it's not too bandwidth-intensive, I'd be quite happy to keep nightly backups for the entire site & databases on one of my servers.

Yesterday I setup rdiff to backup the entire server nightly to a separate machine. That should be good enough, right? I have 3 servers so I may backup each one on all the others. They're at the same datacenter but it wouldn't be tough to transfer them to separate ones.

/search/all on the forums is depressingly missing - that's the feature I use the most!

It's back!

And finally, a huge thanks to Nicky for making this site what it is - without all your efforts this wouldn't exist. I'm sure I speak for everyone when I say it's much appreciated.

Thanks! It's been a lot of fun and I love seeing the community develop.

I see you changed the forum theme, too bad it's the wrong color xP

Which color is wrong? It's a bit different than last time but I kinda like it better the way I made it this time around.

Lately I've been using Dropbox heavily both to synchronize files between my computers and to keep them backed up...

Dropbox is great; I'm also just starting a tryout of Backblaze for my personal backup.

Thanks guys!
-Nicky

Challenge is the Opportunity for Greatness
tumbledesign.com

Thumbs up Thumbs down

  • From: Borlange, SE
  • Registered: 2009-09-06
  • Posts: 219
  • User Karma: -4

Re: Massive Server Issues :(

I would prefer if it were the same color as everything else, the 'bar' and the blogs are light beige, while the forums are...grey-brown. A color I don't seem to like.

I think there were this problem before, but I have more issues with it now. It seems like I'm browsing to effective, I can go to a blog, read the new comments and move on to the next blog before the new-messages tag is gone, resulting in that it says that I haven't read it. The same with the now post star.

Thumbs up Thumbs down

  • Nicky
  • Administrator
  • Offline
  • From: Troy, NY, US
  • Registered: 2009-08-21
  • Posts: 167
  • User Karma: 11

Re: Massive Server Issues :(

Hatten,

So you are saying that the blog doesn't register your read fast enough? I think I have it waiting 5 seconds now before marking a blog as read.

Is anyone else experiencing the forums 'read' icons behaving oddly?

Right after reading something, a post is still marked unread. A few minutes later it's marked read. I'll have to look into that.

-Nicky

Challenge is the Opportunity for Greatness
tumbledesign.com

Thumbs up Thumbs down

  • asd
  • Member
  • Offline
  • Registered: 2010-01-20
  • Posts: 35
  • User Karma: 0

Re: Massive Server Issues :(

Hey if there is any way we can help..... just let us/me know! smile

Thumbs up Thumbs down

  • From: Wellington, NZ
  • Registered: 2009-08-30
  • Posts: 68
  • User Karma: 4

Re: Massive Server Issues :(

asd wrote:

Hey if there is any way we can help..... just let us/me know! smile

Ditto - I'm also happy to help with the site if there's anything that needs doing.

"...people will sleep extra if they can, just like they'll have a second helping of pie if you let them. The existence of a second piece of pie on your plate is hardly an indicator that you're starving!" --PD

Erayd's Website

Thumbs up Thumbs down

  • From: Borlange, SE
  • Registered: 2009-09-06
  • Posts: 219
  • User Karma: -4

Re: Massive Server Issues :(

@Nicky, it's certainly behaving odd for me, for one, I have an additional icon on the front page at the side of some post. A black box, indicating that the picture is missing. If I go into that blog entry I get a permanent speech bubble that doesn't fade away. Then when refresh I get a star that disappears, without fading.
That was not a problem before the server crash.

I'm surfing with the keyboard only, and I can easily go to a blog, go to the bottom, read a short message, go to the top, and go to the next blog before 5 seconds have passed. Right now I have a black box on one blog because one of my own messages are unread, I went out of there without waiting 5 seconds. big_smile

EDIT: that black box behaves rather oddly, but now that I went into the blog that I commented on, there was a speech bubble that faded away. I had to wait more than 5 seconds before leaving though if I wanted to remove the black box.

Last edited by hatten (2010-02-09 15:49:25)

Thumbs up Thumbs down