Author Topic: ClubArnage server was DOWN this morning :-(  (Read 4192 times)

Offline DJet

  • Server Administrator
  • Administrator
  • Club Arnage Demi God
  • *******
  • Posts: 580
  • serveradmin
    • View Profile
ClubArnage server was DOWN this morning :-(
« on: January 15, 2010, 02:51:47 pm »
The server for this forum was just unavailable for the longest time since
it was installed some 10 years ago (!)

As part of a due diligence New Year cleanup, I have been busy upgrading
the (long) list of programs that makes up the server on which this forum
runs.

Most applications and forums were done last week, and the last thing is
the operating system itself. This requires a reboot, so I sdaved it for
Friday morning, where teh traffic to the server (all sites) is lowest.

I took a deep breath, and rebooted ...

When the server started booting, it reported it couldn#'t find its disk(!)
arrgh!

Then I rebooted it again, by which time it simply went away - and I never
heard from it again, until I had some of my friends in the datacentre locate
the machine and powercycle it. (!)

Once this was done, It actually booted normally, so it is unexplained why it
couldn't find the boot-disk in the first place. Possibly it got confused by
running with a serial console connected (never heard of that before though)

Anyway, the machine came up, but now it turned out that the web-accelerator
proxy (varnish) I run in front of all sites, had been modified at the update
last week, but the upgrade did not restart the proxy - so it wasn't until
todays reboot that I discovered this.

Since I am actually at work, I can't spend too much time messing about with
this, but I got help from my good friend Poul-Henning Kamp, who is in fact
the author of that web-accelerator, and he quickly spotted that the new
version carried a default portnumber with it, as it wasn't defined in my
old config (didn't have to with the old version).

Problem sorted!

However, I still need to upgrade the Operating System "userland" - but that
really should only be a quick reboot. (famous last words)

Sorry for the downtime, but at least we are now upgraded to the newest STABLE
branch of the operating system and all applications.

/Lennart

Offline Perdu

  • CA Veteran
  • Club Arnage God
  • ******
  • Posts: 1914
  • llama's in the basement mixing up the medicine
    • View Profile
Re: ClubArnage server was DOWN this morning :-(
« Reply #1 on: January 15, 2010, 03:05:58 pm »
Lennart

thanks for ALL the work you do for us

I'm glad you are winning (very glad :)  )

bill
"Ha ha you can't a fool me, there ain't a no sanity clause!"

Offline mgmark

  • CA Veteran
  • Club Arnage God
  • ******
  • Posts: 2489
  • Fun is not a straight line.....
    • View Profile
Re: ClubArnage server was DOWN this morning :-(
« Reply #2 on: January 15, 2010, 07:56:22 pm »
Lennart,

Can't pretend to understand how it all works, but many thanks as ever for making it do so!

MG Mark
"If everything seems under control, you're just not going fast enough." Mario Andretti

Offline Barry

  • CA Veteran
  • Club Arnage Master
  • *******
  • Posts: 3381
  • Kick out the jams, motherf*ckers!
    • View Profile
Re: ClubArnage server was DOWN this morning :-(
« Reply #3 on: January 15, 2010, 08:12:54 pm »
All witchcraft as far as I'm concerned.
That makes Lennart the chief wizard!!!

Thanks for the effort you put in to keep CA running smoothly.

Offline Lorry

  • CA Veteran
  • Club Arnage Master
  • *******
  • Posts: 2531
  • I won't join any club that'll have me as a member
    • View Profile
    • Beautiful Kent
Re: ClubArnage server was DOWN this morning :-(
« Reply #4 on: January 15, 2010, 11:05:12 pm »
Thanks.  These things are sent to test us.

My best camera stopped working, so I thought I'd have to spend ages finding the receipt, so I could get it  mended under warranty, so I did a bit of research ont'internet, and found a string of similar problems, and the answer was a reboot.  You forget all these f***ing things are computers
GENTLEMEN  -  Start your livers

For and on behalf of the Kent Kronenberg Owners Club