About the downtime tonight... Democratic Underground failed you, and we are sorry.

[Skinner]

Democratic Underground failed you tonight. We should have been able to provide access to the site, but we did not. I want you to know that the DU Administrators believe that repeated extended downtime like this is not acceptable. We are sorry.

Here's what happened: We lost power AGAIN.

This wasn't supposed to happen. After we had this problem during the VP debate, we had a conference call with our web hosting company to discuss our power situation. They ran power analysis to figure out how much power we were using, and how much we would need to handle a big spike like tonight. They split the servers onto separate circuits to spread the load around this time. But we lost power again anyway. Apparently this time was a different circuit. (For the DU computer geeks: It was the master switch.) But even if it was a different circuit, it shouldn't have happened.

You all have been extremely patient and understanding. Thank you for that. I think blowing a circuit once is understandable. But having it happen *twice* is simply not acceptable.

We are going to do everything we can to make sure this doesn't happen again. We're going to be discussing this issue with our web host tomorrow. If we have to camp a technician next to our servers for the next three weeks, we'll do it.

Thank you for your understanding. Again, we apologize.

David Allen
DU Administrator

[Bluebear]

Fuggedabboudit. Just don't do it 11/4 :)

[tavalon]

58. Are you kidding?

That's going to be the hardest day ever for DU, ever. We hope it won't go down but gosh, that's gonna be a hard one. Actually I didn't realize it was down. I'm assuming it was during the debate and I was busy...............watching the debate.

[wildbilln864]

1. shit happens dude...

we're past it! Thanks for everything else you do. :hi:

[spindrifter]

2. We still love you.

[dubeskin]

3. No problem! All that matters is that it's back up!

Thanks so much for all the hard work you guys are doing to keep this site so wonderful!

Just remember, you only have to survive a few more weeks, and it's ALL going to be over :D

[TACstrat]

9. Should have blamed it on the republicans - nt

[johnfunk]

84. Who wired your server farm? Enron?

Must have been a "rolling blackout."

[madaboutharry]

4. I think you are a being a little hard

on yourself. If you ever needed to beat yourself up it would have been over that stupid "recommended" experiment a few weeks ago. :)

[Liberal_in_LA]

5. Next time DU goes down will be Nov. 4th, 8pm ;-)

Just kidding. :hug:

[MiniMe]

51. Probably more about 11pm or midnight that night

Have to wait for the left coast. But I do think DU will be overwhelmed that night.

[swag]

6. Servers have downtime. Everybody should understand that.

No apologies necessary.

[NMDemDist2]

7. Dude, maybe time to shop hosts??

you guys musta been freaking and peaking even worse that we were.

but please PLEASE make sure there's a tech ONSITE on election night.

I just can't do it without you!

seriously.

[rateyes]

8. We love you Skinner.

No problem. :hi:

[malaise]

10. Thanks for explaining the problem

:hi:

[Ichingcarpenter]

11. I know you don't believe in conspiracies

But Honestly you can trace this stuff.

[last1standing]

12. Don't be ridiculous.

Servers go down when they get overwhelmed. That's just how it goes. This site is generally up 24/7/365 and on the occasional crash we can manage for the few minutes it takes to reboot.

[kansasblue]

13. badge of honor! Load's too heavy! sweet Grow DU

[LiberalHeart]

14. Who's your web host voting for?

[kansasblue]

19. hmmmmmmmmmmmmm good questions

check their political contributions

:-)

[Ichingcarpenter]

36. Geeee........... good question? Something stinks

Maybe it will go to the 9-11 forum

[kansasblue]

40. That was humor.

[LiberalHeart]

41. Don't you love it when you have to explain your jokes?

[Ichingcarpenter]

44. Sorry, the data doesn't support it as a joke

You can do analysis on sites on line with traffic in real time.

It wasn't that heavy to support it.

[LiberalHeart]

46. Um...thus, the joke that it was the web host disconnecting the power, dear.

[Ichingcarpenter]

54. Oh, that's Ok...... they shut down and locked my other analysis thread that I got from Europe

So, everything is OK now.

[cboy4]

66. It's good to see you're just as irritating as ever.

:thumbsdown:

[Ichingcarpenter]

70. And you ..... being the totality of your

arrogance on your youthful ignorance....

You have never won an intellectual argument here
and you have exceeded your youthful muscle bound stupidity
that we have all come to enjoy and observed in humor
from the post you do.


Listen, I know you lift weights, you try to say
that in your avatar. What type of insure
bastard would place that in their avatar
except an insure adolencent?...... I lift weights!
I'm cute!

The only wieght here you lift
is your own muscle bound ignorance is
that you have proved you are not
well read or informed

Now, go get an education
You never give a shread of intellectual
credibly except your emotional opinion.

[cboy4]

72. That's fascinating. Me flying an avatar of Metallica

bassist Jason Newsted equates to me lifting weights?

Gee, yet another thing Inchingcarpenter is wrong about. I need a scorecard.



You must be intoxicated 24/7.

It can be the only explanation.

[ladjf]

15. Running normally now. Don't sweat it.

[Old and In the Way]

Small price to pay fora great site.

I think DU hosts the best real time political conversation on the innertubes. Obviously, there are lots of people who want in when an event like these debates occur.

[mohinoaklawnillinois]

16. Skinner it's okay.

:hug: :hug:

[Trajan]

17. Those multiple uPs start glowing like hot coals .... Im sure ...

Feces occurs .... You guys STILL rock ....

[PufPuf23]

18. I appreciate this fine place

Both outages I just thought DU was too popular -- which is the essential technical problem.

Thank you

[2speak]

20. Quit tripping on the power cord....Sheesh

[lpbk2713]

21. Um ... do you get service from the lowest bidder?

That might be the problem.

:eyes:

[OneGrassRoot]

22. You guys are awesome. I wish all business people had the same...

integrity. Life would be so much easier and more pleasant.

:)

Your efforts are always much appreciated!

[Subdivisions]

23. Thanks, Skinner! =) n/t

[TACstrat]

24. No sweat ... nt

[lonestarnot]

25. We still love you Skinner!

:loveya:

[Stinky The Clown]

26. Psssst. Skinner ........ a rat .... in a cage .... hooked to a small generator

Wow ..... we cooked a circuit!

Cool!


We be hot ...... sssssssssssssssssssst!

[fulllib]

27. This website has never failed me!

What this website does for Dems and libs goes beyond the failure tonight.

Heck, I should have had my head out of the computer to watch the debate anyway!

[Dem2theMax]

28. Suggestion for election eve.

Can you do what you did last time? I think it was 'registered members only' had access to DU that night. Or maybe set it up up so if you have to go to Defcon 4, then 'registered members only' kicks in?
Just a suggestion. I hate being locked out of DU. I'd rather be locked out of my house! LOL.

I HAVE to be able to get on DU on election night. I'll go out of my mind if I can't be here. Just plain :crazy:

;)

[Pavulon]

47. akamai and other clouds

and others offer a cloud solution. No one server or in a bigger system datacenter failure should take a site offline.

Not my current line of work but power problems in a NOC tend to indicate basic design problems. Like having roof leaks or foundation cracks.

Basically web content can be distributed pretty easily. Hopefully they will continue the move to cloud solutions that unbind applications (web, db) from hardware.

I am sure these are out there but are cost prohibitive.

[supernova]

29. Was at a debate party

without my computer! :wow:

So, didn't know about it til I got home.

Sorry to hear that. But we're going now!

I hope you can get it straightened out with your hosting service. We're gonna be looking for a marathon on Election day!!!! :D

:hug: for Admins

[lostnfound]

30. We still love you Skinner. Life goes on..

:grouphug: :yourock:

[greyl]

31. Blow it once, shame on...

Peance freance!

[mooseprime]

32. DU is great and God bless you for all you do. n/t

[Marrah_G]

33. 50 Lashes with a wet noodle for you!

[texpatriot2004]

34. Thanks for letting us know what happened and Thanks for your

dedication to DU.

[UpInArms]

35. we all knew that Obama was winning the debate

and that DU would eventually catch up with it

:hi:

I{heart}DU

You guys rock!

:yourock:

[applegrove]

37. We got your back dude. You do your best. We know that.

[UTUSN]

38. DU never fails me except for not chasing down & tombstoning my stalkers & flamers!1 n/t

[4lbs]

39. It's ok. We're all under a Bush-McCain energy plan right now and we should expect some "blackouts"

:D

That should change in about a month.

[JimWis]

42. No sweat. This place is great.

[yodermon]

43. How many times was the word "FUCK" uttered in DU HQ tonite?

;)

[ReadTomPaine]

45. Thank you for saying this, but you shouldn't have to take the heat for someone else's incompetence.

[Canuckistanian]

48. Don't sweat it

I know firsthand how vulnerable servers are during extreme loads.

Sometimes, you can't anticipate.

Thanks for the explanation.

K&R.

[yodoobo]

49. Power? As in electricity?

What the heck is your ISP doing? Are they saying power as a euphemism for something else? Bandwidth I understand, database issues - you bet, memory corruption - oh yea.

Are you sure there isn't some disgruntled kicking the circuit breaker at your colo? I know those places are secured, but there are still people running around in them.

I'm seriously scratching my head. As a 20 year network engineer veteran, I have never, and I mean never ever heard of an AC cirucit breaker popping solely as a result of massive network traffic. In fact network devices demonstrate almost very little increase in power consumption under load. (power is always being "transmitted" data circuits, with traffic its just modulated differently)

Honestly, I'd give your an ISP a through 3rd degree over that explanation.

Nonetheless, no apology needed. Shit happens

[FatDave]

83. You are absolutely correct.

I posted similar below. And what about a UPS? You've got a hosting company that just plugs their servers straight into a wall? I've got my fucking desktop PC on a UPS for cryin' out loud!

[ChromeFoundry]

90. The Hosting Provider has this statement on their site...

"The AC electrical system in the building is highly scalable and serviced by redundant 50 KVA online UPS systems with an on-site, automatic diesel generator backup."

And I just installed a jet engine on a Yugo!

I have never heard of traffic causing a circuit to throw. The power supplies on the servers and storage arrays have MAX and In-Rush ratings, so unless they overloaded a single circuit... It's not DUs fault that the hosting facility cannot perform simple addition.

[yodoobo]

93. Yep. you either have power problems, or you don't

Yea that ISP is blowing some crazy smoke.

If you have power problems at peak load, you'll also have power problems at no load - and everywhere in between. In fact most electrical issues I've seen aren't even related to load at all, but to improper grounding.

Breaking it down very very simply..Network devices transmit continuously. They might be transmitting a crap load of threads discussion the election, or they might be transmitting silence cause nobody is using the site. Either way, they are burning power to transmit. This is true of just about any network transmission medium i can think from electrical to optical based to wireless. Probably the only exception would be IP over avian :) http://en.wikipedia.org/wiki/IP_over_Avian_Carriers

As I said early, I have never ever heard of a power problem directly related to bandwidth usage. Never, and I've seen it all.

In fact, Id say it was a database engine blowing out its memory from too many open connections or possibly a load-balancer gone nuts from what i saw of the end-user output.

Either best of luck and don't let the ISP lie to you.

[Greyhound]

87. I guess this is what happens when an industry fires most of the people that know what they're

doing because they are making a decent living. I'm just sick over the dramatic decline of IT competence, it seems nobody knows how to code or understand how computers work anymore, everything is tools and certs and working for less than the other guy.

[yodoobo]

94. Roger that

Sickening what they have done to this industry.

[maddezmom]

50. I actually got some stuff done while watching the debate

All laundry is sorted and folded and hubby has a full on supply of undies, t-shirts and work socks. Ah life is grand. :D

[JackBeck]

52. Truth be told, it really pissed me off tonight.

Last year during a DU drive, we were promised a brand new DU in 2008 (Spring?) that wouldn't crash during this current election cycle.

Other rollouts were promised during that drive, none explicitly detailed, but if the rating system was the only thing that was planned, I hope the admins have other things up their sleeves before the end of the year. Y'all told us there were great things to expect in 2008, yet haven't made any of them a reality.

Two strikes now. How can I be guaranteed this place will be up and running with not one moment of downtime on election night? Other progressive sites are able to manage the traffic. Why the fuck is wrong with this place?

It's the only place online that consistantly offers excuses instead of solutions.

What are you guys doing to foster an environment where this website can be trusted as a place to congregate as a community during a very important election?

[tavalon]

59. This one wasn't DU's fault

Anyway, if the other sites are up and running, what's the problem? Go there. It will make DU come up that much quicker if we have less traffic here, unless you are saying that this is the best site for election night coverage, bar none (I would, btw, agree with that statement). If you are saying that it's the best, then don't bitch. Do you have any idea about the exponential growth this place has gone through? The slashdot phenomenon has happened here too many times to count and each time, the people who run DU have stepped up their servers and their service to meet it. They have gone above and beyond so many times that we're just plain spoiled. At least I know it.! I can't even go to any other discussion sites because the features here are so great and yet, so intuitive.

Two strikes? Does that mean three strikes and they're out?

[JackBeck]

63. delete

[JackBeck]

64. Context.

We were told last year during fundraising drives that DU was raising money to make sure we wouldn't crash during the election cycle.

Well, we've crashed twice now. We were told that DU would be prepared for this. Months in advance. And they still crashed. Twice.

Other progressive sites have figured out how to stay live. Why hasn't DU?

[WritersBlock]

89. You're joking, right? I mean, you can't possibly be serious here, can you?

How much did you donate during the fund raising drive you speak of?

Cos I'll tell ya what, if Skinner will give you your money back right now, I'll match your donation so the site won't be out any money. Then you can go fuck off to some other environment that's been fostered as a trusted place to congregate as a community during this election. If you can find one.

Jesus Christ, some people will bitch about anything.

[upi402]

53. electrical power? get a shunt trip breaker -auto-switche to UPS

http://en.wikipedia.org/wiki/Uninterruptible_power_supply
maybe expensive...

[TahitiNut]

55. No big deal. DUers are short-circuiting and blowing circuits all the time.

:dunce: :shrug: Fresh gerbils for the generator wheels!

[Contrary1]

56. Skinner, I regret to inform you that I, who was previously undecided,

will not be casting my vote for you for president.

You can only blame yourself, and your poorly run campaign.

[depakid]

57. Actually, I'm impressed that you were able to get the site up and running so fast

Props for that!

[Generic Other]

60. I always put on my tinfoil hat and blame the vast rightwing conspiracy

Thanks for all you do! It's been a wild 8 years, no?

[Swamp Rat]

61. GEAUXBAMA!!!

:patriot:

[nadinbrzezinski]

62. Master Switch... oh my....

<-------- Yes, geekese spoken here

the gods didn't like you...

Shit happens...

[ismnotwasm]

65. It's ok

I had to actually watch the debate instead of reading DU commentary as I was going to. I'm glad I did, Obama kicked ass, and McCain looked and sounded like an evil little old bastard.

[ferrferr]

67. I was going a little crazy, and wanted to cry. ;)

[JDPriestly]

68. Hey! I miss DU when it is down, but I'm happy that so many

people are on the site. I sometimes go to another site that goes out in times of intense interest in politics. It's one of those small inconveniences in life that make you appreciate all the more what you have when you have it.

[notadmblnd]

69. Power Distribution Units aren't complicated things,

basically just bigger (much bigger) versions of the surge protectors we use at home I'm not saying it's impossible, but it's Doesn't make sense. Do you know if DU is sharing space in the rack with another client? If so, are other clients also experiencing the same outages? I would suggest that you talk to your hosting people and suggest replacing the unit with one that has built in surge protection. They also have models with meters so power usage can be monitored along with alarms that will alert the customer (your vendor) if the load is such that it may fail. CDW can have one to them the next day. The cost? Around 500.00. I would also suggest that on election day, you request that they provide on site monitoring of the rack for the entire day.

On edit: UPS units aren't that expensive either: I would suggest this be done at your vendor's expense.

• UPS
• 6 Outlets
• 750VA
• 500 Watt
• Serial/USB
Specifications are provided by the manufacturer. Refer to the manufacturer for an explanation of the print speed and other ratings.


Performance power protection for servers, and voice and data networks
APC Smart-UPS® protects your data by supplying reliable, network-grade power in traditional Tower. Award winning APC Smart-UPS® is the perfect UPS for protecting business critical fileservers (Intel or UNIX based), minicomputers, network switches and hubs, Point of Sale, retail/bank back office and ATM's.

With included PowerChute® management software for servers and workstations, IT administrators can provide safe system shutdown and advanced UPS management (All major operating systems supported). Connectivity is through serial or USB port. Additional manageability is available through the SmartSlot, an internal accessory slot that allows you to install optional accessories to enhance the performance of your UPS. Network connection with Web browser management and/or environmental monitoring, serial port expansion, and out-of-band management options are available. With pure sine-wave output ensuring compatibility with all connected devices, Intelligent Battery Management ensuring a highly available UPS and an advanced 16 segment bar graph display ensuring information and management, the Smart-UPS® is a UPS you can count on. Price: $294.99


PDU
Features:
Power conditioning
Temperature-compensated battery charging
Intelligent Battery Management
Automatic restart of loads after UPS shutdown
Automatic self-test
Boost and Trim Automatic Voltage Regulation (AVR)
Cold-start capable
Disconnected battery notification
Generator Compatible
Audible Alarms
LED status indicators
Disconnected battery notification
Serial Connectivity
USB Connectivity
Hot-swappable batteries
Predictive failure notification
Resettable circuit breakers
Adjustable voltage-transfer points
Adjustable voltage sensitivity
Field-replaceable power distribution panel

• Rack PDU
• Metered
• Zero U
• 30A
• 208V with (20)C13 & (4)C19 outlets with NEMA L6-30P input connector
Specifications are provided by the manufacturer. Refer to the manufacturer for an explanation of the print speed and other ratings.


Power distribution units that monitor the aggregate power consumption of connected equipment
APC's Metered Rack Power Distribution Units (PDUs) provide power distribution as well as equips on-site installers with the ability to monitor the aggregate current draw as equipment is connected to the unit. These Metered Rack PDUs also have alarm thresholds that when exceeded alert users of potential problems. These features help eliminate the possibility of an overloaded circuit.

Features:
Alarm Thresholds
Flash Upgradeable
Load indicator LED
Local Current Monitoring Display
Network Management
Single input power source
Toolless Mounting Capabilities
Vertical Mounting Capabilities

Includes:
Installation Guide
Rack Mounting Brackets
Toolless Mounting Kit

Product Pricing
Price: $459.99
Lease from $14.28/month
Qty:

http://www.cdw.com/shop/products/default.aspx?EDC=736158&cm_sp=Product-_-Overview-_-Main+Tab

[LibertyorDeath]

71. Everyone blows a circuit once and a while

Cheers

For all your hard work! :toast:

[lligrd]

73. Problem With Being Popular nt

[snot]

74. thank you, skinner.

[JerseygirlCT]

75. Thanks. I thought it was me at first! nt

[hootinholler]

76. Maybe you can get Joe the Plumber to hook up them tubes right!

:rofl:

Lost the whole rack, eh? Here's hoping it's all straight for the 4th.

-Hoot

[BlueCollar]

77. I'm not buying it...

I've been a member of this site for a long time and DU has never failed me.

Occasionally, there's a glitch and something happens. If I can't log in then I know that you guys are working hard to fix the glitch.

You are usually up and running pretty quickly so don't worry about it.

I'm proud to be a part of this community. I've met some good people on here, had a laugh, been entertained and informed. Occasionally a moment of grief when we hear that one of us has passed, and sadness when we hear of tragedy.

DU is still my favorite website. Occasional glitches happen.

No apologies needed. Keep up the good work.

edit: forgot spell check

[DarbyUSMC]

78. Not the end of the world, buddy. Hakuna Matata.

[mainegreen]

79. Hope you find a solution n/t

[raccoon]

80. That's OK, Skinner. After all you do, don't give it a thought. nt

[wryter2000]

81. Stuff happens

No biggie.

[FatDave]

82. How does high traffic affect power consumption?

I don't get it. Makes no sense.

If you've got an underpowered power supply in a server, and you've got it spinning a bunch of drives (though large drive arrays typically have their own dedicated power supply), you might overload the server's power supply. But there's no fucking way that power supply is going to start drawing additional power off of the grid. And even if power did go out, your hosting company doesn't have its servers on UPS's? What kind of rinky-dink outfit is this? They don't seem to know shit. Have a fundraising drive and move to a reputable hosting company.

[lpbk2713]

85. All those ones and zeros cause a lot of friction. And friction creates heat.

:eyes:


















Just in case it's necessary ------ :sarcasm:

[ChromeFoundry]

91. Most servers and storage arrays have redundant power supplies

that are to be fed from separate circuits.

The hosting facility claims to have: "redundant 50 KVA online UPS systems with an on-site, automatic diesel generator backup." - taken from the provider's site

[ailsagirl]

86. Thanks for explaining, Skinner. :)

:grouphug:

[La Lioness Priyanka]

88. well we should have been paying attention to the debate anyway.

so no biggie

[needledriver]

92. Downtime explained

http://www.democraticunderground.com/discuss/duboard.php?az=view_all&address=389x4249036

[Javaman]

95. I think you need to check the political stance of your web hosting company. nt

[vademocrat]

96. At least it made me go to bed at a decent hour!

When I realized that DU was unavailable I called it a day. If I'd gotten on DU who knows how long I would've been up? And then I would've been tired and cranky today...My co-workers thank you!

Seriously, thanks for all that you and the other admins do to keep the site running as well as it <usually> does!

[tularetom]

97. Bastardize the grovelbot

and use the parts for backup servers,

[bagimin]

98. I assumed your bong tipped over on the server

no pro bro.

[bluesmail]

99. Don't know if this is related but I broke down and tried to

contribute to Obama/Biden campaign but wasn't able to last weekend.