Welcome to DU! The truly grassroots left-of-center political community where regular people, not algorithms, drive the discussions and set the standards. Join the community: Create a free account Support DU (and get rid of ads!): Become a Star Member Latest Breaking News General Discussion The DU Lounge All Forums Issue Forums Culture Forums Alliance Forums Region Forums Support Forums Help & Search

EarlG

(22,093 posts)
Wed Feb 28, 2024, 08:39 PM Feb 2024

IMPORTANT: Today's update failed and we have had to restore a backup, but we did not lose data

As most of you know, today we attempted to update some of our software to a new version. We first performed the update on our test site, and after observing that everything was running smoothly, we updated the main site. It was at this point that we ran into issues, because the main site slowed to a crawl.

After investigating, we discovered that the latest version of the software involves some significant -- and unanticipated -- technical changes to the way DU works behind the scenes. Unfortunately these changes were not included in the documentation because, as we discovered too late, they would "only affect a handful of users." Yup, we are one of those users. This means we have to resolve this particular issue before we can continue with the update.

In order to bring the site back we had to restore a backup, but the good news is that because the backup was taken right before the update happened, we did not have to go back to an older version, and we were able to recover everything.

We are very sorry about the service interruption which, unfortunately, ended up being for nothing today. The good news is that we are now back to normal. We are back in our original configuration, so things should work correctly as of now. We expect that there will be some initial slowness as the restoration process continues -- you may notice it when first arriving at the site, but it should clear up fairly quickly.

We do still need to perform this software update, but we are now well aware of these specific issues which caused our last attempt to fail. We will not be attempting this update again immediately, but it should happens sometime within the next few weeks, and we are optimistic that our next attempt will be much more successful.

Once again, I apologize for today's downtime. Although we knew that this would be a challenging update, it's rare for us to run into a significant issue like this, so we're really disappointed that we weren't able to get everything squared away as intended today.

I'll update you again when we have more information about our next attempt to perform the update.

Thanks for your patience,

EarlG

78 replies = new reply since forum marked as read
Highlight: NoneDon't highlight anything 5 newestHighlight 5 most recent replies
IMPORTANT: Today's update failed and we have had to restore a backup, but we did not lose data (Original Post) EarlG Feb 2024 OP
No harm in trying and failing. ffr Feb 2024 #1
Thank you and Elad for your great work! Wild blueberry Feb 2024 #2
I'm very sorry the update failed, but glad to see you back, and without any data lost. highplainsdem Feb 2024 #3
No problem. A suggestion perhaps. Schedule your updates late at night. OAITW r.2.0 Feb 2024 #4
They addressed that in their notice yesterday rurallib Feb 2024 #8
Did not read closely, my bad. Suggestion removed. OAITW r.2.0 Feb 2024 #12
Any way to blame it on Mike Johnson and get the RNC to pay for our therapy? Hermit-The-Prog Feb 2024 #5
Ha! Or at least the senseandsensibility Feb 2024 #17
It's clearly Joe Biden's fault Ray Bruns Feb 2024 #72
Well, it's good you chose a slow news day. jls4561 Feb 2024 #6
At least you picked a slow news day. NT mahatmakanejeeves Feb 2024 #7
the site being down was way too reminiscent of 2016. SleeplessinSoCal Feb 2024 #16
Thanks for all that you do! Think. Again. Feb 2024 #9
Got to say - been there, done that rurallib Feb 2024 #10
My sentiments exactly. Gidney N Cloyd Feb 2024 #42
Good luck! AltairIV Feb 2024 #11
Create a restore point DoBW Feb 2024 #13
Never let a computer know you are depending on it. n/t dobleremolque Feb 2024 #14
Thanks for the update and I am ready for whenever your next shot at it is-n/t marked50 Feb 2024 #15
Keep up the good work. It will happen! GreenWave Feb 2024 #18
No apology needed, but thanks just the same! ShazzieB Feb 2024 #19
Thank you for all you do! Hope22 Feb 2024 #20
The joys of MIS... dlk Feb 2024 #21
I'm just happy that you are up and running. live love laugh Feb 2024 #22
Bummer... 2naSalit Feb 2024 #23
Failure is important. madaboutharry Feb 2024 #24
Been there done that slightlv Feb 2024 #25
At least it wasn't a Neuralink patient you were working on! True Dough Feb 2024 #26
Almost felt like I was going through withdrawal or something. calimary Feb 2024 #27
No worries MustLoveBeagles Feb 2024 #28
So glad we are back online! I'm sorry that you had a more Tanuki Feb 2024 #29
As someone who works in IT, I fully understand! CousinIT Feb 2024 #30
Thanks for all you do. virgdem Feb 2024 #31
Thank you & Elad for All Your Cha Feb 2024 #32
appreciate all you do... nt WarGamer Feb 2024 #33
Thanks for trying malaise Feb 2024 #34
Thank you for all you do and I also have a clean apartment, extra clean...LOL. SheilaAnn Feb 2024 #35
It's cool. Happens all the time deep in software. Carry on. . . . nt Bernardo de La Paz Feb 2024 #36
No worries. Stuff happens. LoisB Feb 2024 #37
I know you guys were like BumRushDaShow Feb 2024 #38
No need to apologize. You guys are still the best. EarnestPutz Feb 2024 #39
Thank you for the update on the update that didn't update but will update!! chowder66 Feb 2024 #40
Don't... orangecrush Feb 2024 #41
No need for apologies. UTUSN Feb 2024 #43
No worries. Deeply appreciate all you do for us and du. cbabe Feb 2024 #44
I AM OUTRAGED!!!111one!! AnrothElf Feb 2024 #45
such is the natural cussedness of things OldSWODog Feb 2024 #46
To EarlG and Elad ... we understand and we'll get through this ... Rome wasn't built in a day, or even in a year FakeNoose Feb 2024 #47
Thanks for all you do both visible and behind the scenes Golden Raisin Feb 2024 #48
Won't work on my desk top computer Liberal In Texas Feb 2024 #49
As a software developer, this failure was the best possible failure. eggplant Feb 2024 #50
I know that was frustrating, but it's reassuring Pinback Feb 2024 #51
No apologies necessary!! LoveTheDU Feb 2024 #52
Thanks for doing this for us. MuseRider Feb 2024 #53
I was getting worried; glad to see DU back. In case you didn't know, I was up at 4:00 AM ET when Backseat Driver Feb 2024 #54
Situation Normal All Fouled Up (SNAFU) Jacson6 Feb 2024 #55
Not a problem. PoindexterOglethorpe Feb 2024 #56
Fun, fun, fun! Almost anyone who's worked in a company w computers knows there might be some downtime, somewhen... electric_blue68 Feb 2024 #57
No worries mvd Feb 2024 #58
I don't think anyone expects you to be clairvoyant. soldierant Feb 2024 #59
Thanks for what you tried to do. greatauntoftriplets Feb 2024 #60
Thanks for all you do to keep the site AKwannabe Feb 2024 #61
Thanks for protecting our data. sinkingfeeling Feb 2024 #62
I will admit that after the first couple of hours, I started having flashbacks niyad Feb 2024 #63
Stuff happens. I work in IT, so I get it. Kennah Feb 2024 #64
We're on your side.Do what needs to be done. We have have faith in y'all. marble falls Feb 2024 #65
Them new fangled computer things sure are complicated, aren't they? Aussie105 Feb 2024 #66
Absence makes the heart grow fonder. tazkcmo Feb 2024 #67
Noticed a lot of missing Avatars today KS Toronado Feb 2024 #68
I used to hate Windows updates Klondike Kat Feb 2024 #69
Have you tried turning it off and then back on? twodogsbarking Feb 2024 #70
No Worries EarlG.... Mr. Mustard 2023 Feb 2024 #71
DU will not be available on February 30 all day Jerry2144 Feb 2024 #73
I've noted a large number of user's avatars aren't displaying. n/t SpankMe Feb 2024 #74
Thanks EarlG for the efforts and the update sdfernando Feb 2024 #75
Could you try and pick a day for the update when SCOTUS isn't selling our Democracy in a fire sale to protect the...... usaf-vet Feb 2024 #76
friend of mine said that failure is your friend . dont be too harsh on yourself. dont do any good. AllaN01Bear Feb 2024 #77
Will our journals be coming back? Rocknation Mar 2024 #78

OAITW r.2.0

(25,215 posts)
4. No problem. A suggestion perhaps. Schedule your updates late at night.
Wed Feb 28, 2024, 08:54 PM
Feb 2024

Then we don't miss reacting to SCOTUS, Richard Lewis....I have realized that DU is an addiction for me. Can't tell you how many times I hit refresh today with no joy....

rurallib

(62,582 posts)
8. They addressed that in their notice yesterday
Wed Feb 28, 2024, 08:56 PM
Feb 2024

No support late at nightlife their is during normal business hours.
Makes sense.

jls4561

(1,305 posts)
6. Well, it's good you chose a slow news day.
Wed Feb 28, 2024, 08:55 PM
Feb 2024

But seriously folks… the admins should be commended for their hard work and perseverance.

I am sure no one was more frustrated than they.

SleeplessinSoCal

(9,304 posts)
16. the site being down was way too reminiscent of 2016.
Wed Feb 28, 2024, 09:00 PM
Feb 2024

Especially with SCOTUS putting its corrupt weight on the scales of justice. It's got me contemplating suicide as I don't think I could tolerate so much corruption from the right destroying the noble experiment of America.

AltairIV

(439 posts)
11. Good luck!
Wed Feb 28, 2024, 08:57 PM
Feb 2024

You have absolutely no need to apologize. I hope your next attempt is successful and does not induce any number of headaches( literally or figuratively)

ShazzieB

(16,990 posts)
19. No apology needed, but thanks just the same!
Wed Feb 28, 2024, 09:03 PM
Feb 2024

And thanks for filling us in on what happened. As the hours ticked by, I was getting more and more worried, and was ecstatic to find DU finally working again.

As always, thank you for all your hard work that keeps this site running!

dlk

(11,702 posts)
21. The joys of MIS...
Wed Feb 28, 2024, 09:10 PM
Feb 2024

Hang in there. You do a great job and I’m truly grateful for all of your efforts.

2naSalit

(87,775 posts)
23. Bummer...
Wed Feb 28, 2024, 09:12 PM
Feb 2024

But at least you could bring everything back without a lot of grief.

Thank you for everything!

slightlv

(3,143 posts)
25. Been there done that
Wed Feb 28, 2024, 09:20 PM
Feb 2024

So many times before... You are to be commended on doing a backup before the update, no matter how it went in the sandbox. I've worked with others who weren't nearly as cautious. Thank you for all your hard work, and how professionally you do it. It is so very much appreciated!!

True Dough

(17,557 posts)
26. At least it wasn't a Neuralink patient you were working on!
Wed Feb 28, 2024, 09:23 PM
Feb 2024

Just kidding, Earl!

Best of luck with the next run at it. You guys got this!

calimary

(82,000 posts)
27. Almost felt like I was going through withdrawal or something.
Wed Feb 28, 2024, 09:25 PM
Feb 2024

Glad DU is back up, at least to help us all get through these - uh, rather unusual and VERY trying times together.

THANKS SO MUCH for all you do, EarlG and Elad! If I knew what to do to help on the technical end, I’d be jumping in with both feet and both hands to try to help.

Tanuki

(14,957 posts)
29. So glad we are back online! I'm sorry that you had a more
Wed Feb 28, 2024, 09:28 PM
Feb 2024

difficult and frustrating afternoon than expected. Thanks for all you do!

CousinIT

(9,376 posts)
30. As someone who works in IT, I fully understand!
Wed Feb 28, 2024, 09:29 PM
Feb 2024

You'll get it sorted and it will work next time.

You planned properly (backups, backout plan) and THAT is the most important thing.

BumRushDaShow

(131,829 posts)
38. I know you guys were like
Wed Feb 28, 2024, 09:46 PM
Feb 2024


But had a feeling and am real glad you were able to get back up and running!

(time for some adult beverages )

OldSWODog

(44 posts)
46. such is the natural cussedness of things
Wed Feb 28, 2024, 10:37 PM
Feb 2024

EG, thanks much for your efforts...of the many things I learned in the navy, this one stands out: "there is nothing so easy or simple it can't be made hard"...I'm not a digital snipe (steam and diesel were my bags) but can well imagine what you and others are up against...no worries, I look forward to the new install when it's ready...remember to breathe; we're here now and we will be here when you light off the update...btw, it's a maxim of navy damage control to "hold what you've got" (in other words, good planning to run a backup before shifting)...now I'll read the responses to date...cheers, OSD

FakeNoose

(33,438 posts)
47. To EarlG and Elad ... we understand and we'll get through this ... Rome wasn't built in a day, or even in a year
Wed Feb 28, 2024, 10:42 PM
Feb 2024

Thanks for all your efforts to keep DU in such great shape.
You guys are awesome!

Golden Raisin

(4,625 posts)
48. Thanks for all you do both visible and behind the scenes
Wed Feb 28, 2024, 10:43 PM
Feb 2024

to keep DU running smoothly. Hope the next update "takes".

Liberal In Texas

(13,748 posts)
49. Won't work on my desk top computer
Wed Feb 28, 2024, 10:44 PM
Feb 2024

Using Edge.
I can get on with my phone. But not computer.
Will try another browser....
ON EDIT:
Works in Chrome
Will do a reboot.

It's always something LOL

eggplant

(3,930 posts)
50. As a software developer, this failure was the best possible failure.
Wed Feb 28, 2024, 10:44 PM
Feb 2024

Being able to return to the pre-upgrade state is surprisingly uncommon in the field. Congrats on doing things the right way. Remember, you aren't back to where you started... You now have significant knowledge that you didn't have yesterday to help guide the next attempt.

Hang in there.

Pinback

(12,202 posts)
51. I know that was frustrating, but it's reassuring
Wed Feb 28, 2024, 10:44 PM
Feb 2024

to know that your backup scheme and disaster recovery plans are sound. Speaking as someone who worked in IT for years and has managed a few hobby and nonprofit websites, I am unfortunately well acquainted with the feeling of sweating bullets over a site that’s down.

Congrats on the successful recovery, and thanks to you and Elad for the excellent work.

LoveTheDU

(107 posts)
52. No apologies necessary!!
Wed Feb 28, 2024, 10:45 PM
Feb 2024

You are providing a wonderful service to all DU members who deeply value this site. You are amazing.
Thank you!

MuseRider

(34,212 posts)
53. Thanks for doing this for us.
Wed Feb 28, 2024, 10:59 PM
Feb 2024

We do appreciate all you do to keep the site up. Also all the fun things we get to do here, the hearts are always fun.

We will be here whenever it is all done.

Backseat Driver

(4,437 posts)
54. I was getting worried; glad to see DU back. In case you didn't know, I was up at 4:00 AM ET when
Wed Feb 28, 2024, 11:01 PM
Feb 2024

my NWS weather wailer went off for a TORNADO WATCH that quickly turned to a WARNING in my little burb community. I gathered the jeans and shoes, the doggies, the flashlight, the chaise cushion and the ID papers in the purse and headed to the 1/2 bath. It's not the safest but the best choice we got. The sirens were on now...Yeah, prepared and traumatized. Guess what? NOTHING HAPPENED at all - I guess the storm blew right over or went other directions. Anyhow, I spent the afternoon DU downtime taking 3 long naps today. Now I'm all out of whack for sleeping tonight, LOL! Get some rest guys and thanks-- for trying the version update and then returning DU to its addicts in good shape. Next time will be better...no tornado and no software version kinks. Thanks!

PoindexterOglethorpe

(25,997 posts)
56. Not a problem.
Wed Feb 28, 2024, 11:07 PM
Feb 2024

I'm an old computer user. In 1969 I went to work for the very second airline in this country that got a computerized reservation system. American Airlines was the first.

From experience, I know (sort of) how complicated and difficult your job is.

I do love DU.

electric_blue68

(15,386 posts)
57. Fun, fun, fun! Almost anyone who's worked in a company w computers knows there might be some downtime, somewhen...
Wed Feb 28, 2024, 11:09 PM
Feb 2024

Good luck next round. 👍

mvd

(65,202 posts)
58. No worries
Wed Feb 28, 2024, 11:24 PM
Feb 2024

Sometimes you have to try and fail to get things right. I have full confidence you will implement the update fine.

greatauntoftriplets

(175,867 posts)
60. Thanks for what you tried to do.
Wed Feb 28, 2024, 11:50 PM
Feb 2024

I wasn't having the best day today for many reasons unrelated to DU (and not even particularly important except that they pissed me off). When I saw that the update was taking so long, I didn't have a good feeling about that.

Fortunately, nothing was lost and we're back up until the next time.

Thanks, EarlG and Elad.

niyad

(115,029 posts)
63. I will admit that after the first couple of hours, I started having flashbacks
Thu Feb 29, 2024, 12:00 AM
Feb 2024

to 2016. After six hours, was getting really nervous.

Thank goodness that you and Elad are so good at what you do. HUGE sigh of relief! We know that we are in good hands. For now, please take care of yourselves, and know that we are right there with you.

In utter, grateful relief,
Niyad

KS Toronado

(17,874 posts)
68. Noticed a lot of missing Avatars today
Thu Feb 29, 2024, 08:40 AM
Feb 2024

Mine was one of them and couldn't find it where it normally is, so uploaded one off the internet.

Klondike Kat

(818 posts)
69. I used to hate Windows updates
Thu Feb 29, 2024, 09:04 AM
Feb 2024

especially Windows server O/S migrations.

When I was hired as the System Administration role at the job from which I recently retired, I quickly discovered that my predecessor had stopped migrating the Windows server O/S at NT - and Server 2003 was already heavily adopted. It took over a year to get the ducks in a row before I could start a migration to Server 2003 (lots of third party application vendors). However, once the ducks were aligned the migration went fairly smoothly (a few user accounts didn't get transferred, but it was only a handful).

Mr. Mustard 2023

(144 posts)
71. No Worries EarlG....
Thu Feb 29, 2024, 11:04 AM
Feb 2024

I think it's safe to say we've all been there, and it sucks.

You'd think humans would learn not to rush into new technologies after the automobile debacle, but no, if there's a buck to be made, we dive right in.

I don't think it wise to vote by machine, or to post our nations defense systems on digital systems, or to allow social media companies to addict our kids to their systems but that's just me.

Take your time EarlG, do what you have to do and we'll be here, and yes I know the irony of D.U. being delivered digitally.

Jerry2144

(2,183 posts)
73. DU will not be available on February 30 all day
Thu Feb 29, 2024, 01:05 PM
Feb 2024

For maintenance. They will bring it back promptly on March 1.

sdfernando

(5,014 posts)
75. Thanks EarlG for the efforts and the update
Thu Feb 29, 2024, 01:38 PM
Feb 2024

Great that you took a backup just before starting. We take snapshots of servers when we make changes so rollback is a breeze if required....but we also take backups as another failsafe.

I feel for you guys and totally understand! Glad DU is back up and running and hope the issues get resolved so the update can be deployed again.

usaf-vet

(6,375 posts)
76. Could you try and pick a day for the update when SCOTUS isn't selling our Democracy in a fire sale to protect the......
Thu Feb 29, 2024, 01:42 PM
Feb 2024

..... protect the arsonist?



Retired IT consultant. Who always dread updates.

AllaN01Bear

(19,936 posts)
77. friend of mine said that failure is your friend . dont be too harsh on yourself. dont do any good.
Thu Feb 29, 2024, 08:34 PM
Feb 2024

thank god you had backups . sometimes its better to downgrade than up grade.

Latest Discussions»Help & Search»Announcements»IMPORTANT: Today's update...