RSSDisgruntled IT guy fells blogging site
By Angela Gunn | Published January 5, 2009, 8:40 PM
It's better than having some jerk walk back in with a gun, but it's sure not good: The journalspace.com blog site has shut down after a "disgruntled" former IT employee used his own data-backup choice to obliterate its entire data store.
Techish reader, would you rely on RAID as your sole "backup" structure for a mission-critical SQL server? That's what the now-former keepers of journalspace did. The chosen RAID setup wrote all data to two large drives, so in theory it was a perfectly redundant disk-array (not backup!) system; if one drive blew up, the other would hold everything and life would go on smoothly.
This theory, however, failed to take into account what the folks at journalspace describe as "a disgruntled member" of the IT team, or any of the rather short list of other entities that could entirely, irretrievably overwrite both drives. (A catastrophic OS failure is a dim possibility; readers who've heard of anything similar happening on an OS X Server box are invited to pipe up.)
The site failed on December 18. When admins went to restore the data, they discovered why RAID is a fault-tolerance system, not a backup system: Someone had deleted all the data. Suspicion fell on an IT worker recently nabbed for theft -- ironically, the very fellow who set up the RAID "backup system" in the first place. The drives were removed from the hosting site and taken to DriveSavers on December 22; efforts to revive the pair were unsuccessful.
Reezle, a photo-sharing site also hosted at San Jose-based Lagomorphics, was unaffected by the incident. Both sites as well as Lagomophics are owned by half-brothers Dylan and Holland Rhodes.
Journalspace, a blogging platform, was founded in 2002. It is perhaps best known for hosting a blog for Ellen Simonetti, the Delta flight attendant fired in 2003 for posting photos of herself on a company plane.
At Dorrie's Fun Forum, a virtual watering hole for a number of journalspace users, annoyance over the outage (sample quote from "benb" on December 18: "Will JS ever get their act straight? How many times do we have to go through this before this kind of nonsense stops happening? My fear is what is happening to all of our journals right now -- what kind of carnage will we return to? Oh well. Nothing new under the sun.") through a remarkable amount of fretting within the community over various personality issues connected with the small firm.
Eventually, though, the community began to accept the new reality (from "Betty" on December 31: "I'm sick. I'm just sick. I can't believe JS is gone. Everything from almost 5 years. It can't be just over, just like that. It can't be. All my friends are scattered everywhere. I just know we'll all lose touch. I have links all over the place but have had so much to deal with here that I just can't find the energy to keep up.").
Efforts to restore the data were unsuccessful, though some bloggers have reported some success in snatching archive victory from the jaws of Google. The journalspace.net and .com domains and the journalsapce trademark are for sale on eBay. (The winner is also entitled to six months of free hosting at Lagomorphics.) According to the official blog, the site's source code may be released to the open-source community at a later time.
This is a textbook example of a royal FU at EVERY LEVEL of the organization, including its utter lack of best practices planning in every area!
But look at all of the sophisticates debating just WHICH aspect is wrong!
And with such a discussion, the surprise is that this isn't happening in MORE places!
Score: 0
|What does Raid failures have to do with anything about this article? This guy broke into a server illegally and destroyed data that wasn't his. It may make the owners stupid but that is not the point. I guess we should blame the victims instead of the law breakers. "I'm sorry sir but if you didn't have nice things in your house you wouldn't get robbed!" jeeezzz...
Score: 0
|What does Raid failures have to do with anything about this article?
This is a tech news site. Techies like to read that sort of stuff. If it were on CBS.COM, I'm sure they'd leave that part out.
Also, remember this was the same IT guy that setup the array as the "backup" solution in the first place.
Score: 0
|The owners should share much blame since they had no process in place to monitor internal employees in case of sabotage. A basic audit of backups should be part of any businesses process.
Score: 1
|"The owners should share much blame since they had no process in place to monitor internal employees in case of sabotage. A basic audit of backups should be part of any businesses process".
JUST WOW...stupidity not with-standing, NO they should not share "much blame" for what someone else did illegally to their servers.
And far as the other poster, if you want to talk raid that is fine fine but you and the article writer missed the whole point of losing your files/website to a person that probably deserved firing. There are plenty of ways to lose your entire data without it being a raid issue....
Score: 0
|Yup, remember this is a "tech site", when most spend their time whining about how incredily kompleekated formating a new drive is to remove all of the preloaded nonsense loaded by the manufacturer!
Yup. Very sophisitcated techs! Sorta like those working at the company cited in this article!
LOL!
Score: 0
|Owners need to have trust with them managers.
But with out testing the system who knows if the data is backed up. This is why I do not use SAAS solutions.
Backups should be test every MONTH restore random data!
Score: 0
|Dylan Holland did, in one of the posts I linked to above, take responsibility for not giving serious scrutiny to the system after the company kicked the aforementioned thief to the curb. Sounds like the dude was a real piece of work, so you can certainly argue that the company should have been on guard against more trouble, but you know how it can be in seriously understaffed shops; the daily mayhem can obscure your vision of oncoming catastrophe.
Score: 0
|"There are plenty of ways to lose your entire data without it being a raid issue.... "
You prove my point.
Any management worth their salt should have processes in place for knowing their own disaster recovery, with full audit capability. It is management's fault that they did not have this process, ultimately. Thousands of employees a year are disgruntled. It is shy they fire on Fridays, fire with at least two people in room, you pull the security cards, etc.
Score: 0
|"You prove my point".
HUH?
You can have backup and still lose everything. Rare, but it happens. My POINT was putting the blame on someone other than the person who deliberately destroying someone else's data. More than likely the data would never have been lost and hopefully the next person coming in would back it up right. Another great example of blame the victim....blaahh
Score: 0
|I like to pass along things that work, in hopes that good ideas make their way back to me. Data breaches and thefts are due to a lagging business culture. As CIO, I look for ways to help my business and IT teams further their education. Check your local library: A book that is required reading is "I.T. WARS: Managing the Business-Technology Weave in the New Millennium." It also helps outside agencies understand your values and practices.
The author, David Scott, has an interview that is a great exposure: http://businessforum.com/DScott_02.html -
The book came to us as a tip from an intern who attended a course at University of Wisconsin, where the book is an MBA text. It has helped us to understand that, while various systems of security are important, no system can overcome laxity, ignorance, or deliberate intent to harm. Necessary is a sustained culture and awareness; an efficient prism through which every activity is viewed from a security perspective prior to action.
In the realm of risk, unmanaged possibilities become probabilities – read the book BEFORE you suffer a breach. Let me say that just one more time for the wise: READ THE BOOK BEFORE YOU SUFFER A BREACH.
Score: 0
|Typo near the end I think.. "journalsapce trademark" should be journalspace trademark.
Score: 0
|RAID Parity Error.
Score: 0
|I suspect this situation is far more common than people realize.
Score: 0
|"readers who've heard of anything similar happening on an OS X Server box are invited to pipe up."
Better yet, readers who've heard of *anyone* using an OS X Server box are invited to pipe up.
Oh...my mistake; they have a little better than a seven percent market share.
Score: 0
|I've got an OS X server box running and it's a pretty rock solid machine. Those with hate for OS X typically haven't even given it a shot - they just like to flame cause they have nothing better to do with their time.
Score: 0
|You call a server with HFS+ a "rock solid machine"?
Score: 1
|HFS+ is not the only file system available to OSX. Look it up.
Score: 0
|I don't think that was his point...
Score: 0
|His?!
Score: 0
|"readers who've heard of anything similar happening on an OS X Server box are invited to pipe up."
Heh... You just can't resist it, can you?
/me grins wickedly.
The problem with using RAID as a backup is that _any_ change gets written to both disks....even those that you would normally use a backup to correct. (Like deleting a file)
Anyone using RAID as a "backup" is either clueless, or looking to be able to accomplish something similar to this in the not-too-distant future. No-one with any kind of IT education would ever even consider such a setup. (Although this does not keep Management from making stupid calls and forcing such issues)
Score: 0
|Not to mention the possibility of simultaneous multiple-drive failure. I've lost two drives in a RAID-5 twice so far in my life.
Score: 1
|Agreed. Fault-tolerance and backup are two completely different things.
Score: 0
|"the sites source code may be released to the open-source community"
-Written by a highschool sophamore during homeroom.
Score: -1
|Moordrake, there isn't a blogging platform on the planet that can do what JS software was capable of, buggy as it occasionally was. Do you blog? Do you have favorite blogs you like to read on your current platform? Are they all in one place, and can you send personal messages to any blogger you chose on that platform? Is there a homepage that constantly refreshes with the last updated blog, or can you peek at comments randomly chosen to appear on the homepage? That "feature" was one of my favorite things there; comments without context could be very amusing. What about stats? Can you tell exactly which of your blogmates are hitting your blog, by name, even if they don't leave a comment?
When Blogspot or Wordpress or the inscrutable Livejournal can do those sorts of things, then you can call the JS platform sophmorific.
Score: 0
|Hi cageliner -- thanks for mentioning that. JS had some very, very interesting thinking evident in it, some nice insight into what bloggers want to *know* about. I'm hopeful that it really will see daylight again, in whatever context. (But don't stress too much re moordrake; s/he may have been complaining about the missing apostrophe in the quoted passage, though if s/he was I have to smile at the misspelling of 'sophomore.' And if it was commentary re the platform, there's really not enough substance in the comment to raise it above the level of random carping, is there now...)
Score: 0
|My bad. In my haste to lend support for a site where I spent over three years building friendships (which I have been able to retain, thanks to Dorrie) I forgot about the drive by shootings common on forums whenever one makes a slip in spelling or grammar.
Nice to meet you, Angela.
Score: 0
|