RSSIBM Builds Super Fast File System
By Ed Oswald | Published March 9, 2006, 5:44 PM
IBM on Thursday announced that it had scored a breakthrough in file system technology that increases the speed of data access by seven times. Researchers were able to attain a 102-gigabyte per second transfer rate on the ASC Purple supercomputer at Lawrence Livermore National Laboratory in a recent test.
The file system was an astonishing 1.6 petabytes in size, the largest ever in the world, and performance was maintained even as 1,000 clients pushed workloads into the file. The project used 104 Power-based eServer p575 nodes and 416 storage controllers, IBM said in a statement.
The development permits a whole new class of applications, says the company. "Computing capability has been growing very fast, but the file system capacity has not kept up," IBM distinguished engineer Dr. Rama Govindaraju told BetaNews in an interview earlier this week.
Called the General Parallel File System (GPFS), the technology allows for high-speed access to files across multiple nodes of a Linux or AIX cluster. The file system could be used in a variety of fields, including engineering design, digital media and entertainment, data mining, financial analysis, seismic data processing and scientific research.
Govindaraju says that the biggest bottleneck to prevent more feature-rich applications were the file system's capacity and size. "More and more, computers are transitioning from numbers and strings-based to media-based analytics," he added.
With the enhanced capacity of GPFS, entirely new applications have been made possible. For example, the system is already seeing use in medical imaging, and is allowing for doctors to search and compare through thousands of images sometimes hundreds of megabytes in size each.
Uses in the medical field could go beyond just imaging, to enable more intelligent medicine, better medicine design, and for use in educational purposes as well.
The system could also have use in homeland security applications. Govindaraju offered a hypothetical situation where cameras at an airport could be connected to a GPFS-enabled computer allowing for pictures to be taken of passengers at multiple angles and compared to databases of known terror suspects.
"These pictures would have scanned and analyzed before the passenger gets to the immigration officer," he said.
IBM will push GPFS on several fronts, including an effort to even promote its use on non-IBM hardware. The source code behind the file system will be released to eligible clients who can develop upon the technology and share their work with others.
However, the impetus behind the development of GPFS is the user's changing computer needs, says Govidaraju. "The kind of data people are operating on is completely different from even ten years back," he said.
I'll bet i know who will use this: DHS. It's public record that IBM has DoD contracts.
Score: 0
Wow, Ibm is in the news. Why is it the company that started the I.T. world essentially is doing what now? I don't even know nothing they do is in the news apearently. I was also trying to figure out how a industry which is based on their own instruction set, is dominated by everyone else?
Score: 0
i want 1.6 petabytes storage for my music n movies, it would last me atleast a year or two!!!!!!!!!!!!!!!!!!!!!! currently i have to buy a 250 gig hdd every month and i can only copy use 1/15 of my download capacity!!!!
Score: 0
I'm a GPFS user and it isn't new by any means. On AIX the client requirements are satisfied with the rsct and mmfs filesets. Yes, the old multimedia filesystem....
Score: 0
this is very cool technology. sounds a bit like the already existing/proven google filesystem. not something to compare with m$ ntfs, you near sighted idiots (go apply some updates or something). this is something i'd suspect that has no relevence to home users. this thing sits on top of a cluster of aix or linux nodes (as cleary described in the article). this is cool because i.t. departments potentially could have a google filesystem right in their very own data center without having to invent it themselves. i wonder what the client requirements are, what do the underlying protocols look like, network or san centric?,?, etc.. this is obviously distributed, is it redundant, are there advanced features like snapshots, versioning, integrity/corruption, concurrency? i wonder, how in the heck do you back something like this up at night? hi over there at llnl :-)
Score: 0
i want xp to use this filesystem
Score: 0
This file system is not meant for end-user/single PC.
It's meant for clusters of servers.
Score: 0
I gather it doesn't lose performance when many multiple pieces of hardware have to talk to eachother.
You probably want ReiserFS on your XP computer. Only problem with it is the windows beta port is read-only and BSOD's for some people.
Reiser4 is apparently the most efficient single-computer filesystem when dealing with small files. You're looking at dozens of times the performance over NTFS, since it doesn't slow down much with tiny stuff. If you want an example, then try deleting 4 500mb files, and try copying your windows CD over 4 times and deleting the folder with them in it.
If you don't have XP(which hides it), deleting should take oodles longer.
Oh, and don't let your computer lose power with Reiser. It caches stuff in memory, so if you lose power you should get major data corruption. If you want to mess with the unstable windows port, buy a UPS. :D
Score: 0
IBM's is a stable mature technology company. Most of their research doesn't get productized by itself, but the concepts get wrapped up in other products and sold to high-end businesses. Over time, the technology gets watered down for home use and is implemented by everyone.
IBM is not a house hold name, but thank goodness they do what they do.
Score: 0
"IBM isn't a Houshold name" you can't be serious?
Score: 0
"IBM is not a house hold name"
HAHAHA Damn young whipper snappers! IBM has been a household name for decades.
Score: 0
IBM approach: Super fast file system for super expensive implementations which require super numbers of our super consultants to implement over a super long period of time (usually, infinity).
Score: 0
^^
Sometime we laugh because it's funny
Sometimes we laugh because it's true
But this time, I'm laughing because it's both
You summed up IBM quite nicely there.
Score: 0
Hey, another great IBM innovation that will sit on a shelf collecting dust while M$ creates a cheap knock-off and makes millions. LOL!
Score: 0
This might be aimed at a slightly different market. IBM will make plenty.
Score: 0
yet another ignorant kid
Score: 0
Microsoft Revenue (ttm): 41.36B
IBM Revenue (ttm): 91.13B
Oh yes, microsoft is the big bad company making all the money!
Score: 0
IBM provides a service to businesses mainly. MS provides a service to home users, which turns many away when they get viruses and malware and other things and have to spend lots of money getting it fixed up.
MS is actually helping Apple expand, in that respect...
Score: 0
A little roomier than NTFS, eh?
Score: 0
No, this is talking about the "file system" being 1.6 petabytes, NOT a single "volume." the current version of NTFS is limited to 128 terabytes per volume, but multiple volumes can be aggregated into a file system using volume mount points. This isn't something which is commonly done today, but of course people don't commonly have petabytes or even multiple terabytes worth of hardware to play with. If you did, there's no technical reason why NTFS couldn't support it. It's likely that performance would be poor, because Microsoft has obviously never tried to test or tune for that kind of capacity, but IBM probably didn't achieve their reported speed on the first try either.
Score: 0