How does "data may be queried only when there is a reasonable suspicion" = 20 million/month?

[dkf]

28. Which is why the government is at the forefront in developing ways to handle uber data.

A computing and software revolution, launched in Silicon Valley a few years earlier, made sifting all that data easier. That was particularly true with the development of Hadoop, a piece of free software that lets users distribute big-data projects across hundreds or thousands of computers.

Named after a child's toy elephant and developed at Yahoo Inc., YHOO +0.66% the software reached commercial scale for Internet-wide tasks in 2008 and soon became a favored application for handling big-data demands. Twitter Inc. and Facebook snapped it up to manage their own giant databases of user information.

The NSA also became an early adopter. At a 2009 conference on so-called cloud computing, an NSA official said the agency was developing a new system by linking its various databases and using Hadoop software to analyze them, according to comments reported by the trade publication InformationWeek.

The system would hold "essentially every kind of data there is," said Randy Garrett, who was then director of technology for the NSA's integrated intelligence program. "The object is to do things that were essentially impossible before."

Mr. Garrett now runs RTRG's successor program, which was moved to the Defense Advanced Research Projects Agency and renamed Nexus 7. That effort has been using Hadoop and similar software to help manage large masses of data. One of the pieces of software, called Accumulo, was developed by the NSA using technology from Google, said a person briefed on the program.

"They've changed the paradigm so you no longer need supercomputers," this person said. "Randy has sought to leverage a lot of those big-data advances to improve U.S. government analysis on extremely large data sets."

Still, the NSA continues to face an enormous challenge in handling all its information. "The ability to manage it remains slim, because there's so much data out there," said one former NSA official.

http://online.wsj.com/article/SB10001424127887323495604578535290627442964.html