I hate to repeat a post, but the latest NSA revelations are more profound than I can describe

[Pholus]

15. Naw...consider their history! Metadata was a baby step.

Traffic analyses were always easier to do than actual interceptions and have a long history of being valuable from a military point of view (The Battle of Midway, for example, which was a codebreaker victory first and foremost). It also would be data more relevant to investigators trying to bust up a terror network from a handful of suspects (known associates). As we discussed to death last summer, it requires a tiny, tiny, tiny fraction of the server space in one of our many fusion centers. Given the size of a metadata record and the number of calls made per day in the US, several years worth of calls can be saved on pretty much a single rack of servers. If it was "just metadata" this project could run from a single nondescript office in some technical park on the outskirts of a major city.

When the post-9/11 gloves came off, metadata came first because it could be done quickly -- the telcoms already had the info! The rest came later.

But I've never doubted that eventually storing everything was a goal and that it was certainly to enable the exact functionality I fear regardless of other reasons being given. The technical problem is trivial even if the scale is huge, but not impossibly so, and there are metrics! Anyone beholden to a funding authority knows that a precise looking "completion" bar graph which moves is gold, regardless of debates about the value of the actual end product.

Sorting it all? They're not there yet, otherwise DARPA would not be advertising research money to "detect anomalous patterns" in massive datasets. They have too much data, making sense of it in a usable way is actual hard (near impossible) project.