Welcome to DU! The truly grassroots left-of-center political community where regular people, not algorithms, drive the discussions and set the standards. Join the community: Create a free account Support DU (and get rid of ads!): Become a Star Member Latest Breaking News General Discussion The DU Lounge All Forums Issue Forums Culture Forums Alliance Forums Region Forums Support Forums Help & Search

struggle4progress

(118,280 posts)
Sat May 11, 2013, 08:29 AM May 2013

More crackpot thinking from Glenn Greenwald

Last edited Sat May 11, 2013, 09:34 AM - Edit history (1)

... all digital communications -- meaning telephone calls, emails, online chats and the like -- are automatically recorded and stored and accessible to the government after the fact ...
Are All Telephone Calls Recorded And Accessible To The US Government?
By Glenn Greenwald
Saturday 4 May 2013 08.22 EDT
http://www.guardian.co.uk/commentisfree/2013/may/04/telephone-calls-recorded-fbi-boston

Wireless survey: 91% of Americans use cell phones
by Chris Foresman - Mar 24 2010, 8:28pm EDT
... The survey of wireless carriers revealed that over 285 million Americans are mobile subscribers, about 91 percent of the total population. That's up 15 million over the same time last year, and growth has slowed somewhat due to market saturation. Those 285 million callers used 1.12 trillion minutes of talk time in the last half of 2009, up 3.4 percent of the same period in 2008. That breaks down to an average of 6.1 billion minutes used per day, or about 21 minutes per person per day ...
http://arstechnica.com/tech-policy/2010/03/wireless-survey-91-of-americans-have-cell-phones/

Voice apps can strain storage
http://storageMagazine.techtarget.com/magItem/0,291266,sid35_gci1179419,00.html
by: Ray Lucchesi
Issue:Apr2006
... VoIP recording consumes approximately 1 MB of storage per minute of phone conversation ...
http://silvertonconsulting.com/cms1/wp-content/uploads/filebase/archive/article/Voice_apps_strain_storage.pdf

OK. Let's do a little calculation. Storing 6.1 billion cellphone minutes daily at 1 MB of storage per minute is 6.1 billion MB per day or 6.1 million GB per day. That's every day of the year, so the annual storage consumed is about 2.2 trillion GB just to store the messages themselves.

What are the channel requirements? 6.1 billion cellphone minutes daily means that on average you've got over 4 million minutes of cell phone voice message to record every single minute of the day (more in peak times).

That's just cellphone voice messages. Note we're not counting landlines, we're not counting text messages (1.138 trillion annually), and we're not counting emails. Nor are we counting the essential cost of indexing those messages by phone numbers and dates and sorting the indices so they can be found and examined when needed

You're crazy, Glenn

99 replies = new reply since forum marked as read
Highlight: NoneDon't highlight anything 5 newestHighlight 5 most recent replies
More crackpot thinking from Glenn Greenwald (Original Post) struggle4progress May 2013 OP
Not a problem if you're NSA and have a yottabyte of storage capacity, you ninny leveymg May 2013 #1
You're overlooking the technical problems of constantly streaming 4 million conversations struggle4progress May 2013 #8
Google precesses 24 petabytes/day. That's one company. Apparently you can't grasp that reality. leveymg May 2013 #11
Google does process about a billion queries daily. But they all arrive at the company struggle4progress May 2013 #24
The scale of data is roughly the same: Google 24 petabytes/day; ATT network 19 petabytes/day leveymg May 2013 #26
Correction, Google does store web pages. check the "cache" link on most search results. bobduca May 2013 #91
Yeah, you're right. My mistake. But google doesn't save the whole history of that page: struggle4progress May 2013 #95
I think your point basically stands, but the numbers (while still large) are a bit off... Silent3 May 2013 #2
The OP is WAY, WAY off. At 60KB/minute storage for voice audio, 2.2 trillion min/yr requires leveymg May 2013 #7
It depends on the use you plan to make of the recording. It may be that by reducing quality struggle4progress May 2013 #12
Data compression is applied by the cell phones themselves... Silent3 May 2013 #18
One shouldn't confuse the radio channel signal with the conversation. When cell phone companies struggle4progress May 2013 #82
Nothing but very compressed data ever gets out of your phone into the air, however Silent3 May 2013 #90
This message was self-deleted by its author RudynJack May 2013 #3
Greenwald seems to be repeating a claim made by a former FBI counter-terrorism agent. Jim__ May 2013 #4
He indeed reports the claim, then he repeats it in manner indicating that he accepts it: struggle4progress May 2013 #14
I'll go ahead and Godwin this thread Fumesucker May 2013 #5
Google's servers process 24 petabytes per day. That would leave a lot of spare capacity to process leveymg May 2013 #10
It's largely text processing struggle4progress May 2013 #17
If you're recording conversations for intelligence analysis purposes, you want the best quality struggle4progress May 2013 #20
I wrote a voice recording app for the Commodore 128 back when it was the latest thing Fumesucker May 2013 #21
This is correct. n/t whatchamacallit May 2013 #50
don't jump on me but ... Ligyron May 2013 #6
If you've got 4 million conversations coming in at every minute, and you're gonna listen to struggle4progress May 2013 #16
That wasn't Greenwald's claim Fumesucker May 2013 #19
No, his claim was that all calls were recorded and stored. I'm simply doing the math here on what struggle4progress May 2013 #22
No one has ever claimed that all the calls are listened to by humans Fumesucker May 2013 #23
I was responding to a comment made in #6: please do read subthreads when commenting struggle4progress May 2013 #25
thanks for doing the math Ligyron May 2013 #33
What a stupid argument. woo me with science May 2013 #28
I didn't argue any such thing: I've actually pointed out that there are too many to listen to struggle4progress May 2013 #36
How naive. woo me with science May 2013 #37
It is of course true, and always has been true, that there are unprincipled people struggle4progress May 2013 #43
So where is the proof? All you offer is speculation and with it you are smearing a good name. cui bono May 2013 #56
GG's PoV seems to be that if a "former FBI agent" suggests the Administration struggle4progress May 2013 #86
You'll never find "most Americans" loudly and vocally oppose anything political. Most Americans cui bono May 2013 #88
Violating our rights is not okay, ever. n/t cui bono May 2013 #54
Of course it's not OK! Ligyron May 2013 #89
OK!!! cui bono May 2013 #92
You better pay attention to this. alarimer May 2013 #9
From what I've read, your statement is 100% correct. byeya May 2013 #13
Yup. woo me with science May 2013 #27
But I bet Glenn was behind that article!!!111 bobduca May 2013 #35
Well, Tim Clemente is getting lots of attention for making this claim, but actually struggle4progress May 2013 #40
OMG. And DU posters are making claims that it isn't true and there's still no reason to believe cui bono May 2013 #53
There are currently something like 14K FBI employees. The FBI has no statutory authority struggle4progress May 2013 #84
And you're refuting what the FBI agent says based on your ideas of what is possible digitally which cui bono May 2013 #87
We've been hearing about this for years. Not sure why the OP is freaking out about the reporting now cui bono May 2013 #58
His specialty. nt MineralMan May 2013 #15
he's giving Alex Jones a run for his money dlwickham May 2013 #29
The OP is baseless slander. leveymg May 2013 #30
But they don't have the legal authority. ucrdem May 2013 #38
Your theory requires the view that ISPs, landline companies, and wireless communication companies struggle4progress May 2013 #39
Installation of the intercept equipment by telcos has been mandatory under CALEA since 1994 leveymg May 2013 #44
Doesn't most or all of this pertain to Bush-era domestic surveillance ucrdem May 2013 #46
The 2008 Amendments to FISA legalized most of it. leveymg May 2013 #49
It legalized some FISA warrants, sure, ucrdem May 2013 #55
Do you know how a "driftnet" warrant of the type legalized by the '08 Amendment works? leveymg May 2013 #69
Apparently it's not so legal. ucrdem May 2013 #70
Torture isn't legal either Fumesucker May 2013 #71
So who has Obama tortured? ucrdem May 2013 #72
Are you trying to say that America has never tortured anyone? Fumesucker May 2013 #73
The EFF suit against Bush Admin officials is still alive, the ATT case was dismissed. leveymg May 2013 #74
Okay thanks. ucrdem May 2013 #76
The point is, TIA never stopped. It just grew into the current NSA program. leveymg May 2013 #80
Sounds like it never started. ucrdem May 2013 #85
Large core components of TIA were shifted to NSA and DNI, and never ended. leveymg May 2013 #93
It looks like TIA was D.O.A. by 2004 ucrdem May 2013 #94
Like ThinThread, the technologies TIA incubated migrated to other programs leveymg May 2013 #96
You haven't understood what CALEA does: struggle4progress May 2013 #75
You're reading the 1995 regulations, ninny. leveymg May 2013 #78
I did not locate any further FR notices establishing capacity requirements. Following links from one struggle4progress May 2013 #83
You need to pay attention Ichingcarpenter May 2013 #31
Doesn't corroroborate Greenwald's claim ucrdem May 2013 #41
Oh, the "national security" establishment is certainly out of control: it's been oversized struggle4progress May 2013 #42
Redundancy and waste are only two of the three threats: There is institutional jealousy and lust for byeya May 2013 #45
Institutional competitiveness can be a good thing for democracy: it means none of them struggle4progress May 2013 #48
I think having 2, or more, agencies with the same mission is a waste of money and Cabinent byeya May 2013 #57
The OP is the one giving Alex Jones a run for his money! n/t cui bono May 2013 #64
They tell us they are spying on us to "protect" us from tyranny. Of course, they miss the irony. Tierra_y_Libertad May 2013 #32
Glenn Greenwald never loved Obama!! 111 bobduca May 2013 #34
And just in case that doesn't work, woo me with science May 2013 #47
Haters gonna hate whatchamacallit May 2013 #51
UNREC - Wow. The Glenn Greenwald smearing is truly disgusting. cui bono May 2013 #52
or ... AtomicKitten May 2013 #59
Nope. That's definitely not it. n/t cui bono May 2013 #60
Uh huh. n/t AtomicKitten May 2013 #61
Well, back up your speculation then. n/t cui bono May 2013 #62
okay AtomicKitten May 2013 #67
And that proves what, how? cui bono May 2013 #68
You hit it on the head. This post is propaganda that works with other similar smears on this board leveymg May 2013 #63
He's a racist libertarian. nt msanthrope May 2013 #66
I keep all of Glenn Greenwald's phone calls on mi iPod arely staircase May 2013 #65
Eventually, someone will take the government to court to find out. JDPriestly May 2013 #77
The old left activists I knew decades ago would never discuss any concrete plans over the phone struggle4progress May 2013 #79
I used to know a guy whose job it was to censor the Western press in Poland. JDPriestly May 2013 #81
I'm not a fan of Greenwald, but he might be right here. stevenleser May 2013 #97
The Stellar Wind program appears to have been an offshoot of Thin Thread. See #93, 96 above. leveymg May 2013 #99
Glenn Greenwald is a good person Douglas Carpenter May 2013 #98

leveymg

(36,418 posts)
1. Not a problem if you're NSA and have a yottabyte of storage capacity, you ninny
Sat May 11, 2013, 09:02 AM
May 2013

Last edited Sat May 11, 2013, 12:15 PM - Edit history (1)

$2 billion that's going to build the Utah server farm buys (will house up to) a yottabyte of data - that's many acres of stacks upon stacks upon stacks of servers. A petabyte is a quadrillion bytes. A yottabyte is a billion petabytes (10 to the 24th bytes).

The petabyte entry in Wikipedia offers some illuminating benchmarks:

60 petabytes: Storage capacity of the German Climate Computing Center in 2009
50 petabytes: Claimed capacity of Teradata's (TDC) Database-12 product
35 petabytes: Video gaming data delivered by Steamworks during its 2010 holiday sale
32 petabytes: Reported capacity of Hitachi's 2004-era TagmaStore Universal Storage Platform
24 petabytes: How much data Google (GOOG) was processesing every day in 2008
19 petabytes: Amount of data transferred through AT&T's (T) network every day in 2008
1.3 petabytes: Space required by a single game -- World of Warcraft -- to store its data.

All this pales beside the $2 billion storage facility the National Security Agency is reported to be building in Utah. Estimates for how much surveillance data it will store range from hundreds of petabytes to one yottabyte, although that second number strains credulity. There are a thousand petabytes in an exabyte, a thousand exabytes in a zettabyte, and a thousand zettabytes in a yottabyte. http://tech.fortune.cnn.com/2011/04/07/whats-12-petabytes-to-apple/


You should have done some basic research before posting this nasty, ignorant rant. You're not even remotely competent to comment on these things, and your OP is embarrassing.

struggle4progress

(118,280 posts)
8. You're overlooking the technical problems of constantly streaming 4 million conversations
Sat May 11, 2013, 09:45 AM
May 2013

Last edited Sun May 12, 2013, 03:35 AM - Edit history (1)

to a central local for storage and recording those conversations without equipment failures, which would occur constantly considering the quantity of equipment needed to record those conversations

BTW the reported $2 billion figure for the Utah facility seems to be for building construction, not for equipment

Exact figures will be lacking, but reports suggest another $2 is currently budgeted for equipment: that certainly won't buy yottabytes of storage. Cheapest media costs might be in the range of $0.01/gigabyte putting a yottabyte at $10K billion, which is two-thirds of the size of the US economy -- and that only includes the storage media, without any other equipment.

At 6.1 billion GB/day recording all US cell phone conversations would come to about 20 billion annually just for storage media of the messages -- which are almost entirely junk ("Hi Bunny! I'm at the pizza joint! Where are you?&quot of not interest whatsoever from either an intelligence PoV or a criminal investigations PoV. Nobody's going to pay that much to store mountains of conversational junk, which would be produced in such quantities that it would be impossible to process and the majority of which therefore would never be listened to

leveymg

(36,418 posts)
11. Google precesses 24 petabytes/day. That's one company. Apparently you can't grasp that reality.
Sat May 11, 2013, 09:51 AM
May 2013

I recommend you either figure it out and correct your diary or delete it. It makes you look like an ignoramus with a dull axe.

struggle4progress

(118,280 posts)
24. Google does process about a billion queries daily. But they all arrive at the company
Sat May 11, 2013, 11:19 AM
May 2013

in a fairly standard short text form from their webpage. And the company essentially maintains several million servers dedicated to indices and index algorithms. The company does not store webpages: it creates and stores indices. The company indexes or reindexes webpages at much slower rate. They find their webpages by following links. Most webpages are available 24/7

The problem of grabbing several billion phone calls a day and indexing them and storing them is a rather different problem

leveymg

(36,418 posts)
26. The scale of data is roughly the same: Google 24 petabytes/day; ATT network 19 petabytes/day
Sat May 11, 2013, 11:57 AM
May 2013

So is the data processing involved. The sort of initial parsing and categorization that NSA does with that data is not dissimilar to Google's protocols and algorithms.

Analysis of this semi-finished data set is a separate set of technologies than collection, categorization and indexing. The indexed relational NSA database is in turn run through more specialized software developed by other alphabet agencies to carry out things like profiling, predictive analysis, social network analysis. NSA still does the bulk of the cryptology, but the FBI, CIA, DIA, etc. do their own focused sorts of analyses, a lot of which overlaps, and this intelligence is in turn disseminated to intelligence users according to clearance levels and compartmentalized programs.

Domestic collection of phone, email, text is done through splitters that link into the data network hubs operated by ATT, the other telcos (and has been since the 1994 CALEA Act - if you don't know the law I'm referring to here, you need to study up); VOIP and web traffic is collected through similar mandatory data splitters that connect into the main ISPs; NSA also accesses all data in the domain registries, and private data networks interception is carried out mainly by signals interception -- cable taps, satellite intercepts, microwave, old fashioned over-the-air transmissions -- and related technical means of collection conducted by the NSA and the military.

Storage of all this "take" is the simplest task of all - it's just adding more stacks on racks, in the same way you or I might add additional memory by buying another external hard-drive and plugging it in to your processor.

You are the crackpot here, and an ignorant one.

bobduca

(1,763 posts)
91. Correction, Google does store web pages. check the "cache" link on most search results.
Sun May 12, 2013, 12:33 PM
May 2013

You can visit a webpage from google's cache still for most websites.

they are not magically recreating taht page from a search index. the entire page, and its images are cached.

This has been the case since google opened its "browseable cache" feature, many many years ago.

struggle4progress

(118,280 posts)
95. Yeah, you're right. My mistake. But google doesn't save the whole history of that page:
Sun May 12, 2013, 11:58 PM
May 2013

it only saves the version of the page from its own most recent visit. And a typical request to google is served by directing the user to the existing page, not by serving up the cached version

Silent3

(15,204 posts)
2. I think your point basically stands, but the numbers (while still large) are a bit off...
Sat May 11, 2013, 09:05 AM
May 2013

...by estimating 1 MB/minute of voice data. Voice-quality audio is typically sampled at only around 8 KHz, with 8-bit samples, for a raw data rate under half a MB per minute. That raw data rate can be compressed quite a lot by standard codecs, down to where you only need 60 KB/minute.

leveymg

(36,418 posts)
7. The OP is WAY, WAY off. At 60KB/minute storage for voice audio, 2.2 trillion min/yr requires
Sat May 11, 2013, 09:32 AM
May 2013

only a tiny fraction of a yottabyte.

132 thousand trillion bytes is 132 petabytes. By comparison, Google’s servers process over 24 petabytes of information each day.

thousand - 10 to the 3rd power
trillion - 10 to the 12th

132 thousand trillion - 132X10 to the 15th power - 132 petabyte is needed to process or store a year's worth of U.S. cell phone calls.

1 yottabyte - 10 to 24th power

struggle4progress

(118,280 posts)
12. It depends on the use you plan to make of the recording. It may be that by reducing quality
Sat May 11, 2013, 10:01 AM
May 2013

one can reduce storage requirements by a factor of 15 or so. Note, however, that at any given time there are on average about 4 million conversations occurring, so an enormous amount of processing must occur and on average it must occur in real time, since otherwise one simply has no chance of catching up on backlogs

Note further that if the intended use were (say) criminal investigation, it seems likely that the investigators would opt for a higher quality recording that could stand up in court

And one should expect similar considerations to apply if the recording were made for purposes of an intelligence or counter-intelligence operation. While the human ear is able decode fairly degraded and distorted recordings, it is much easier to work with good quality recordings: this would especially be true if (as is sometimes alleged) the purported NSA capture of all phone traffic is computer processed using voice recognition software

The requirements for constantly processing 4 million incoming conversations in real time seem to me obviously prohibitive, whether the processing is to reduce data storage requirements by some data compression scheme or to scan conversations using voice recognition software

Silent3

(15,204 posts)
18. Data compression is applied by the cell phones themselves...
Sat May 11, 2013, 10:15 AM
May 2013

Last edited Sat May 11, 2013, 11:46 AM - Edit history (1)

...before a voice signal even leaves a phone to go out over the airwaves. Compression is part of the design of the digital cellular network to reduce the bandwidth requirements for voice data. The degradation to the forensic value of these signals caused by lossy audio codecs is unavoidable because the losses have already occurred before the signal becomes accessible.

struggle4progress

(118,280 posts)
82. One shouldn't confuse the radio channel signal with the conversation. When cell phone companies
Sun May 12, 2013, 02:58 AM
May 2013

sample a conversation for digital conversion, with the aim of increasing the number of conversations carried at a certain frequency, their aim is to chop the conversation into packets that can be interspersed with packets from other conversations, so the signal along the radio channel typically consists of interspersed packets from a number of conversations, and this signal itself generally has no law enforcement or intelligence interest

What might have actual law enforcement or intelligence value is a particular reconstructed conversation, and the federal statutes, that require telecom companies to be able to comply with multiple wiretap court orders, actually require the companies to be able to deliver the conversations themselves from the channel stream

For law enforcement purposes, it will be much more satisfactory to have the telecom itself reconstruct the conversation, with the existing network technology actually used for the communication, than to have the telecom provide to a government agency the compressed version of the conversation and to have the government then reconstruct the signal, because off-site third-party reconstruction is subject to defense objections that reconstruction errors have introduced additional misleading audio distortions

The same sort of objection applies to governmental recompression of reconstructed audio for storage purposes. If (for example) the original conversation was sampled at a certain rate to produce numerical data for compression based on fourier analysis, the sampling rate limits the frequency modes that can be accurately reproduced, and higher frequency modes are not accurately captured by rather appear as aliasing errors in the reconstruction. If the original reconstruction is subsequently compressed and decompressed by an algorithm differing from the original algorithm, it is therefore possible under some circumstances to introduce additional deviations from the original

So agencies obtaining cell phone conversations under court order are likely to want the provider's highest-possible quality reconstruction of the conversation from the signal, rather than the radio channel data, and (so far as is feasible) they are unlikely to further compress and decompress the reconstruction delivered to them by the provider

Silent3

(15,204 posts)
90. Nothing but very compressed data ever gets out of your phone into the air, however
Sun May 12, 2013, 09:52 AM
May 2013

I'm not talking about the telecom providers doing extra compression of the signal they're receiving from your phone, and the packet structure has nothing to do with this. The telecoms only ever see a compressed, low-fidelity, low date rate signal from you to begin with.

You speak into the microphone of your cell phone, the signal from the mic gets digitized at a low bit rate from the start, like 8-bit/8 KHz, then that already low-fidelity digital signal gets compressed about 8-to-1 before it ever gets transmitted anywhere. Nothing close to 1MB/minute of voice data is ever leaving your phone, no matter what the packet structure is.

If someone was hellbent on high-fidelity spying they'd have to hack into your phone and plant a bug with its own transmission circuitry that taps into your mic early in the signal chain. It might be easier on a smartphone -- many of them can do (relatively) high-fidelity audio sampling, so you might be able to do a hack completely through software that sends higher-fidelity audio data out the cellular data channel or over wifi.

Even if the mic in your phone is originally sampled at a high data rate, however, for use with the normal cellular voice data path that signal is going to get downsampled and filtered to 8-bit/8 KHz or there about, and then get the usual 8-to-1 compression before transmission.

Response to struggle4progress (Original post)

Jim__

(14,075 posts)
4. Greenwald seems to be repeating a claim made by a former FBI counter-terrorism agent.
Sat May 11, 2013, 09:09 AM
May 2013

From the article you cited, Clemente is a former FBI counter-terrorism agent:


BURNETT: Tim, is there any way, obviously, there is a voice mail they can try to get the phone companies to give that up at this point. It's not a voice mail. It's just a conversation. There's no way they actually can find out what happened, right, unless she tells them?

CLEMENTE: "No, there is a way. We certainly have ways in national security investigations to find out exactly what was said in that conversation. It's not necessarily something that the FBI is going to want to present in court, but it may help lead the investigation and/or lead to questioning of her. We certainly can find that out.

BURNETT: "So they can actually get that? People are saying, look, that is incredible.

CLEMENTE: "No, welcome to America. All of that stuff is being captured as we speak whether we know it or like it or not."


The article also states that Clemente has verified that claim on other shows. Could Clemente be lying? Sure. But his claim is that all digital communication is recorded.

The storage requirements that you cite may not be accurate. The article is from 2006, the government could have better compression routines; things may be stored selectively.

I don't think Greenwald is necessarily crazy because he accepts a claim by a former federal agent about the spying capabilities of the US government - even if those capabilities sound fantastic to the average citizen.

struggle4progress

(118,280 posts)
14. He indeed reports the claim, then he repeats it in manner indicating that he accepts it:
Sat May 11, 2013, 10:06 AM
May 2013
...That no human communications can be allowed to take place without the scrutinizing eye of the US government is indeed the animating principle of the US Surveillance State. Still, this revelation, made in passing on CNN, that every single telephone call made by and among Americans is recorded and stored is something which most people undoubtedly do not know...

GG's claim is not selective storage: it is that everything is recorded and stored

As I point out elsewhere in the thread, the assumption that what's stored is compressed requires on average that at every instant 4 million conversations are being processed for compression

Fumesucker

(45,851 posts)
5. I'll go ahead and Godwin this thread
Sat May 11, 2013, 09:23 AM
May 2013

The best minds of the Third Reich assumed it was impossible for the German Enigma code to be broken.

Churchill actually read many of Hitler's top secret messages before Hitler did.

Making negative assumptions based on presumed technical incapacity to do something is not a very smart thing to do.

For example, the assumption of 1 MB per minute for a voice conversation is off by a factor of at least two just for MP3 encoding, there are better and more efficient codecs than MP3 available to the public these days, I"m sure the NSA can do better than what is available commercially.

http://www.audiomountain.com/tech/audio-file-size.html

At 48 kbps an MP3 file uses about 360 KB per minute, that's fine for voice recording although sounds crappy for music.

leveymg

(36,418 posts)
10. Google's servers process 24 petabytes per day. That would leave a lot of spare capacity to process
Sat May 11, 2013, 09:46 AM
May 2013

and store the 6.1 billion cell phone minutes used per day in the US, according to the figure quoted by the OP.

struggle4progress

(118,280 posts)
20. If you're recording conversations for intelligence analysis purposes, you want the best quality
Sat May 11, 2013, 10:17 AM
May 2013

recordings you can get: you want to be able to hear background noises and conversations in the room; you want to be able to identity accents clearly or things that might be mumbled

Fumesucker

(45,851 posts)
21. I wrote a voice recording app for the Commodore 128 back when it was the latest thing
Sat May 11, 2013, 10:27 AM
May 2013

Even wrote a successive approximation A/D converter routine in machine code to speed up the A/D conversion about eight times, I do have a clue what I'm talking about.

Phone conversations are bandwidth limited to about 300-3,000 Hz in the phone anyway, recording at a higher than the Nyquist sampling rate gains you almost nothing, an 8 kHz rate is more than adequate for recording phone conversations.

http://en.wikipedia.org/wiki/Nyquist_rate

Ligyron

(7,627 posts)
6. don't jump on me but ...
Sat May 11, 2013, 09:29 AM
May 2013

here's what I don't get about people who worry the gov is spying on them in this scenario: Who's actually going to monitor all those conversations and mine all that data? Would it not take a ridiculous amount of co-ordinated manpower, filtering apps non withstanding?

Then what? Do they (whoever "they" are) send field agents out to do more follow-up investigation most of which will turn out to be useless?

Slightly aside and off-topic I realize but ...

struggle4progress

(118,280 posts)
16. If you've got 4 million conversations coming in at every minute, and you're gonna listen to
Sat May 11, 2013, 10:13 AM
May 2013

all of them as they come in, you'd need something like 12 million listeners employed in eight hour shifts: that would be one in every 25 Americans employed to do that

struggle4progress

(118,280 posts)
22. No, his claim was that all calls were recorded and stored. I'm simply doing the math here on what
Sat May 11, 2013, 10:30 AM
May 2013

it would take to actually listen to all those calls

woo me with science

(32,139 posts)
28. What a stupid argument.
Sat May 11, 2013, 12:27 PM
May 2013

Why on earth would you try to argue that they need to listen to ALL of them, in order for their accessibility to be abused?

struggle4progress

(118,280 posts)
36. I didn't argue any such thing: I've actually pointed out that there are too many to listen to
Sat May 11, 2013, 01:27 PM
May 2013

and also that the vast majority are simply junk from a law enforcement or intelligence perspective -- which strongly suggesrts nobody in their right mind would archive all US phone calls even if they could, regardless of whatever crackpot ideas GG has

woo me with science

(32,139 posts)
37. How naive.
Sat May 11, 2013, 01:28 PM
May 2013

Of course those in search of power and wealth, and interrupting challenges to that power and wealth, will try to do it if they can. The potential uses are unfathomable, and the practical and technical obstacles are constantly decreasing.

Your comment reminds me of people who said TV would never catch on.

struggle4progress

(118,280 posts)
43. It is of course true, and always has been true, that there are unprincipled people
Sat May 11, 2013, 02:10 PM
May 2013

who will try to augment their power and riches by any means available, and there are no doubt people who would like to sell the idea that we should collect and store all electronic communications: that doesn't mean we are collecting and storing all electronic communications

cui bono

(19,926 posts)
56. So where is the proof? All you offer is speculation and with it you are smearing a good name.
Sat May 11, 2013, 03:42 PM
May 2013

You don't even accept that he's using a former FBI agent's info.

Come back when you have hard evidence that Greenwald is wrong, or at least change your post heading and edit the OP. But this witch hunt has got to stop. It's beneath us. Or at least it's supposed to be.

struggle4progress

(118,280 posts)
86. GG's PoV seems to be that if a "former FBI agent" suggests the Administration
Sun May 12, 2013, 04:13 AM
May 2013

is engaged in a massive illegal circumvention of the Constitution prohibition of warrantless searches and seizures -- in the form of a constant ongoing effect to intercept and record and store the millions and millions of electronic communications that occur between people in the US every minute of every day -- then we must believe the claim

Never mind that most Americans would loudly and vocally oppose such a scheme or that most Senators and Representatives would stand up to oppose it. Never mind that the courts would find such a scheme unconstitutional. Never mind that there's no statutory authorization for such a scheme. Never mind that in many cases wiretap warrants to intercept electronic communications are currently unenforceable due to technical limitations. Never mind that most telecom companies have been complaining for years that its too difficult and expensive for them to provide access to even a small fraction of a percent of electronic communications. Never mind that conspiracy theorists have been making this claim for at least thirty years.

Nope. If a "former FBI agent" suggests "the government" is intercepting and recording and storing all electronic communications in the US, that's good enough for GG

cui bono

(19,926 posts)
88. You'll never find "most Americans" loudly and vocally oppose anything political. Most Americans
Sun May 12, 2013, 05:06 AM
May 2013

aren't paying attention. And that goes double for Dems because most of them got complacent when we elected a Dem president.

As to Senators and Reps, most of them voted for the Patriot Act. And they voted to abdicate their responsibility to be the ones who can legally declare war. You really trust them to protect our rights?

And for it to get to the courts there has to be a lawsuit filed. But how does one get the proof to use in the case? The telecoms are not allowed to disclose that information due to the Patriot Act. Do you have links about telecom companies making the complaints you are claiming? There were articles about how AT&T had an entire back room devoted to the warrantless wiretapping storage irrc.
I would like to see links for all the claims you just made.

And again you bring up conspiracy theorists. What does that have to do with anything and why do you keep wanting to pretend that there are no conspiracies and that a conspiracy theory is a bad thing? Ever hear of Michael Parenti? His term is conspiracy actuality. And he's no slouch. Give him a listen sometime.


Ligyron

(7,627 posts)
89. Of course it's not OK!
Sun May 12, 2013, 08:20 AM
May 2013

It's never OK!!

I'm simply asking about the enormous technical and logistical difficulties these agencies would face in doing so.

struggle4progress

(118,280 posts)
40. Well, Tim Clemente is getting lots of attention for making this claim, but actually
Sat May 11, 2013, 01:50 PM
May 2013

its nothing new: conspiracy theorists having been making the claim in various forms for decades and there's still no reason to believe that it's true

cui bono

(19,926 posts)
53. OMG. And DU posters are making claims that it isn't true and there's still no reason to believe
Sat May 11, 2013, 03:37 PM
May 2013

them. You haven't proven anything and based on the facts you haven't even instilled doubt. Some of us read and don't just speculate to fit our preconceived ideas about what someone would or wouldn't do. And as you can see by the responses, some people on here actually have the technical knowledge to disprove your OP, but don't let that stop you. Keep trying to smear one of the best sources for information out there.

Seriously, playing the conspiracy theory card??? That just shows you got nothing.



struggle4progress

(118,280 posts)
84. There are currently something like 14K FBI employees. The FBI has no statutory authority
Sun May 12, 2013, 03:28 AM
May 2013

to intercept and store all electronic communications occurring in the US. And many persons (across the entire political spectrum) would materially support court challenges such activity, whether or not the activity were pursuant to any statute

If Tim Clemente has any actual evidence that all electronic communications in the US are intercepted and stored, he therefore has an opportunity to become a hero to millions and millions of Americans -- including anarchists and libertarians, anti-regulatory telecom executives and anti-government conservatives, civil liberties lawyers and privacy activists, anybody who ever illegally smoked pot, anybody who was ever the target of government scrutiny for their political beliefs ... the list is almost endless

If Tim Clemente has any actual evidence, he could have most of the country up in arms: almost every politician in the US would stand up to denounce such spying

No. He's just a former FBI agent who's getting attention by making grandiose claims

cui bono

(19,926 posts)
87. And you're refuting what the FBI agent says based on your ideas of what is possible digitally which
Sun May 12, 2013, 04:57 AM
May 2013

has been shown to be incorrect in posts above.

It's not like govt agencies never do anything illegal. It happens all the time. And do you really think that just because one has evidence of something that they could have most of the country up in arms? We all know that's not true.

I read years ago that Obama increased the illegal warrantless wiretapping. I thought it was common knowledge here, albeit with that loyal group always denying that he does anything wrong.

Not to mention, you feel the need to turn your disagreement with Greenwald into a smearfest with your sensationalist headline. I won't tell you what group of people you sound like when you do that, I'm not allowed to. After that display you have to work extra hard to get my attention as you've discredited yourself a lot with it. I don't understand the smear Greenwald campaign on DU. This site is so ridiculous sometimes.

cui bono

(19,926 posts)
58. We've been hearing about this for years. Not sure why the OP is freaking out about the reporting now
Sat May 11, 2013, 03:44 PM
May 2013

except for their agenda to smear Glenn Greenwald.

dlwickham

(3,316 posts)
29. he's giving Alex Jones a run for his money
Sat May 11, 2013, 12:38 PM
May 2013

maybe we should go back to using tin cans and string because the government can take pictures of smoke signals

leveymg

(36,418 posts)
30. The OP is baseless slander.
Sat May 11, 2013, 12:50 PM
May 2013

Read the thread.

Whatever you might think of Greenwald, the criticism offered by SFP is ill-informed nonsense. The USG certainly does have the technical capability to record and store everyone's phone, email, and internet records, or soon will.

ucrdem

(15,512 posts)
38. But they don't have the legal authority.
Sat May 11, 2013, 01:39 PM
May 2013

That's not to say it can't be done, or isn't being done illegally by a private contractor or some other party, but the USG doesn't as far as I know have the legal authority to conduct this kind of 24/7 dragnet. So basically Greenwald is accusing the Obama administration of flagrant criminality. Personally I don't buy it without something more substantial than Glenn's say-so on CNN.

struggle4progress

(118,280 posts)
39. Your theory requires the view that ISPs, landline companies, and wireless communication companies
Sat May 11, 2013, 01:46 PM
May 2013

will all voluntarily consent to the wholesale collection of their customers' messages, without search warrant and in violation of existing law

18 USC § 2518 - Procedure for interception of wire, oral, or electronic communications
(1) Each application for an order authorizing or approving the interception of a wire, oral, or electronic communication under this chapter shall be made in writing upon oath or affirmation to a judge of competent jurisdiction and shall state the applicant’s authority to make such application ...
http://www.law.cornell.edu/uscode/text/18/2518

18 USC Chapter 121 - STORED WIRE AND ELECTRONIC COMMUNICATIONS AND TRANSACTIONAL RECORDS ACCESS ...
18 USC § 2703 - Required disclosure of customer communications or records
(a) Contents of Wire or Electronic Communications in Electronic Storage.— A governmental entity may require the disclosure by a provider of electronic communication service of the contents of a wire or electronic communication, that is in electronic storage in an electronic communications system for one hundred and eighty days or less, only pursuant to a warrant issued using the procedures described in the Federal Rules of Criminal Procedure (or, in the case of a State court, issued using State warrant procedures) by a court of competent jurisdiction ...
18 USC § 2712 - Civil actions against the United States
(a) In General.— Any person who is aggrieved by any willful violation of this chapter or of chapter 119 of this title or of sections 106(a), 305(a), or 405(a) of the Foreign Intelligence Surveillance Act of 1978 (50 U.S.C. 1801 et seq.) may commence an action in United States District Court against the United States to recover money damages ...
http://www.law.cornell.edu/uscode/text/18/part-I/chapter-121

leveymg

(36,418 posts)
44. Installation of the intercept equipment by telcos has been mandatory under CALEA since 1994
Sat May 11, 2013, 02:25 PM
May 2013

What the government does with the data after it's diverted is still largely classified. Those who've tried to open it up in court have run into a brick wall of secrecy, national security, and plain old obstruction, but enough details have emerged, nonetheless, to confirm that mass warrantless wiretapping has been a reality for at least a decade. As you will see when and if you start sifting through the available information on the subject, is not that there aren't enough facts available, but that there is too much for most people to analyze for themselves, so that most people don't even try.

But, if you want to start educating yourself (and stop making ignorant statements on topics you clearly haven't even started to try to understand), here's a good place to start. See, Electronic Privacy Information Center, http://epic.org/privacy/nsa/foia/



Background

In December 2005, the New York Times reported that President Bush secretly issued an executive order in 2002 authorizing the National Security Agency to conduct warrantless surveillance of international telephone and Internet communications on American soil. President Bush acknowledged the existence of the NSA surveillance program and vowed that its activities would continue.

< . . .>

Senate to Investigate NSA "Overcollection": Senator Dianne Feinstein has announced that the Senate Intelligence Committee will hold a hearing on the National Security Agency's interception of phone calls and private e-mail messages of Americans. Recently, the New York Times reported that the NSA's activities went beyond the legal limits established by the Congress last year. EPIC has a related lawsuit asking a federal court to force the release of memos on the legal authority for domestic surveillance of American citizens. For more information, see EPIC's page on Freedom of Information Act Work on the National Security Agency's Warrantless Surveillance Program. (Apr. 17, 2009)

EPIC v. DOJ - EPIC Urges Court to Require Disclosure of Warrantless Wiretap Memos: EPIC, the ACLU, and the National Security Archive asked a federal court in Washington, DC to order the immediate disclosure of government memos describing the legal basis for the warrantless wiretapping of American citizens by the Bush Administration. The court is currently reviewing the documents, prepared by the Office of Legal Counsel, as part of an EPIC Freedom of Information Act lawsuit. This week, the Attorney General released several related memos, which previously were secret. The new release follows President Obama's recent statement on government transparency. "The Freedom of Information Act should be administered with a clear presumption: In the face of doubt, openness prevails," the President said. For more information, see EPIC v. DOJ. (Mar. 6, 2009)

Justice Department Releases Domestic Surveillance Memos and Opinions: Attorney General Eric Holder announced that the Department of Justice will make public memos and opinions concerning warrantless surveillance, and other controversial claims of Presidential authority, that were prepared in the wake of 9/11. The documents describe the legal basis for President Bush's domestic surveillance program. After learning of the warrantless wiretap program, EPIC sued the Department of Justice under the Freedom of Information Act to compel disclosure of legal memos concerning the program. Government lawyers subsequently disavowed the justifications for the warrantless surveillance. For more, see EPIC's "National Security Agency's Warrantless Surveillance Program" page. (Mar. 3, 2009)

In EPIC Case for Wiretap Memos, Federal Judge to Review Justice Department Documents. In EPIC v. DOJ, EPIC, the ACLU, and the National Security Archive are seeking government documents regarding the President's warrantless wiretapping program. Today, a federal court ordered the Department of Justice to provide for inspection copies of legal memos authored by government lawyers. The opinions, prepared by the Office of Legal Counsel, provided the legal basis for the President to wiretap US citizens in the United States without court approval. EPIC began the Freedom of Information Act lawsuit in December 2005, after the New York Times first reported the details of the wiretap program. (Oct. 31)

EPIC Urges Court to Assess Government Secrecy Claims in Domestic Surveillance Case. EPIC, in a joint brief (pdf) with the American Civil Liberties Union and the National Security Archive, asked a federal court to order the Department of Justice to produce legal opinions that were prepared to justify the President's domestic surveillance program. The brief renews EPIC's request that a federal judge review the documents held by the agency and determine whether they should be kept from the public. In December 2005, immediately after press reports uncovered the President's surveillance program, EPIC requested the legal opinions that were prepared to justify the program. The government refused to produce many key documents, and EPIC sued under the Freedom of Information Act. (February 6, 2008)

EPIC v. DOJ: Court Rejects Secrecy Claims in FOIA Case. A federal district court today ordered (pdf) the Department of Justice to be more forthcoming about the basis for withholding documents concerning the President's domestic surveillance program. In December 2005, immediately following the press report of the program, EPIC requested legal opinions and related documents that were prepared to justify and monitor the warrantless surveillance program. The American Civil Liberties Union and the National Security Archive also submitted FOIA requests. A federal court has now ruled in EPIC v. DOJ (pdf) that the Department of Justice's basis for withholding the documents were "too vague and general," and that the FBI's justifications, in particular, are "wholly inadequate." By October 12, the Department must provide far more detailed information to EPIC and the other plaintiffs. (Sept. 5, 2007)

In Court Filing, EPIC Urges Release of Documents Concerning the NSA Surveillance Program In papers (pdf) filed in Washington, DC, today, EPIC, the American Civil Liberties Union, and the National Security Archive urged a federal district court to require the Justice Department to disclose documents about the NSA Domestic Surveillance program. The motion follows the testimony (pdf) of former Deputy Attorney General James Comey before the Senate Judiciary Committee that indicated that top officials at the Department of Justice believed that the program was illegal. EPIC first sought documents regarding the legal basis for the program just hours after the warrantless surveillance program was first reported in the New York Times in December 2005. (March 23, 2007)

ucrdem

(15,512 posts)
46. Doesn't most or all of this pertain to Bush-era domestic surveillance
Sat May 11, 2013, 02:51 PM
May 2013

the illegality of which is not exactly in dispute? What's in dispute here is Greenwald's claim that all domestic electronic communications are currently collected and stored by some US agency. My personal view is that Obama and Holder are sticklers for legality and unlike the previous administration make sure their legal i's and t's are dotted and crossed before they do anything. So I don't see them taking over this particular boondoggle unless it has in fact been made legal. But I defer to SFP.

leveymg

(36,418 posts)
49. The 2008 Amendments to FISA legalized most of it.
Sat May 11, 2013, 03:14 PM
May 2013

That, of course, doesn't make Greenwald's statement crackpot. Quite the opposite.

What's shocking is how little many Americans seem to know about how little privacy they have left in their communications, and how much data the gov't, in fact, has compiled on each of us. Some here apparently don't even want us to raise questions about this, and will try to destroy the reputations of those, like GG, who dare criticize these programs and how they are managed.

I see no reason to defer to SFP, who doesn't seem to understand that questioning gov't authority is a civic virtue, and doing it well, as Greenwald does most of the time, is a gift. On the other hand, uninformed hostility toward critics is no virtue, but some still confuse it with loyalty.

ucrdem

(15,512 posts)
55. It legalized some FISA warrants, sure,
Sat May 11, 2013, 03:41 PM
May 2013

but nothing like wholesale warrantless domestic surveillance on the scale Greenwald is suggesting, or any scale. FISA is limited to foreign intelligence gathering and still requires warrants to wiretap American citizens, even after the 2008 amendments. From the 2012 re-authoritarian:

“On July 10, 2008, President Bush signed into law the FISA Amendments Act of 2008, which passed with a bipartisan majority of Congress and broad support from the intelligence community. The Act allows intelligence professionals to more quickly and effectively monitor terrorist communications, while protecting the civil liberties of Americans. Among other things, the law accomplishes the following:

(snip)

“Requires court orders to target Americans for foreign intelligence surveillance, no matter where they are, and requires court review of the procedures used to protect information about Americans.

http://www.gop.gov/bill/112/2/hr5949


What Greenwald is suggesting is a whole different ballgame.

leveymg

(36,418 posts)
69. Do you know how a "driftnet" warrant of the type legalized by the '08 Amendment works?
Sat May 11, 2013, 04:01 PM
May 2013

It basically says, we have an interest in Individual X who lives in city Y, who calls non-US person Z. We want a FISA warrant, and in 100% of the cases last year, it was granted. The driftnet scoops up all the phone or email messages that go through the servers in that city or area.

By the way, FISA has a host of exceptions to warrant requirements for a number of exigent circumstances, including serious crimes not connected to terrorism or national security. I know, that normally requires a Title III warrant, but go figure.

That's wholesale domestic surveillance, and it's now legal. That's the new ballpark, my friend.

ucrdem

(15,512 posts)
70. Apparently it's not so legal.
Sat May 11, 2013, 04:25 PM
May 2013

It seems the issue issue been brought before the Supreme Court twice this year, first by the ACLU, in a case dismissed in February, and currently by the EFF, in case that I can't figure out the status of. Maybe you know more about it:

https://www.eff.org/cases/jewel

Both evidently originated in 2008 in response to Bush-era surveillance. So whether or not there's a technical capacity to rig up a total domestic surveillance system based on FISA warrants, the legality of doing it is by no means settled, and that being the case, I have no reason to think Obama and Holder are doing it absent hard evidence.

Fumesucker

(45,851 posts)
73. Are you trying to say that America has never tortured anyone?
Sat May 11, 2013, 04:40 PM
May 2013

Because we all know that's not true and we all know that no one been prosecuted for that torture.

leveymg

(36,418 posts)
74. The EFF suit against Bush Admin officials is still alive, the ATT case was dismissed.
Sat May 11, 2013, 04:43 PM
May 2013

The Jewel v NSA suit alleges that the Bush Admin broke the pre-2008 law. Telco immunity, which sank the Hepting v ATT case doesn't apply to gov't officials.


What is the status of Hepting v. AT&T case?

In June of 2009, a federal judge dismissed Hepting and dozens of other lawsuits against telecoms, EFF appealed that decision but it was affirmed, and in October, 2012, the Supreme Court declined to hear the case.


You have problems believing that NSA conducts driftnet surveillance, even though its been legal for 4 years? Why would they cease after the 2008 Amendment? Why else would they need huge new server farms if they aren't, in fact, pulling in huge amounts of data?

ucrdem

(15,512 posts)
76. Okay thanks.
Sat May 11, 2013, 04:54 PM
May 2013

That being the case I don't think Obama and Holder would authorize any TIA at least until that case is settled, and I doubt they will ever. It doesn't strike me as a project they'd waste time and money on.

leveymg

(36,418 posts)
80. The point is, TIA never stopped. It just grew into the current NSA program.
Sat May 11, 2013, 06:11 PM
May 2013
http://www.wbez.org/worldview/2012-04-12/segment/new-data-center-nsa-poised-achieve-%E2%80%98total-information-awareness%E2%80%99-98172
With new data center, NSA poised to achieve ‘total information awareness’
April 12, 2012
(Photo courtesy of Creative Commons)
When it opens, the Utah Data Center will be five times the size of the U.S. Capitol.

The National Security Agency has poured more than $1.2 billion into their newest project: the country's biggest "spy center." Slated to open in 2013 in Bluffdale, Utah, the Utah Data Center will function as a bottomless database warehouse that will record, store and monitor any electronic communication -- from grocery store receipts to diplomatic transactions. The goal is to provide "total information awareness" for the U.S. government. Journalist James Bamford exposed the most detailed account to date of the center's structure and mission in last month's Wired magazine. He joins Worldview to talk about the NSA's ambitions and what it means for American security and privacy.

ucrdem

(15,512 posts)
85. Sounds like it never started.
Sun May 12, 2013, 03:39 AM
May 2013

Bushco spent eight years building Fortified Amerika and this sounds like one of their grand schemes. Just because BO didn't pull the plug on the construction doesn't mean they're actually going to start warrantlessly collecting and saving every grocery receipt. But if they are I have a couple of dusty shoeboxes I'll throw away.

Look at it this way: If BO and Holder were wiretapping and recording the entire nation 24/7, with or without FISA warrants or some equally dubious figleaf, I think we would have heard about it in 2010 and 2012, don't you? It's not like the noise machine has any shame, or thinks twice about blaming BO for their own crimes, see Benghazi for example.

leveymg

(36,418 posts)
93. Large core components of TIA were shifted to NSA and DNI, and never ended.
Sun May 12, 2013, 04:38 PM
May 2013
http://en.wikipedia.org/wiki/Information_Awareness_Office
Components of TIA projects that continue to be developed

Despite the withdrawal of funding for the TIA and the closing of the IAO, the core of the project survived.[3][4][25] Legislators included a classified annex to the Defense Appropriations Act that preserved funding for TIA's component technologies, if they were transferred to other government agencies. TIA projects continued to be funded under classified annexes to Defense and Intelligence appropriation bills. However, the act also stipulated that the technologies only be used for military or foreign intelligence purposes against foreigners.[26]

TIA's two core projects are now operated by Advanced Research and Development Activity (ARDA) located among the 60-odd buildings of "Crypto City" at NSA headquarters in Fort Meade, MD. ARDA itself has been shifted from the NSA to the Disruptive Technology Office (run by to the Director of National Intelligence). They are funded by National Foreign Intelligence Program for foreign counterterrorism intelligence purposes.

One technology, codenamed "Basketball" is the Information Awareness Prototype System, the core architecture to integrate all the TIA's information extraction, analysis, and dissemination tools. Work on this project is conducted by SAIC through its former Hicks & Associates consulting arm run by former Defense and military officials and which had originally been awarded US$19 million IAO contract to build the prototype system in late 2002.[27]


Going backwards, related programs included Trailblazer, an NSA program that focused on interception and analysis of data carried on web communications networks, cell phones, VOIP, and e-mail. After receiving adverse publicity Trailblazer was shutdown but reportedly morphed into the NSA Turbulance Program. Thin Thread was a rival NSA program that went operational, resulting in massive domestic surveillance. This is described by Jane Mayer in a 2011 New Yorker article: http://www.newyorker.com/reporting/2011/05/23/110523fa_fact_mayer?currentPage=all

Code-named ThinThread, it had been developed by technological wizards in a kind of Skunk Works on the N.S.A. campus. Formally, the project was supervised by the agency’s Signals Intelligence Automation Research Center, or SARC.

While most of the N.S.A. was reeling on September 11th, inside SARC the horror unfolded “almost like an ‘I-told-you-so’ moment,” according to J. Kirk Wiebe, an intelligence analyst who worked there. “We knew we weren’t keeping up.” SARC was led by a crypto-mathematician named Bill Binney, whom Wiebe describes as “one of the best analysts in history.” Binney and a team of some twenty others believed that they had pinpointed the N.S.A.’s biggest problem—data overload—and then solved it. But the agency’s management hadn’t agreed.

Binney, who is six feet three, is a bespectacled sixty-seven-year-old man with wisps of dark hair; he has the quiet, tense air of a preoccupied intellectual. Now retired and suffering gravely from diabetes, which has already claimed his left leg, he agreed recently to speak publicly for the first time about the Drake case. When we met, at a restaurant near N.S.A. headquarters, he leaned crutches against an extra chair. “This is too serious not to talk about,” he said.

Binney expressed terrible remorse over the way some of his algorithms were used after 9/11. ThinThread, the “little program” that he invented to track enemies outside the U.S., “got twisted,” and was used for both foreign and domestic spying: “I should apologize to the American people. It’s violated everyone’s rights. It can be used to eavesdrop on the whole world.” According to Binney, Drake took his side against the N.S.A.’s management and, as a result, became a political target within the agency.


Binney described Thin Thread to Mayer, who describes The Program this way:

ThinThread would correlate data from financial transactions, travel records, Web searches, G.P.S. equipment, and any other “attributes” that an analyst might find useful in pinpointing “the bad guys.” By 2000, Binney, using fibre optics, had set up a computer network that could chart relationships among people in real time. It also turned the N.S.A.’s data-collection paradigm upside down. Instead of vacuuming up information around the world and then sending it all back to headquarters for analysis, ThinThread processed information as it was collected—discarding useless information on the spot and avoiding the overload problem that plagued centralized systems. Binney says, “The beauty of it is that it was open-ended, so it could keep expanding.”

Pilot tests of ThinThread proved almost too successful, according to a former intelligence expert who analyzed it. “It was nearly perfect,” the official says. “But it processed such a large amount of data that it picked up more Americans than the other systems.” Though ThinThread was intended to intercept foreign communications, it continued documenting signals when a trail crossed into the U.S.
< . . .>
Binney, for his part, believes that the agency now stores copies of all e-mails transmitted in America, in case the government wants to retrieve the details later. In the past few years, the N.S.A. has built enormous electronic-storage facilities in Texas and Utah. Binney says that an N.S.A. e-mail database can be searched with “dictionary selection,” in the manner of Google. After 9/11, he says, “General Hayden reassured everyone that the N.S.A. didn’t put out dragnets, and that was true. It had no need—it was getting every fish in the sea.”



In addition to NSA's Thinthread and Trailblazer programs, DIA operated its own domestic-focused pre-9/11 surveillance program. At least one of these monitored AQ cells operating inside the US. The story of the Able Danger has been well documented and fairly widely known. Able Danger was shut down by Undersecretary of Defense for Intelligence Steve Cambone in late 2000-early 2001. That operation built up a social network analysis focusing electronic surveillance on members of the so-called Brooklyn Cell that had remained in place after its establishment by CIA as part of Operation Cyclone, the Agency's operation that recruited and trained Jihadis for war against the Soviets in Afghanistan. Much of bin Laden's organization grew out of the US and Saudi organized covert operations against the Russia and its allies in Central Asia and the oil-rich region of the Transcaucasus that flared up again as wars in Bosnia, Kosovo, Dagestan, and Chechnya.

As early as 1997, as the US and Saudi paramilitary organized by bin Laden were cooperating in Kosovo, US intelligence officials were bragging that they had "mapped out" bin Laden's financial and donor network by human and technical means. Al Qaeda was already the focus of multiple surveillance operations inside the US, but this was gravely complicated by intelligence agencies from several countries stumbling over each other and the fatal duplicity of their network of interwoven double-agents that created the opportunity for the 9/11 attacks.

NSA and DIA technical collection and analysis, elsewhere referred as "The Program" survived the reorganization of intelligence that followed 9/11, and the closing down some legacy programs that followed a series of disclosures and scandals involving "The Program" in the middle of the decade. Part of this was described by Shane Harris in The National Journal, "TIA LIVES ON":
February 23, 2006, http://nationaljournal.com/about/njweekly/stories/2006/0223nj1.htm or http://mediachannel.org/blog/node/3509

As early as February 2003, the Pentagon planned to use Genoa II technologies at the Army's Information Awareness Center at Fort Belvoir, Va., according to an unclassified Defense budget document. The awareness center was an early tester of various TIA tools, according to former employees. A 2003 Pentagon report to Congress shows that the Army center was part of an expansive network of intelligence agencies, including the NSA, that experimented with the tools. The center was also home to the Army's Able Danger program, which has come under scrutiny after some of its members said they used data-analysis tools to discover the name and photograph of 9/11 ringleader Mohamed Atta more than a year before the attacks.

The other project has been re-designated "TopSail" (formerly Genoa II) and would provide IT tools to help anticipate and preempt terrorist attacks. SAIC has also been contracted to work on Topsail, including a US$3.7 million contract in 2005.

ucrdem

(15,512 posts)
94. It looks like TIA was D.O.A. by 2004
Sun May 12, 2013, 05:18 PM
May 2013

per your link:

As a result House and Senate negotiators moved to prohibit further funding for the TIA program by adding provisions to the Department of Defense Appropriations Act, 2004 (signed into law by President Bush on October 1, 2003). Further, the Joint Explanatory Statement included in the conference committee report specifically directed that the IAO as program manager for TIA be terminated immediately.


And the parts that live on, and will probably take up residence in that Utah facility, are dedicated to foreign surveillance and cyberwarfare, which by all accounts we're champs at. Could a dastardly White House turn all this bandwidth into a domestic digital vacuum cleaner? Maybe but I doubt that Obama would consider doing it illegally, and if Bush couldn't get the NYT and Congress on board in 2003 how would Obama now?

leveymg

(36,418 posts)
96. Like ThinThread, the technologies TIA incubated migrated to other programs
Mon May 13, 2013, 08:38 AM
May 2013

Regardless of what program code names have been changed to, the capabilities to carry out mass surveillance and profiling developed by these earlier programs are still in use and have been vastly expanded, albeit it may be that the current programs have returned some of the safeguards originally intended by Binney, Drake and others at NSA who were sacrificed in the process of bringing this thing to life.

The Utah facility due to open in September, and other server farms currently operating around the country are testaments to the fact that TIA and Thin Thread are very much alive. The 2008 FISA Amendment legalized much of The Program, the Bush era domestic surveillance, guaranteeing the growth of these programs.

The most alarming aspect of all this is that despite the fact that the NSA effectively picks up on and "maps out" terrorist groups and support networks operating inside the US, other IC agencies continue to run them as assets, granting known terrorists virtual immunity. That is the fatal flaw in the system that has led to every major "intelligence failure" resulting in a terrorist attack in the U.S. from 9/11 through the Boston Bombing.

struggle4progress

(118,280 posts)
75. You haven't understood what CALEA does:
Sat May 11, 2013, 04:50 PM
May 2013

47 USC § 1002
http://www.law.cornell.edu/uscode/text/47/1002
says carriers must be able to isolate and deliver to law enforcement packets covered by a court order
47 USC § 1003
http://www.law.cornell.edu/uscode/text/47/1003
says the AG shall establish what that means in terms of capacity requirement

See 60 FR 199 (Mon 16 Oct 95)
http://www.gpo.gov/fdsys/pkg/FR-1995-10-16/pdf/95-25562.pdf
"... Each telecommunications carrier must provide the ability to meet the capability assistance requirements
defined in section 103 of the CALEA for a number of simultaneous pen register, trap and trace, and communication interceptions equal to 0.25% of the engineered capacity of the equipment, facilities, or services that provide a customer or subscriber with the ability to originate, terminate, or direct communications..."


That's not a plan for intercepting and storing ALL communications


leveymg

(36,418 posts)
78. You're reading the 1995 regulations, ninny.
Sat May 11, 2013, 05:20 PM
May 2013

That applied to first generation CALEA compliant equipment, when dinosaurs roamed the earth with Jesus in the saddle. Some people believe they still do, apparently.

Initial Capacity Notice

On October 16, 1995, the FBI published in the Federal Register its Initial Capacity Notice. The estimates of actual and maximum capacity, expressed as a percentage of engineered capacity, were stated as follows in the Initial Capacity Notice:

ESTIMATES FOR PROJECTED MAXIMUM AND ACTUAL CAPACITY AS PUBLISHED IN THE

FEDERAL REGISTER
------------------------------------------------------------------------

Maximum Actual
------------------------------------------------------------------------
Category I...................................... 1.00% 0.50%
Category II..................................... 0.50% 0.25%
Category III.................................... 0.25% 0.05%
------------------------------------------------------------------------

http://privacy.org/cgibin/mt/mt-search.cgi?IncludeBlogs=14,10,14&tag=wiretapping&limit=50


Try this one, instead:

OIG Audit Report 06-13 - Department of Justice
http://www.justice.gov/oig/reports/FBI/a0613/findings.htm
Specifically, all of the wireline carriers stated that 100 percent of their post-1995 switches were CALEA-compliant,

struggle4progress

(118,280 posts)
83. I did not locate any further FR notices establishing capacity requirements. Following links from one
Sun May 12, 2013, 03:09 AM
May 2013

of your links suggests that over-all actual capacity requirements, as indicated by actual wiretap warrants do not exceed 0.001% of conversations: this, of course, strongly suggests that capacity requirement standards in the range 0.25-1.00% grossly exceed actual needs

The 2006 OIG audit information you provide is not modifying capacity requirement standards but is rather reporting what fraction of their network traffic various telecom providers report as complying with the capacity requirements: the original capacity requirement deadlines have been postponed on multiple occasions due to provider claims they could not meet the deadlines

Ichingcarpenter

(36,988 posts)
31. You need to pay attention
Sat May 11, 2013, 12:51 PM
May 2013

A hidden world, growing beyond control



These are some of the findings of a two-year investigation by The Washington Post that discovered what amounts to an alternative geography of the United States, a Top Secret America hidden from public view and lacking in thorough oversight. After nine years of unprecedented spending and growth, the result is that the system put in place to keep the United States safe is so massive that its effectiveness is impossible to determine.

The investigation's other findings include:

* Some 1,271 government organizations and 1,931 private companies work on programs related to counterterrorism, homeland security and intelligence in about 10,000 locations across the United States.

* An estimated 854,000 people, nearly 1.5 times as many people as live in Washington, D.C., hold top-secret security clearances.

* In Washington and the surrounding area, 33 building complexes for top-secret intelligence work are under construction or have been built since September 2001. Together they occupy the equivalent of almost three Pentagons or 22 U.S. Capitol buildings - about 17 million square feet of space.

* Many security and intelligence agencies do the same work, creating redundancy and waste. For example, 51 federal organizations and military commands, operating in 15 U.S. cities, track the flow of money to and from terrorist networks.

* Analysts who make sense of documents and conversations obtained by foreign and domestic spying share their judgment by publishing 50,000 intelligence reports each year - a volume so large that many are routinely ignored.



More;

http://projects.washingtonpost.com/top-secret-america/articles/a-hidden-world-growing-beyond-control/1/

ucrdem

(15,512 posts)
41. Doesn't corroroborate Greenwald's claim
Sat May 11, 2013, 02:05 PM
May 2013

at least that I could find. I just skimmed all 7 pages and nowhere found a claim that all US domestic digital communications are collected and stored by any US agency. The point of the article seems to be that since 911, many different agencies are collecting many kinds of info under various legal pretexts but that they aren't coordinated.

struggle4progress

(118,280 posts)
42. Oh, the "national security" establishment is certainly out of control: it's been oversized
Sat May 11, 2013, 02:06 PM
May 2013

and without meaningful oversight for decades:

The Washington Post
December 22, 1963 - page A11
Harry Truman Writes:
Limit CIA Role To Intelligence
... We have grown up as a nation, respected for our free institutions and for our ability to maintain a free and open society. There is something about the way the CIA has been functioning that is casting a shadow over our historic position and I feel that we need to correct it.
http://www.maebrussell.com/Prouty/Harry%20Truman%27s%20CIA%20article.html

And the situation today is no doubt far worse than when Truman had his second thoughts fifty years ago

But one can believe that we ought to limit our "black budgets" and the size of the US "national security" establishment, and that we should try to restrict the secrecy that veils all that, without buying into GG's crackpot notion that "the government" illegally intercepts and stores all electronic communications

 

byeya

(2,842 posts)
45. Redundancy and waste are only two of the three threats: There is institutional jealousy and lust for
Sat May 11, 2013, 02:41 PM
May 2013

funding that sometimes runs amok and chews up innocent people so the agency can parade before Congress and show off the latest example of "how they've saved American lives"
Don't undersell this angle; it's behind the old stories of the fbi not sharing information with the cia and vice versa; now you've got 50+ agencies in the mix all wanting a bigger appropriation and not caring all that much how they get it.

struggle4progress

(118,280 posts)
48. Institutional competitiveness can be a good thing for democracy: it means none of them
Sat May 11, 2013, 03:09 PM
May 2013

have complete control

 

byeya

(2,842 posts)
57. I think having 2, or more, agencies with the same mission is a waste of money and Cabinent
Sat May 11, 2013, 03:42 PM
May 2013

level staff should be monitoring the one agency for compliance with both the law and with the president's policies.

cui bono

(19,926 posts)
52. UNREC - Wow. The Glenn Greenwald smearing is truly disgusting.
Sat May 11, 2013, 03:23 PM
May 2013

If you can't stand the criticism, then pressure your idol stop doing things worthy of it.

And... you might consider not using slimy tactics such as putting up a sensationalist smear headline to grab attention and put propaganda in people's heads so that it keeps being visible and even those who don't read the post have had it planted subliminally.

Seriously, Glenn Greenwald a crackpot??? Have you really ever read his articles? Heard him speak? I doubt it.

 

AtomicKitten

(46,585 posts)
59. or ...
Sat May 11, 2013, 03:44 PM
May 2013

Perhaps it is Glenn Greenwald that is perceived as an idol by some here with the requisite bowing and scraping, his words devoured by the adoring masses as rock-solid gospel.

hmmmm

 

AtomicKitten

(46,585 posts)
67. okay
Sat May 11, 2013, 03:56 PM
May 2013
Seriously, Glenn Greenwald a crackpot??? Have you really ever read his articles? Heard him speak? I doubt it.


It appears critical analysis has been compromised by blind devotion as in if it's Glenn Greenwald it must be true. Right?

GG has his own skeletons, not the least of which was supporting the Iraq war.

Nobody is perfect nor are they always right, and that includes both Mr. Greenwald and the president.

leveymg

(36,418 posts)
63. You hit it on the head. This post is propaganda that works with other similar smears on this board
Sat May 11, 2013, 03:50 PM
May 2013

to reinforce an image in people's heads. Propaganda is directed at those who aren't really paying attention to create an attitude toward a subject - in this case, the agenda is to paint GG, an intelligent critic of this Administration, as a Conspiracy Theorist.

Of course, the smear of gov't critics is itself a form of conspiracy theoriorizing, and one of the most common because it requires little or no creativity or effort at all beyond the assertion that someone is "crazy" or a "crackpot". The post above is an illustrative example of that sad fact.

JDPriestly

(57,936 posts)
77. Eventually, someone will take the government to court to find out.
Sat May 11, 2013, 05:09 PM
May 2013

If our government refuses to allow the suit to be heard due to national security concerns, we can assume it is true enough to be a problem even if not absolutely true, even if the eavesdropping is not as extreme as some fear.

If they deny it and produce all records from any agency that might be recording or keeping some sort of electronic data on all this stuff, then we can believe what we want. Because it would be hard for the government to disprove the accusation in some way that people could trust.

In any case, we will probably never know.

But one thing is for certain, we cannot assume that they are not recording everything. I would like to see a lot more openness about what their capacity is in this regard.

One tip: if you need to talk to a lawyer, go to the lawyer's office. Let's say a business deal has gone sour and you don't know what your rights are. It might be something very personal. It might be something that happened that you fear may be illegal. Make sure that big brother is not listening. You need to be able to feel that you are sharing a confidence. The same might be true of a doctor or some other professional with whom you wish to have a confidential discussion. Just be careful. You never know.

Just the claim that the government MIGHT be eavesdropping on all electronic communications chills speech. That is the danger. The public needs to know whether we should watch what we say.

I'm way beyond the time of life in which this could possibly be a problem for me, but the thought has occurred to me that we are approaching a time what with cameras and eavesdropping equipment everywhere that marital infidelity is no longer possible. The scandals we will read about as we approach that point should be pretty juicy. I suppose that the phones of seedy motels would be tapped by this huge program (real or not). Could make for some spicy sounds.

Anyway, knowing human nature as I do, I suspect the guys doing the eavesdropping, sorting through all the junk (most calls are to and from family members or business associates, aren't they? Boring, boring, boring), there will probably be a huge leak of a giant file of pornographic noises and shocking telephone calls. Who knows? It may be the art of the future. Anyway, comedians should have a lot of fun with this.

If the project exists, it is a huge waste of money.

struggle4progress

(118,280 posts)
79. The old left activists I knew decades ago would never discuss any concrete plans over the phone
Sat May 11, 2013, 05:23 PM
May 2013

These were people from a variety of backgrounds. One had fled Nazi Germany as a teenager. They didn't believe the government was wiretapping everybody. But they did think that if you challenged certain aspects of the status quo and began to win, the status quo would try to shut you down however it could. Those folks had years of stories to back up their beliefs too

JDPriestly

(57,936 posts)
81. I used to know a guy whose job it was to censor the Western press in Poland.
Sat May 11, 2013, 11:43 PM
May 2013

He was among the first to leave Poland. He was just too disgusted with the controlling big-brother state and figured a way out.

We are not there yet, but we are headed toward a complete disconnect between government, the national security aspect of it, and the people of our country. It will take time, but unless the government begins to understand that it is working for the people and not the people for it, that it should be more open with the people and less interested in the people's personal lives and opinions and activities, we will reach the point at which the government simply falls apart under its own weight.

To do all the surveillance, etc. that is needed to keep so many secrets and snoop on so many people, you need a pretty big bureaucracy. The STASI in E. Germany and the related agencies got so large, they nearly overwhelmed everything.

That could happen here unless we have real reform in our national security system. We do need national security. We need also to watch that it does not overwhelm everything else as it eventually did in E. Germany.

leveymg

(36,418 posts)
99. The Stellar Wind program appears to have been an offshoot of Thin Thread. See #93, 96 above.
Mon May 13, 2013, 09:38 AM
May 2013

Binny was a principal manager of Thin Thread, and he left the agency in later 2001 after it became clear that the Bush Admin. was operating the program without safeguards that would have protected specific US persons communications and data collected from warrantlless analysis and investigation. You should read Jane Mayers' excellent New Yorker article linked above.

Latest Discussions»General Discussion»More crackpot thinking fr...