General Discussion

struggle4progress

(125,715 posts)

24. Google does process about a billion queries daily. But they all arrive at the company

Sat May 11, 2013, 10:19 AM

May 2013

in a fairly standard short text form from their webpage. And the company essentially maintains several million servers dedicated to indices and index algorithms. The company does not store webpages: it creates and stores indices. The company indexes or reindexes webpages at much slower rate. They find their webpages by following links. Most webpages are available 24/7

The problem of grabbing several billion phone calls a day and indexing them and storing them is a rather different problem

Click here to purchase valentine hearts!

Edit history

Please sign in to view edit histories.

Recommendations

0 members have recommended this reply (displayed in chronological order):

99 replies

= new reply since forum marked as read

Highlight:

More crackpot thinking from Glenn Greenwald [View all] struggle4progress May 2013 OP

Not a problem if you're NSA and have a yottabyte of storage capacity, you ninny leveymg May 2013 #1

You're overlooking the technical problems of constantly streaming 4 million conversations struggle4progress May 2013 #8

Google precesses 24 petabytes/day. That's one company. Apparently you can't grasp that reality. leveymg May 2013 #11

Google does process about a billion queries daily. But they all arrive at the company struggle4progress May 2013 #24

The scale of data is roughly the same: Google 24 petabytes/day; ATT network 19 petabytes/day leveymg May 2013 #26

Correction, Google does store web pages. check the "cache" link on most search results. bobduca May 2013 #91

Yeah, you're right. My mistake. But google doesn't save the whole history of that page: struggle4progress May 2013 #95

I think your point basically stands, but the numbers (while still large) are a bit off... Silent3 May 2013 #2

The OP is WAY, WAY off. At 60KB/minute storage for voice audio, 2.2 trillion min/yr requires leveymg May 2013 #7

It depends on the use you plan to make of the recording. It may be that by reducing quality struggle4progress May 2013 #12

Data compression is applied by the cell phones themselves... Silent3 May 2013 #18

One shouldn't confuse the radio channel signal with the conversation. When cell phone companies struggle4progress May 2013 #82

Nothing but very compressed data ever gets out of your phone into the air, however Silent3 May 2013 #90

This message was self-deleted by its author RudynJack May 2013 #3

Greenwald seems to be repeating a claim made by a former FBI counter-terrorism agent. Jim__ May 2013 #4

He indeed reports the claim, then he repeats it in manner indicating that he accepts it: struggle4progress May 2013 #14

I'll go ahead and Godwin this thread Fumesucker May 2013 #5

Google's servers process 24 petabytes per day. That would leave a lot of spare capacity to process leveymg May 2013 #10

It's largely text processing struggle4progress May 2013 #17

If you're recording conversations for intelligence analysis purposes, you want the best quality struggle4progress May 2013 #20

I wrote a voice recording app for the Commodore 128 back when it was the latest thing Fumesucker May 2013 #21

This is correct. n/t whatchamacallit May 2013 #50

don't jump on me but ... Ligyron May 2013 #6

If you've got 4 million conversations coming in at every minute, and you're gonna listen to struggle4progress May 2013 #16

That wasn't Greenwald's claim Fumesucker May 2013 #19

No, his claim was that all calls were recorded and stored. I'm simply doing the math here on what struggle4progress May 2013 #22

No one has ever claimed that all the calls are listened to by humans Fumesucker May 2013 #23

I was responding to a comment made in #6: please do read subthreads when commenting struggle4progress May 2013 #25

thanks for doing the math Ligyron May 2013 #33

What a stupid argument. woo me with science May 2013 #28

I didn't argue any such thing: I've actually pointed out that there are too many to listen to struggle4progress May 2013 #36

How naive. woo me with science May 2013 #37

It is of course true, and always has been true, that there are unprincipled people struggle4progress May 2013 #43

So where is the proof? All you offer is speculation and with it you are smearing a good name. cui bono May 2013 #56

GG's PoV seems to be that if a "former FBI agent" suggests the Administration struggle4progress May 2013 #86

You'll never find "most Americans" loudly and vocally oppose anything political. Most Americans cui bono May 2013 #88

Violating our rights is not okay, ever. n/t cui bono May 2013 #54

Of course it's not OK! Ligyron May 2013 #89

OK!!! cui bono May 2013 #92

You better pay attention to this. alarimer May 2013 #9

From what I've read, your statement is 100% correct. byeya May 2013 #13

Yup. woo me with science May 2013 #27

But I bet Glenn was behind that article!!!111 bobduca May 2013 #35

Well, Tim Clemente is getting lots of attention for making this claim, but actually struggle4progress May 2013 #40

OMG. And DU posters are making claims that it isn't true and there's still no reason to believe cui bono May 2013 #53

There are currently something like 14K FBI employees. The FBI has no statutory authority struggle4progress May 2013 #84

And you're refuting what the FBI agent says based on your ideas of what is possible digitally which cui bono May 2013 #87

We've been hearing about this for years. Not sure why the OP is freaking out about the reporting now cui bono May 2013 #58

His specialty. nt MineralMan May 2013 #15

he's giving Alex Jones a run for his money dlwickham May 2013 #29

The OP is baseless slander. leveymg May 2013 #30

But they don't have the legal authority. ucrdem May 2013 #38

Your theory requires the view that ISPs, landline companies, and wireless communication companies struggle4progress May 2013 #39

Installation of the intercept equipment by telcos has been mandatory under CALEA since 1994 leveymg May 2013 #44

Doesn't most or all of this pertain to Bush-era domestic surveillance ucrdem May 2013 #46

The 2008 Amendments to FISA legalized most of it. leveymg May 2013 #49

It legalized some FISA warrants, sure, ucrdem May 2013 #55

Do you know how a "driftnet" warrant of the type legalized by the '08 Amendment works? leveymg May 2013 #69

Apparently it's not so legal. ucrdem May 2013 #70

Torture isn't legal either Fumesucker May 2013 #71

So who has Obama tortured? ucrdem May 2013 #72

Are you trying to say that America has never tortured anyone? Fumesucker May 2013 #73

The EFF suit against Bush Admin officials is still alive, the ATT case was dismissed. leveymg May 2013 #74

Okay thanks. ucrdem May 2013 #76

The point is, TIA never stopped. It just grew into the current NSA program. leveymg May 2013 #80

Sounds like it never started. ucrdem May 2013 #85

Large core components of TIA were shifted to NSA and DNI, and never ended. leveymg May 2013 #93

It looks like TIA was D.O.A. by 2004 ucrdem May 2013 #94

Like ThinThread, the technologies TIA incubated migrated to other programs leveymg May 2013 #96

You haven't understood what CALEA does: struggle4progress May 2013 #75

You're reading the 1995 regulations, ninny. leveymg May 2013 #78

I did not locate any further FR notices establishing capacity requirements. Following links from one struggle4progress May 2013 #83

You need to pay attention Ichingcarpenter May 2013 #31

Doesn't corroroborate Greenwald's claim ucrdem May 2013 #41

Oh, the "national security" establishment is certainly out of control: it's been oversized struggle4progress May 2013 #42

Redundancy and waste are only two of the three threats: There is institutional jealousy and lust for byeya May 2013 #45

Institutional competitiveness can be a good thing for democracy: it means none of them struggle4progress May 2013 #48

I think having 2, or more, agencies with the same mission is a waste of money and Cabinent byeya May 2013 #57

The OP is the one giving Alex Jones a run for his money! n/t cui bono May 2013 #64

They tell us they are spying on us to "protect" us from tyranny. Of course, they miss the irony. Tierra_y_Libertad May 2013 #32

Glenn Greenwald never loved Obama!! 111 bobduca May 2013 #34

And just in case that doesn't work, woo me with science May 2013 #47

Haters gonna hate whatchamacallit May 2013 #51

UNREC - Wow. The Glenn Greenwald smearing is truly disgusting. cui bono May 2013 #52

or ... AtomicKitten May 2013 #59

Nope. That's definitely not it. n/t cui bono May 2013 #60

Uh huh. n/t AtomicKitten May 2013 #61

Well, back up your speculation then. n/t cui bono May 2013 #62

okay AtomicKitten May 2013 #67

And that proves what, how? cui bono May 2013 #68

You hit it on the head. This post is propaganda that works with other similar smears on this board leveymg May 2013 #63

He's a racist libertarian. nt msanthrope May 2013 #66

I keep all of Glenn Greenwald's phone calls on mi iPod arely staircase May 2013 #65

Eventually, someone will take the government to court to find out. JDPriestly May 2013 #77

The old left activists I knew decades ago would never discuss any concrete plans over the phone struggle4progress May 2013 #79

I used to know a guy whose job it was to censor the Western press in Poland. JDPriestly May 2013 #81

I'm not a fan of Greenwald, but he might be right here. stevenleser May 2013 #97

The Stellar Wind program appears to have been an offshoot of Thin Thread. See #93, 96 above. leveymg May 2013 #99

Glenn Greenwald is a good person Douglas Carpenter May 2013 #98