NSA: Since Metadata is usually attached/embedded in Content, what is the procedure to separate it?

[usGovOwesUs3Trillion]

Last edited Wed Jul 31, 2013, 11:12 AM - Edit history (3)

Metadata is big news these days, and even though many more folks know about this term than did say 10 years ago, thanks to mp3 players and 'smart'-phones, most people still do not have a deeper knowledge of what it is exactly, and how it is used.

In short, it is extra data/information used to describe the primary data/information (e.g. content/file like an image, mp3, video, document, etc.)

For example, when you take a picture on your phone, not only will the software save that picture (content/file) to your computer/disk, it will also add/attach extra information (often hidden) to the image file, e.g. location, timestamp, camera settings etc. automatically, usually to help categorize it.

Same thing with our mp3 files, we have software (e.g. iTunes) that helps us manage/organize all these files, and the most popular way to do this, is by adding extra information to the mp3 file itself, e.g. title, description, genre, year, cd, artist, album, etc. The software basically uses the same technique it used with photos, it adds/attaches metadata (extra information) about the mp3 (content) to the mp3 file itself.

This information is usually embedded/hidden in the file itself, and therefore always travels with the file, otherwise it would not be of much use.

Anyways, considering all this recent news, and especially the gov primary defense, "we are only harvesting and storing metadata" (nothing to see here, move along) it got me to thinking (smell the smoke) about it... and being a software professional, who relies on metadata daily to do my job, I could not help but be concerned about how they go about separating this metadata part of the file from the main file.

And then, how do they join it back together again once they have a suspect?

Knowing what I know about people, and the government, from my 50 miraculous years on this beautiful planet, I know that they will usually take the path of least resistance, and it begs the question if they even do separate it.

Either way, I think these questions need to be understood by our leaders on the hill, and then be asked to help us get to the truth of what is really going on.

Does the government force all these private companies to generate all these custom reports of JUST metadata on all their content? And then who stores the actual content that all this metadata references, the government, or the private companies?

If you see where I am going with this, I don't think it makes sense for this separation/joining of data, as it starts to get complex very quickly considering all the disparate data from all these different companies.

I think that the government is harvesting it all up (metadata and it's content) and then analyzing the metadata to decide what actual content to take a closer look at, which of course is right there in their possession as well.

Another question that needs to be addressed is the cost of all this, how can we afford to spy on the whole world when we can not even afford to fix our roads, bridges, schools, etc. nor find full employment for all of our people?

What does DU think, are they just harvesting the metadata, and having the companies store all the content for later retrieval, or are they sucking it all up?

UPDATE: Just got a ton of docs that answers a lot...
http://www.democraticunderground.com/?com=view_post&forum=1002&pid=3378381

[RobertEarl]

1. Its a secret!

They also grab content. But they never ever never look at the content. Not until a Judge tells them they can.

And we know no one in the spy agencies ever does anything until a judge tells them they can do something.

Right?

[RC]

3. Right.That's is the name of the guy operating NSA snack bar?

[usGovOwesUs3Trillion]

32. Oh snap

And no I'm sure they would never do that.

[pkdu]

2. Yep, Verizon has to listen to all my calls every Month before they send me a bill....

Ok , so that was a little snarky , but ponder that for 30 seconds and you should be able to answer your own question .

[usGovOwesUs3Trillion]

6. Not sure what that has to do with the NSA accessing all my data?

Of course we grant these private companies access to our private metadata for billing purposes, yet it would be illegal for them to record our content, or share it, especially with the authorities, without a warrant.

So if that is your 'point' it is ludicrous, as the phone, and other communication companies, are not in the business of spying, in order to arrest, but on providing communication services and billing.

(Not to mention that they are not able to aggregate all of our digital comms from all the separate and private companies they are not affiliated with, unlike the gov)


So, the questions stand.

What is the process of separating, and then joining, the content from the metadata that the gov collects, and do they even do it?

And since the NSA is actually in the business of actually SPYING, I think we should be concerned if they even do separate the content from the metadata in the first place.

BTW: Why do legit questions, challenging government spying, ALWAYS result in juvenile snark from some folks? Especially when it does nothing to advance the discussion, and often ignorant of the topic at hand.

[pkdu]

13. Since you clearly didn't pause to ponder as I suggested , please see post #8 for a full explanation

And as to your BTW question , its because said "legit questions " have decided what what the answer is before asking...again see #8 .

[usGovOwesUs3Trillion]

16. lol - sure, while i provided more content in one post than in all yours combined, right?

BTW: See my response to number 8.

BTW: I write the instructions that the computer (desktop or network box) uses to do it's thing, so you can expect me to respond more quickly than the average bear on this topic.

[pkdu]

20. You are correct in one sense , ie that " more content" equates to

word salad copiously heaped on a starting premise that is fatally flawed.

[usGovOwesUs3Trillion]

21. So says Mr. Content Free

[GiaGiovanni]

41. Verizon does store your texts on a server and they can be

seen by employees.

[pkdu]

42. I read that on the inter tubes too , they project them onto big screens at lunchtime and all have

a big old belly laugh for an hour every day. Two hours on Fridays.

[GiaGiovanni]

43. It is technically true.

You might wish to exaggerate for the purposes of disparaging humor, however.

[randome]

4. Could information be used illegally or abusively? Of course.

Is the NSA doing that? Nothing that S&G stole points to that.

So why don't we use our time and energy to press for more transparency and less secrecy all around rather than worry about imaginary monsters under the bed?
[hr][font color="blue"][center]Don't ever underestimate the long-term effects of a good night's sleep.[/center][/font][hr]

[usGovOwesUs3Trillion]

9. Sorry, but you seem to be contradicting yourself with your snark (VIDEO)

Last edited Tue Jul 30, 2013, 11:18 PM - Edit history (1)

"So why don't we use our time and energy to press for more transparency and less secrecy all around rather than worry about imaginary monsters under the bed?"

Not to mention being ignorant of the topic at hand...

"Is the NSA doing that? Nothing that S&G stole points to that."

They have released, for the first time, documentary evidence that does in fact 'point to that'.

Why do you even think we (and the whole world) is even having these discussions right now?

On edit: This video is for all you suspicionless spying defenders out there, you are certainly in a league of your own...
http://www.buzzfeed.com/andrewkaczynski/is-this-the-most-embarrassing-interview-fox-news-has-ever-do

[randome]

10. Cite the evidence, then. Because nothing I've seen supports illegality or abuse.

[hr][font color="blue"][center]Don't ever underestimate the long-term effects of a good night's sleep.[/center][/font][hr]

[usGovOwesUs3Trillion]

14. It's all over the www... but start with The Gaurdian (LINK)

http://www.theguardian.com/world/the-nsa-files

[randome]

22. I'm not going to read every story the Guardian has published in the past month.

Cite one piece of evidence, please. That should be easy.
[hr][font color="blue"][center]Don't ever underestimate the long-term effects of a good night's sleep.[/center][/font][hr]

[usGovOwesUs3Trillion]

23. I posted a convenient link to the docs.

So, carry on being willfully ignorant, if you must... but that invaluable link wasn't for you anyway, it was meant for any concerned reader of DU.

However, your contribution of another arrogant display of ignorance was a freebie

[randome]

29. Always a pleasure!

'Night all!
[hr][font color="blue"][center]Don't ever underestimate the long-term effects of a good night's sleep.[/center][/font][hr]

[usGovOwesUs3Trillion]

40. likewise

you provide ample teachable moments, in order to refute the totalitarian talking points, and I sincerely thank you for that

[PowerToThePeople]

5. I was under the assumption they were talking about tcp/ip headers

http://linux-ip.net/gl/tcng/node37.html

The header needs to be read by firewalls and routers in order to get your packet where it needs to go. You do not have to read all of the data.

(note- I am not a network specialist)

edit- more info http://www.w3.org/DesignIssues/Metadata.html

[usGovOwesUs3Trillion]

7. That is just one aspect of digital comms (network) but even that meta data is ATTACHED to the packet

e.g. the content.

But it does help illustrate the question, what would the seperation/join process be with network data?

And then the same would apply for all the other comm collected.

[PowerToThePeople]

12. here is something

[usGovOwesUs3Trillion]

15. www.sex.com may turn some off to the video but it does show how the metadata is attached to the cont

Last edited Tue Jul 30, 2013, 11:33 PM - Edit history (1)

the video shows how the metadata is attached to the content at 1:30 mark.

Now, of course this video is just an abstraction but it does serve to illustrate well my point of how the metadata and the content it refers to, are joined-at-the-hip so to speak.

Thanks for sharing

[PowerToThePeople]

17. I found another on youtube without said text hopefully

[usGovOwesUs3Trillion]

18. nice

Thank you

[jmowreader]

8. In telephony, the metadata never really sees the content

If you have a landline, your phone is hooked to a wire that runs to a dial central office - a place that automatically connects up to 10,000 telephones to each other and to long-distance phone circuits. Inside the DCO is equipment collectively known as switchgear.

Switchgear has long been able to generate a listing of the numbers each of its lines calls, and how long the calls last. This is how they generated a long-distance telephone bill back in the days before they had "unlimited nationwide calling" plans. (Those plans came about not because the phone company was being generous but because it's cheaper to just charge you $30 a month and let you talk all you want than it is to meter every telephone call.)

Cell phone switches can do the same thing, but cell phones aren't hardwired to a node in a relay rack the way a landline is.

The switchgear metering file is the "metadata" everyone's talking about...so, in essence, the entire world has gone apeshit because NSA is reading your phone bill.

[usGovOwesUs3Trillion]

11. 1. This is not just about old land lines. 2. As noted above TCP/IP traffic does have the data attach

2. As noted above TCP/IP traffic does have the metadata attached to the content.

In fact most digital files and communication does have the metadata and content attached.

And this OP is NOT about wether or not it is possible to separate the metadata from the content (of course it is) it is about what is that process of separation, and then rejoining.

And as a follow up, do they even take the effort to do it.

And I will add another follow up... who pays for it?

[jmowreader]

24. They've separated metadata forever, in their reporting systems

If you are an intelligence analyst at a station with a reporting mission, the computers that store your reports (there are several) strip the headers off the reports and store them in databases separate from the report content. IIRC the two databases aren't on the same machine; they weren't in the late 1980s-early 1990s timeframe.

The system can also go through a report and strip it down so consumers only receive the information they're authorized to have. If Field Station Hoboken can't hold Top Secret data and Field Station Hoboken is on the distribution list for your report that's half Secret and half Top Secret, the system will redo your report to remove everything Field Station Hoboken isn't allowed to have.

This is all done automatically, no operator intervention required.

[usGovOwesUs3Trillion]

27. That isn't the question. The question is what is the process to separate metadata from the content

And then to join it back up.

And finally are they even doing that considering their history, and the recently revealed documents and statements from the NSA whistleblower.


You are discussing means/process the intelligence agency uses on it's OWN documents/content. Not external disparate content, from many different companies, and many different formats.

And as a bonus question, who is paying for all that, if it even happens in the first place?

[The Straight Story]

19. So, since it is no big deal, why not make that data fully public, and allow us to see congress's

data and all those that work in the government?

I'd like to know who some CEO's call and who in our government are using their phones for possibly illegal purposes.

Obviously some people value privacy (and there are laws against phone companies just releasing your personal data for that reason).

A phone company, or an email company, credit card, etc may well have data on you but you can choose which one to go with and they only have a tiny piece of info, with the government they collect it all and you can't change companies.

[backscatter712]

25. Even today, especially with cell phones, the telcos log your calls for billing purposes.

Which means they pull the metadata the NSA's requesting from their billing records, most likely.

Even if the billing system no longer charges per-minute/per-mile for long distance like they used to, they still log the calls - I get a copy of my calls - my metadata in my phone bill every month.

So when the NSA people flash their badges and demand the metadata, that's where the telcos keep it. At least for phone services.

[usGovOwesUs3Trillion]

36. This isn't just about phone companies

And the question isn't "is it possible" it is "what is the process and costs".

[VanillaRhapsody]

26. Meta Data is just information ABOUT information..

[usGovOwesUs3Trillion]

28. Is there an echo here?

[VanillaRhapsody]

30. Just putting it in simpler terms for the layman...

[usGovOwesUs3Trillion]

31. ok thanks

[shawn703]

33. Well in my work with cloud storage

The technology does actually keep the metadata separate from the actual location - so you would have ~10PB for example of actual data "in the cloud", and in your local gateway you would have only a tiny fraction of that in metadata. When you look at the properties of what is presented locally, it would appear that you have 10PB of data there but it's only the metadata "stubs". Once you try to access the data, it gets pulled in from the cloud and you're good to go. If the actual data isn't available, disconnected etc., you would still see the metadata but just couldn't pull down the actual data. So yes it can be separated and still be available for searching of properties.

I wasn't aware that there was a claim that the government could "reattach" the data to the metadata. When I was reading it, I had the impression that they could search the From fields of emails for people who are receiving messages from a known terrorist, then at that point they could get a FISA warrant or whatever and start collecting all the data on those people. Did they say they could go back and get emails etc., that were transmitted prior to them turning up in a search?

[usGovOwesUs3Trillion]

34. Yep, no doubt it can be done, but how is it done

Across disparate companies.

And yes, it is said that when they git a hit from metadata analysis, they can then review the content, and go back years (with more storage capability added every year).

Thanks for sharing how your company does it at a high level

[usGovOwesUs3Trillion]

35. Just got a LOT of documentary evidence here (LINK)

http://www.democraticunderground.com/?com=view_post&forum=1002&pid=3378381

Holly Shit!

[usGovOwesUs3Trillion]

37. And they are talking about how they use metadata to find the content they want to spy on

[randome]

38. That software is used to monitor foreign suspects, not citizens.

Unless someone shows that to not be the case, that is.
[hr][font color="blue"][center]Don't ever underestimate the long-term effects of a good night's sleep.[/center][/font][hr]

[usGovOwesUs3Trillion]

39. Not according to first-hand accounts, and now even more documented evidence. (LINK)

http://www.theguardian.com/world/2013/jul/31/nsa-top-secret-program-online-data

Now, some may naively believe that Big Brother would never look at their own data (all OUR digital communications, harvested, and stored by them), in their own house (the NSA, and private contractors) without a warrant but that is a STRAWMAN to the FACT that they have ALREADY broken the law by harvesting and storing all of our digital communications in the FIRST PLACE.

But it is good to have this vocal minority of totalitarian defenders on DU as it presents many teachable moments to the wider audience of folks who read DU without ever signing up, let alone posting.

Thanks for all your help in passing the word and educating we the people!