General Discussion
Related: Editorials & Other Articles, Issue Forums, Alliance Forums, Region ForumsHow does "data may be queried only when there is a reasonable suspicion" = 20 million/month?
"NSA cannot review any metadata unless strict requirements are met, i.e., the data may be queried only when there is a reasonable suspicion, based on specific facts, that a phone number is associated with a foreign terrorist organization."
http://mashable.com/2013/08/16/nsa-file-request/?utm_medium=twitter&utm_source=dlvr.it
N.S.A. Calls Violations of Privacy ‘Minuscule’
By CHARLIE SAVAGE
WASHINGTON — The top National Security Agency official charged with making sure analysts comply with rules protecting the privacy of Americans pushed back on Friday against reports that the N.S.A. had frequently violated privacy rules, after the publication of a leaked internal audit showing that there had been 2,776 such “incidents” in a one-year period...John DeLong, the N.S.A. director of compliance, said that the number of mistakes by the agency was extremely low compared with its overall activities. The report showed about 100 errors by analysts in making queries of databases of already-collected communications data; by comparison, he said, the agency performs about 20 million such queries each month.
Mr. DeLong, speaking to reporters on a conference call, also argued that the overwhelming majority of the violations were unintentional human or technical errors and that the existence of the report showed that the agency’s efforts to detect and correct violations of the rules were robust. He said the number of willful errors was “minuscule,” involving a “couple over the past decade.”
http://www.nytimes.com/2013/08/17/us/nsa-calls-violations-of-privacy-minuscule.html
http://www.democraticunderground.com/10023474662
= new reply since forum marked as read
Highlight:
NoneDon't highlight anything
5 newestHighlight 5 most recent replies
leveymg
(36,418 posts)Here's the rule of thumb estimate contained in the longish analysis I posted on June 30 of the second set of NSA training slides released by Snowden: http://election.democraticunderground.com/10023134820
SLIDE 4 shows that 117,000 persons were profiled as of April 5, 2013. That may not seem like a very large number. But, NSA analysts are also trained to look at all communications, “two hops” out from a targeted person. The implication of that is that the initial PROFILING step outlined in SLIDE 1 profiles very large numbers of persons, and thus has no real 4th Amendment protection for US persons profiled.
How can that be if the system segregates US person data and is court approved? Look at SLIDES 1 and 2, again. Keep in mind the flow chart that shows how PRISM works. Everyone who has been in communication with those 117,000 targets indicated in SLIDE 4 is also investigated, and (at another “hop”) everyone they have been talking to or emailing or texting or chatting on Internet boards. Because these numbers grow exponentially with each hop, that means that if those 117K persons targeted by PRISM called 30 people that month, and they in turn called 30, the PRISM analysts will have initiated investigations of some 90 million persons each month. That could be more than a billion people each year are profiled. Profiling on this vast scale is consistent with known NSA's statistics: http://www.guardian.co.uk/world/2013/jun/27/nsa-online-metadata-collection
This confirms, in general, the two hops out approach but shows one of two things: 1) the NSA's search parameters at the initial PRISM profiling stage have additional factors built in that somewhat limits the potential search and targeting pools; or 2) the average number of new numbers dialed per month by the 117,000 NSA designated terrorists isn't 30 (as I guessed), but 9 or 10.
FarCenter
(19,429 posts)Including emails, texts, tweets, etc., there have to be several 10s of billions of pieces of communication globally per day now.
pnwmom
(110,260 posts)
DirkGently
(12,151 posts)Nah.
dkf
(37,305 posts)leveymg
(36,418 posts)human analyst for follow-up after it's red-flagged by the machine. There are maybe two dozen managers who sign off at each stage including the "reasonable suspicion" decision that sends a case over to the FBI unit at the Center for a FISA warrant request.
DirkGently
(12,151 posts)that they tend to erode to the point of meaninglessness.
dickthegrouch
(4,519 posts)20 million per month at 10 seconds per analysis (to determine whether deeper scrutiny is necessary) = 100 million seconds of staffing is needed as an absolute minimum.
1 staffer at 32 hours a week (although actual work time is more like 32 hours a week after vacations and illness, classes and administrative time are taken into account) works 468000 seconds in a month.
100M/468K = 427 analysts
That's assuming those 427 could find, analyze, and dispatch every initially flagged message in 10 seconds all day, every day.
uponit7771
(93,532 posts)...the people who are stoking this shit don't care about facts.
People are taking advantage of what others don't know about computers to stoke fear, watched faux new for 15 mins, they're doing the same thing..
This shit stinks...
smells to high heaven
ljm2002
(10,751 posts)...namely:
This shit stinks...
smells to high heaven
liberal_at_heart
(12,081 posts)They refuse to tell us the full scope of it.
Demo_Chris
(6,234 posts)
Yo_Mama
(8,303 posts)Something like that, anyway.
If there were 20 million of us who could REASONABLY be suspected of being a national security risk in any year, much less a month, the country would long ago have dissolved in flames, and Mad Max movies would no longer be fictional.
PowerToThePeople
(9,610 posts)dkf
(37,305 posts)randome
(34,845 posts)If you want to assume those queries are intentionally applied to American citizens, you're welcome to that opinion. But I think the majority of us would want to see evidence of that first.
[hr][font color="blue"][center]Stop looking for heroes. BE one.[/center][/font][hr]
dkf
(37,305 posts)randome
(34,845 posts)Maybe you're assuming this relates to the metadata? But "communications data" could be anything. They collect a hell of a lot of data from overseas. Just as other countries do to our data.
Sharing that data is done with internationally recognized lawful requests. At least I believe I saw that mentioned somewhere and absent evidence to the contrary, that's what I would assume.
[hr][font color="blue"][center]Stop looking for heroes. BE one.[/center][/font][hr]
dkf
(37,305 posts)
gollygee
(22,336 posts)Any given person under suspicion could have any number of queries. Probably a pretty large number per person.
I agree that there need to be changes with the NSA and our privacy, but I don't agree with making things sound worse than they are either.
dkf
(37,305 posts)Isn't all the data their universe?
gollygee
(22,336 posts)Person A sends X number of email messages a month. Each message gets investigated, and each investigation is a query.
dkf
(37,305 posts)You're not getting it, but it's obvious you don't want to. A query isn't a person. It's a reason to investigate. So if there is a person they're keeping track of, they might have a number of things that person has done that they need to track, and to check if it's connected to other "data in the universe." Each time they check, it's a query. A query is a question, not a person, so every individual thing - google search they make, email they send, web site they visit - they question and want to check is a separate query.
dkf
(37,305 posts)We've seen the input forms for the queries. WAPO posted them.
gollygee
(22,336 posts)The input form having a name listed does not mean it's for everything about that person, just that it's for one query for that person.
dkf
(37,305 posts)"While the FAA 702 minimization procedures approved on 3 October 2011 now allow for use of certain United States person names and identifiers as query terms when reviewing collected FAA 702 data," the glossary states, "analysts may NOT/NOT [not repeat not] implement any USP [US persons] queries until an effective oversight process has been developed by NSA and agreed to by DOJ/ODNI [Office of the Director of National Intelligence]."
The term "identifiers" is NSA jargon for information relating to an individual, such as telephone number, email address, IP address and username as well as their name.
The document – which is undated, though metadata suggests this version was last updated in June 2012 – does not say whether the oversight process it mentions has been established or whether any searches against US person names have taken place.
http://www.theguardian.com/world/2013/aug/09/nsa-loophole-warrantless-searches-email-calls
bemildred
(90,061 posts)dkf
(37,305 posts)bemildred
(90,061 posts)I did a lot of that, custom work mostly, a bit of SQL, I'm not getting into it here, it would sound like gobbledy-gook.
However, generally,
1.) the performance of a data query depends heavily on the size of the data set it has to process
2.) the data sets the spooks are handling are huge, meta-huge, uber-huge
3.) therefore, steps will be taken to partition them into manageable sets, even if one wants to query it all
4.) and also therefore, you will structure a query to access the least data necessary, unless you don't care how long it takes
5.) and I would wager that it can take one hell of a long time if you are careless, like never in your lifetime
That's my argument.
dkf
(37,305 posts)A computing and software revolution, launched in Silicon Valley a few years earlier, made sifting all that data easier. That was particularly true with the development of Hadoop, a piece of free software that lets users distribute big-data projects across hundreds or thousands of computers.
Named after a child's toy elephant and developed at Yahoo Inc., YHOO +0.66% the software reached commercial scale for Internet-wide tasks in 2008 and soon became a favored application for handling big-data demands. Twitter Inc. and Facebook snapped it up to manage their own giant databases of user information.
The NSA also became an early adopter. At a 2009 conference on so-called cloud computing, an NSA official said the agency was developing a new system by linking its various databases and using Hadoop software to analyze them, according to comments reported by the trade publication InformationWeek.
The system would hold "essentially every kind of data there is," said Randy Garrett, who was then director of technology for the NSA's integrated intelligence program. "The object is to do things that were essentially impossible before."
Mr. Garrett now runs RTRG's successor program, which was moved to the Defense Advanced Research Projects Agency and renamed Nexus 7. That effort has been using Hadoop and similar software to help manage large masses of data. One of the pieces of software, called Accumulo, was developed by the NSA using technology from Google, said a person briefed on the program.
"They've changed the paradigm so you no longer need supercomputers," this person said. "Randy has sought to leverage a lot of those big-data advances to improve U.S. government analysis on extremely large data sets."
Still, the NSA continues to face an enormous challenge in handling all its information. "The ability to manage it remains slim, because there's so much data out there," said one former NSA official.
http://online.wsj.com/article/SB10001424127887323495604578535290627442964.html
bemildred
(90,061 posts)Parallel computer architectures IS rocket science. Too bad Turing and Von Neumann were not around to help with that.
You can think of the internet as one big computation going on all the time, and it works because it's really zillions of little separate computations that are completely independent.
In data queries, generally things are not all independent, so it limits what you can do in that direction, some computations cannot be pieced out.
dkf
(37,305 posts)I would say so. Then the question is what are they searching through, as you pointed out.
bemildred
(90,061 posts)It would be a list of separate queries of separate data sets with the same search key, and there would be separate data sets for each data item (attribute) of interest. In database theory (SQL) this is called "normalization", IIRC, and it is usually considered a good thing.
dkf
(37,305 posts)The biggest has maybe a couple hundred tables and maybe 50 queries. It was originally built ages ago...maybe even during windows-nt.
So for database architecture, yes many many "queries". But I don't think that is how they count this 20 million number is it? Honestly I have no idea what they represent by this.
bemildred
(90,061 posts)I have no idea what they actually do, but I'd wager it's pretty bad, technically. The only thing that makes me think it might not be too bad is that it works at all, and I consider that somewhat unproven; when working heuristically, errors and losses don't matter much, the database is just an inefficient storage medium to search later. It might even be there for the sole purpose of being able to say one is not looking at it all, as it would be much faster to filter it on the fly and store the filtrate.
But that's not what they want.
dkf
(37,305 posts)I think that is the larger picture here, gauging the pulse of the populace and targeting who needs to be tamped down.
bemildred
(90,061 posts)dkf
(37,305 posts)But this experience in Iraq was probably eye opening for them, in terms of how they could use info. I am not sure we appreciate that yet and I doubt they are going to tell us.
Lol. I'm not even sure I'm on the right topic anymore. Maybe this post won't make sense.
bemildred
(90,061 posts)
randome
(34,845 posts)They may not find out until they hit the last partition that the person they are trying to track is on American soil. But in that process, they may have pulled dozens or even hundreds of records before hitting that last one.
[hr][font color="blue"][center]Stop looking for heroes. BE one.[/center][/font][hr]
bemildred
(90,061 posts)dkf
(37,305 posts)RALEIGH — As the field of “big data” continues to grow in importance, N.C. State University has landed a big coup – a major lab for the study of data analysis, funded by the National Security Agency.
A $60.75 million grant from the NSA is the largest research grant in NCSU’s history – three times bigger than any previous award.
The Laboratory for Analytic Sciences will be launched in a Centennial Campus building that will be renovated with money from the federal agency, but details about the facility are top secret. Those who work in the lab will be required to have security clearance from the U.S. government.
NCSU officials say the endeavor is expected to bring 100 new jobs to the Triangle during the next several years. The university, already a leader in data science, won the NSA contract through a competitive process.
NCSU university already has strengths in computer science, applied mathematics and statistics and a collaborative project with the NSA on cybersecurity. The university also is in the process of hiring four faculty members for its new data-driven science cluster, adding to its expertise.
Read more here: http://www.newsobserver.com/2013/08/15/3109412/nc-state-teams-up-with-nsa-on.html#storylink=cpy
bemildred
(90,061 posts)The problem is the Pentagon, in it's wisdom, rarely pays much attention to feasibility issues when spending money, vast sums have been spent on software and hardware that sits in warehouses or on obsolete media doing nothing, and which never ever did do anything except pass the fake acceptance test they ran for it.
So while i have no doubt they will piss the money away, I have lots of doubts about whether it will ever do anything useful, or what they intend, if you see.
And as I said, from a theoretical standpoint, bigger data means slower. This is one reason I think these guys are fools. They should be trying to MINIMIZE the size of the data sets they have to go through. This is naive, technically, what they are doing.
dkf
(37,305 posts)Exactly where they are we don't even know.
bemildred
(90,061 posts)We live in a finite world. They can make dramatic improvements, marginal improvements, but in the end, they are going to get stuck.
It's like this horseshit about perpetual geometric economic growth. Bigger is NOT better. More is NOT better. Right is better. Smaller is better, less is better. Just enough is better.
dkf
(37,305 posts)They were adding the price of potatoes to the algos for goodness sakes.
I keep coming back to that WSJ piece, and some others on what they "learned" in Iraq and Afghanistan.
Its only getting worse...
bemildred
(90,061 posts)Although, once again, I find that idea questionable, it means they are naive. The right data will improve your queries, more useless data will slow them down.
They are using the database system heuristically, profiling with it, looking for things, which is a very clumsy way to do things, and almost certainly implies failure with their human intelligence assets. Which sounds about right, now I think of it.
Edit: what I am saying is this is in some ways a very Rube Goldberg operation, they have gone to a good deal of trouble to patch these technologies together in a most hurried and inefficient way so they could search all of this data NOW. That's the point, to search it all, NOW, there is no other reason to grab it, or to build storage for it. I imagine the point of the Utah project is to fix some of those things and grab even more to search even faster looking for things, because it would take way-the-hell too long to implement something better.
dkf
(37,305 posts)They are trying to create correlations and relationships I imagine, going down to behavioral science. That goes beyond looking for terrorists obviously. But my contention is that this has a much bigger goal than Al Qaeda.
bemildred
(90,061 posts)dkf
(37,305 posts)leveymg
(36,418 posts)Last edited Sat Aug 17, 2013, 04:42 PM - Edit history (1)
since the late 1940s, and they seem to know the triggers, fault lines, and breaking point for societies and individuals, and how to get there.
Some of the best books on the subject of how the Pentagon and CIA developed the predictive sociology of propaganda, revolution and insurgency are by Prof. Christopher Simpson at AU: http://www.american.edu/soc/faculty/simpson.cfm
Science of Coercion
Blowback
The Splendid Blond Beast
National Security Directives of the Reagan and Bush Administrations
Universities and Empire
All must reads.
dkf
(37,305 posts)leveymg
(36,418 posts)Some factors may now be out of their immediate control. Resource depletion, price inflation, job erosion, and wasteful wars do that.
Jack Goldstone reduced revolution it to a formula that can be simplified to population and prices, although I think the post-structuralists like Goldstone and determinists like Skocpol and the Tilleys still tend to negate psychological factors too much.
Here's an interesting article that reviews and applied the author's particular Tipping Point theory to Egypt before the coup and some historical examples of failed revolutions: http://sociological-eye.blogspot.com/2013/06/tipping-point-revolutions-and-state.html
A Contested Tipping Point: The Egyptian Revolution
Egypt in January-February 2011, the most famous of the Arab Spring revolutions, fits most closely to the model of 1848 France. Egypt took longer to build up to the tipping point-- 18 days instead of 3; and there were more casualties in the initial phase--- 400 killed and 6000 wounded (compared to 50 killed in February 1848) because there was more struggle before the tipping point was reached. Already from day 7, troops sent to guard Tahrir Square in Cairo declared themselves neutral, and most of the protestors’ causalities came from attacks by unofficial government militias or thugs. By day 16, police who killed demonstrators were arrested, and the dictator Mubarak offered concessions, which were rejected as unacceptable. On the last day of the 18-day revolution, everyone had deserted Mubarak and swung over to the bandwagon, including his own former base of support, the military. This continuity is one reason why the aftermath did not prove so revolutionary.
Again, honeymoon did not last long. By day 43, women who assembled in Tahrir Square were heckled and threatened, and Muslim/Christian violence broke out in Cairo. Tahrir Square continued to be used as a symbolic rallying point, but largely as a scene of clashes between opposing camps. Structural reforms have not gone very deep. The Islamist movement elected in the popular vote relegated to a minority the secularists and liberals who had been most active in the revolution. President Morsi bears some resemblance to Louis Bonaparte, who rose to power on the reputation of an ancestral movement-- both had a record of opposition to the regime, but were ambiguous about their own democratic credentials. The analogy portends a reactionary outcome to a liberating revolution.
Bottom line: tipping point revolutions are too superficial to make deep structural changes. What does?
State Breakdown Revolutions
Three ingredients must come together to produce a state-breakdown revolution.
(1) Fiscal crisis/ paralysis of state organization. The state runs out of money, is crushed by debts, or otherwise is so burdened that it cannot pay its own officials. This often happens through the expense of past wars or huge costs of current war, especially if one is losing. The crisis is deep and structural because it cannot be evaded; it is not a matter of ideology, and whoever takes over responsibility for running the government faces the same problem. When the crisis grows serious, the army, police and officials no longer can enforce order because they themselves are disaffected.
This was the route to the 1789 French Revolution; the 1640 English Revolution; the 1917 Russian Revolution; and the 1853-68 Japanese revolution (which goes under the name of the Meiji Restoration). The 1989-91 anti-Soviet revolution similarly began with struggles to reform the Soviet budget, overburdened by military costs of the Cold War arms race.
(2) Elite deadlock between state faction and economic privilege faction. The fiscal crisis cannot be resolved because the most powerful and privileged groups are split. Those who benefit economically from the regime resist paying for it (whether these are landowners, financiers, or even a socialist military-industrial complex); reformers are those who are directly responsible for keeping the state running. The split is deep and structural, since it does not depend on ideological preferences; whoever takes command, whatever their ideas, must deal with the reality of organizational paralysis. We are not dealing here with conflict between parties in the public sphere or the legislature; such partisan squabbling is common, and it may also exist at the same time as a state crisis. Deadlock between the top elites is far more serious, because it stymies the two most powerful forces, the economic elite and the ruling officials.
(3) Mass mobilization of dissidents. This factor is last in causal order; it becomes important after state crisis and elite deadlock weaken the enforcement power of the regime. This power vacuum provides an opportunity for movements of the public to claim a solution. The ideology of the revolutionaries is often misleading; it may have nothing to do with the causes of the fiscal crisis itself (e.g. claiming the issue is one of political reform, democratic representation, or even returning to some earlier religious or traditional image of utopia). The importance of ideology is mostly tactical, as an emotion-focusing device for group action. And in fact, after taking state power, revolutionary movements often take actions contrary to their ideology (the early Bolshevik policies on land reform, for instance; or the Japanese revolutionary shifts between anti-western antipathy and pro-western imitation). The important thing is that the revolutionary movement is radical enough to attack the fiscal (and typically military) problems, to reorganize resources so that the state itself becomes well-funded. This solves the structural crisis and ends state breakdown, enabling the state to go on with other reforms. That is why state breakdown revolutions are able to make deep changes in institutions: in short, why they become “historic” revolutions.
Reconciling the Two Theories
Tipping point revolutions are far more common than state breakdown revolutions. The two mechanisms sometimes coincide; tipping points may occur in the sequence of a state breakdown, as the third factor, mass mobilization, comes into play. In 1789, once the fiscal crisis and elite deadlock resulted in calling the Estates General, crowd dynamics led to tipping points that are celebrated as the glory days of the French revolution. In 1917 Russia, the initial collapse of the government in February was a crowd-driven tipping point, with a series of abdications reminiscent of France in February 1848; what made this a deep structural revolution was the fiscal crisis of war debts, pressure to continue the war from the Allies who held Russian debt, and eventually a second tipping point in November in favor of the Soviets. But state breakdown revolutions can happen without these kinds of crowd-centered tipping points: the 1640 English Revolution (where fighting went on through 1648); the Chinese revolution stretching from 1911 to 1949; the Japanese revolution of 1853-68. Conversely, tipping point revolutions often fail in the absence of state fiscal crisis and elite deadlock; an example is the 1905 Russian Revolution, which had months of widespread enthusiasm for reform during the opportunity provided by defeat in the Japanese war, but nevertheless ended with the government forcefully putting down the revolution.
A tipping point mechanism, by itself, is a version of mass mobilization which is the final ingredient of a state paralysis revolution. But mass mobilization also has a larger structural basis: resources such as transportation and communication networks that facilitate organizing social movements-- sometimes in the form of revolutionary armies-- to contend for control of the state. If such mobilization concentrates in a capital city, it may generate a tipping point situation. But also such mobilization can take place throughout the countryside; in which case the revolution takes more the form of a civil war.
ljm2002
(10,751 posts)...it's hard to imagine how much time it would take to formulate reasonable suspicion / specific facts for that many queries.
K&R
NoOneMan
(4,795 posts)
bhikkhu
(10,789 posts)The NSA monitors foreign communications. An "incident" is when someone it is monitoring comes to the US and is monitored domestically. As they explain, when they are tracking the calls of a phone in another country, they don't necessarily know if that phone is in another country or here. When people are here the NSA needs a special warrant for monitoring, but if they weren't aware that the person had come here they wind up with a "technical error".