Several of the criticisms about NSA efforts to create a comprehensive database have been concerned with feasibility. Various technical arguments about the capabilities of data mining have been presented.
I, myself, have written about the near impossibility of finding unknown actors who are working alone or in small groups. The London underground bombing is a perfect example. Some have decided that the NSA and other intelligence organizations are "stupid". While there may be a certain amount of effort at over emphasizing dangers in order to enhance the status (and budget) of one's department, it is doubtful that these people are stupid.
So, if what they are presumed to be doing makes no sense than it is time to examine the presumptions.
The likelihood of finding international "terrorists" via examination of telephone records is extremely small. The number of records is vast and the identities of those being sought is unknown. If it were known than a simple warrant would be all that was required. In these days of throw away cell phones, someone wishing to remain untraced would never use a permanent phone number either for calling or receiving messages. Many clandestine organizations stay away from all electronic communications completely. They rely on messengers.
Now, where does monitoring telephone (and banking and credit card) records work well? Where the persons involved are known members of specific groups, or where their involvement with these groups is what is trying to be discovered. What kind of groups are these? Those with stable memberships and a public face, namely groups involved in political activity. In all the prior abuses of government power in the US, the targets have always been groups of such a character. Being able to connect them to some international movement so as to justify a crackdown just makes the task easier.
Examples abound. During the early part of the 20th Century there were crackdowns against organized labor, especially the Wobblies, Socialists and Anarchists. Many of their organizations were infiltrated by federal agents and/or paid informants. Attempts (sometimes successful) to get them to engage in illegal activities were frequent. This would give the police an excuse to arrest people engaged in otherwise peaceful meetings. This effort culminated in the Palmer raids which imprisoned up to 10,000 "subversives".
Later in the century we had a repeat of the situation with Communists and Soviet sympathizers, especially in the 1930's and 1940's. These groups were so caught up in their own infighting as to which brand of leftist ideology was the true one, that they never had any real influence politically. Nevertheless the FBI investigated, infiltrated and we ended up with imprisonment and Joe McCarthy.
As a final example we have the civil rights movement of the 1960's. Many of the moderate groups such as that led by Martin Luther King, Jr. were consistently spied on. More radical groups such as the Black Panthers were infiltrated and there is some indication that agents provocateurs were used to cause incidents which led to violent confrontations with police.
With a forty year neo-con effort starting with Reagan that the conservative vision is the only one that can "save" America, the belief that the end justifies the means has become the norm. Starting with Iran-Contra and extending up to the present, breaking the law, spying on protest movements, and using disinformation campaigns has expanded as a self-justifying effort. The latest NSA efforts are not directed at foreign actors, they are directed at domestic political protest.
A simple example will show how this works. The government obtains the names of the members of an organization. This can be done legally as, for example, by getting the names of all that contribute to a political party or candidate, or illegally. Knowing the names of the members their phone numbers can be monitored to discover their web of associates. Since these people are not doing anything wrong, or trying to hide their identities or activities, their phone numbers are stable and easily watched. If a sudden change in calling patterns or volume occurs this may indicate that an activity or event is upcoming and as these are usually public the details are easily obtained. The phone activity gives indications of who will be attending. This gives the government the opportunity to participate in the activity either passively or actively.
During the Republican Convention in NYC, there was ample proof that there were undercover police operating within the crowd and that actions were taken to give the police an excuse to arrest the protesters even when they were doing nothing wrong. So anxious were the police to get the protesters off the streets that many people who were just bystanders were rounded up as well.
Looked at from this perspective the activities of the NSA and other spy agencies make perfect sense. Monitoring protected first amendment activities cannot be justified by legal means, so illegal means must be used to perform it. Techniques which are ineffective against unknown actors work well against known targets. Use of a (minuscule) foreign threat is used to justify and obscure the surveillance of domestic protest. The bottom line is that the primary purpose of almost all of the domestic spying is control of the populace, not protection from foreign threats. This has been the pattern throughout history, there is no reason to think it is different this time.