From a 2009 LA Times piece:
Wealthy U.S. taxpayers, concerned about an Internal Revenue Service crackdown on the use of secret overseas bank accounts as tax havens, are rushing to meet a Thursday deadline to disclose those accounts or face possible criminal prosecution. The concern was triggered this summer when Switzerland's largest bank, caught up in an international tax evasion dispute, said it would disclose the names of more than 4,000 of its U.S. account holders.
The decision shattered a long-held belief that Swiss banks would guard the identities of its American customers as carefully as they did their money, and it raised concern that other international tax havens might be next. Under an amnesty program, the IRS is allowing taxpayers to avoid prosecution for having failed to report their overseas accounts. As a result, tax attorneys across the nation have been besieged by wealthy clients who are lining up to apply even though they will still face big financial penalties.
Slate's Matthew Yglesias speculates Romney may have taken this 2009 amnesty for non-disclosure of his Swiss bank account.
Romney might well have thought in 2007 and 2008 that there was nothing to fear about a non-disclosed offshore account he'd set up years earlier precisely because it wasn't disclosed. But then came the settlement and the rush of non-disclosers to apply for the amnesty. Failing to apply for the amnesty and then getting charged by the IRS would have been both financially and politically disastrous. So amnesty it was. But even though the amnesty would eliminate any legal or financial liability for past acts, it would hardly eliminate political liability.
This seems highly plausible, either 1) He did disclose his Swiss bank account to the IRS prior to 2009 which would be indicated by his prior returns, in which case releasing these returns would put this all to bed or 2) He did not disclose this account in which case it would not show up on his returns, and releasing them would prove he had acted illegally, almost assuredly dooming his campaign. That fact that he refuses to release any of these past returns leads me to believe it is the second case, otherwise the political pounding he's currently taking would not be worth it.