In 1963, when Johnson was vice president, a disgruntled former business associate named Ralph Hill filed a lawsuit alleging that Baker had taken thousands from Hill to steer a vending machine contract to his company but had then taken the contract for himself. Baker, it turned out, had created his own vending machine company and had used his influence with government officials to win lucrative contracts, amassing a $2 million fortunate on a Senate salary of less than $20,000 a year.
Baker also attracted attention for co-founding the Quorum Club, where lobbyists and members of Congress would drink and cavort with women. Baker also would say that he directed Ellen Rometsch, a suspected East German spy, towards President John. F. Kennedy. Attorney General Robert F. Kennedy, the president's brother, was worried enough about the rumors that he secretly had Rometsch deported. Baker's financial dealings and other illicit activities got him on the covers of Time and Life and created a headache for Kennedy and especially Johnson—and giving an opening to congressional Republicans, who soon began pushing for an investigation into Baker.
Baker resigned from his Senate post hoping to stop the investigation from spreading, but he was unsuccessful. It's quite possible that Johnson's ties to Baker, as well as other factors, would have gotten the vice president removed from Kennedy's 1964 ticket. However, after Kennedy was murdered in November 1963 and Johnson became president, the Baker matter quickly faded from public view. But Baker's own troubles weren't over at all: In 1967, he was indicted for tax evasion, theft, and fraud, and he later served 16 months in prison. Johnson completely cut ties with the man responsible to a great degree for his rise to power and refused to acknowledge him in any way. Baker visited Johnson just before he died in 1973 and recounted in his memoir (aptly titled, "Wheeling and Dealing: Confessions of a Capitol Hill Operator") that the former president told him, "Everything within me wanted to come to your aid. But they would have crucified me."
In 2009 and 2010, Baker spoke to the Senate Historical Office about his experiences in D.C. As Todd Purdum would write for Politico, the "manuscript was so ribald and riveting, so salacious and sensational, that the Historical Office refrained from its usual practice of posting such interviews online." Luckily for history, Purdum would publish Baker's most vivid recollections.
P.S: And if you haven’t read them, we again cannot recommend Caro’s The Years of Lyndon Johnson series highly enough.