from Wikipedia:

He was born to Ann Phoebe Penn Dagworthy (Charlton) and Capt John Ross Key at the family plantation Terra Rubra near Keymar, Maryland.

Around the time of the Star Spangled Banner's composition, Keys was an attorney negotiating a POW's release. He and his party were not allowed to return home because the British planned on bombing Baltimore and his return would have revealed positions.