Gary was elected as a Democrat to the U.S. Senate to fill the vacancy caused by the death of Asbury C. Latimer and served from March 6, 1908, to March 4, 1909; he was not a candidate for reelection in 1908, and after his time in the Senate he was again a member of the State house of representatives in 1910. He was elected judge of the eighth judicial circuit in 1912 and served until his death in Charleston, South Carolina in 1922; interment was in Long Cane Cemetery, Abbeville, South Carolina.
Frank B. Gary was also appointed as special judge in Lexington County in the 1903 trial of James H. Tillman (lieutenant governor of South Carolina and nephew of Senator "Pitchfork" Ben Tillman) for the murder of N.G. Gonzales (founding editor of The State, Columbia, SC's newspaper).
It has been alleged that Gary was a "Tillmanite", although there is no strong evidence of his being partisan in the trial. However, the jury was considered highly rigged and partisan considering Tillman shot Gonzales in broad daylight with many eyewitnesses. Tillman was acquitted ostensibly on a self-defense theory, but more likely because the jury believed Tillman was justified. Gonzales had waged a virtual crusade against Tillman in the newspaper, helping ensure his defeat in the 1902 gubernatorial election.