Rollins was elected as a Republican to the 37th, 38th, and 39th Congresses, serving from March 4, 1861 to March 3, 1867; he was not a candidate for renomination in 1866. While in the House of Representatives, he was chairman of the Committee on Accounts (38th and 39th Congresses). He was secretary and treasurer of the Union Pacific Railroad, and in 1876 was elected to the United States Senate and served from March 4, 1877, to March 3, 1883; he was an unsuccessful candidate for reelection. While in the Senate, he was chairman of the Committee on Manufactures (45th Congress) and a member of the Committee on Enrolled Bills (47th Congress) and the Committee on Public Buildings and Grounds (47th Congress).
From 1886 to 1889, Rollins was president of the Boston, Concord & Montreal Railroad, and was founder of the First National Bank of Concord, New Hampshire, and of the banking house of E. H. Rollins & Sons, Boston.