Kelly played only 32 games at the major league level for the 1920Philadelphia Athletics and the 1928Philadelphia Phillies, hitting only .179 in 84 at bats with no home runs and five runs batted in. But in between, he led the top-level International League in RBI for three consecutive seasons (1924–26) while he played for the Buffalo Bisons, racking up 155, 125 and 151 RBI in that span. He also led the IL in homers in 1924 (with 28) and 1926 (with 44). As a minor leaguer from 1921 to 1931, Kelly batted .308 and slugged 207 home runs.
In 1932, he began his umpiring career in the higher minors, eventually working in all three top-level minor leagues through 1941. After World War II, he moved into the dugout as a manager in the Chicago Cubsfarm system, piloting top-level farm teams with the Los Angeles Angels of the Pacific Coast League (1947–50) and the Springfield Cubs of the International League (1951–52). His 1947 Angels won the regular-season and playoff PCL titles. After hanging up his uniform in 1954, Kelly scouted for the Cubs, Phillies and New York Mets through 1974 and his retirement.