As a child, he earned enough money for a bus ticket to New York by singing on street corners. As a teenager in New York, he met Paul Robeson, the African American singer whose career was cut short when he was blacklisted in the 1950s, and who remained a role model for Fleming throughout his career.

Although Fleming was a natural tenor, the only roles available for him as a young black singer were baritone.

In World War II, he entertained troops. After the war, he went to the Manhattan School of Music on the G.I. Bill, and later made his debut at Carnegie Hall in 1948. He performed with the New York City Opera Company, the NBC Television Opera, at Bach festivals, and at Shakespeare Under the Stars in Ohio.

After a concert tour of Sweden in 1963, Fleming and his wife and son decided to stay there, performing throughout Scandinavia and Europe. Fleming moved to Ohio in the 1980s so his wife could be near her family, and performed there several times, with the Cleveland Institute of Music Orchestra and the Cleveland Orchestra, and taught at the Sutphen School of Music.