The Arlington-born musician—who was described by Andr_ Previn as a "man of Dickensian proportions with added touches of Peter Ustinov and Captain Ahab"—was largely self-taught. In the 1950s, he transcribed musical charts for the Army Band in Washington and played with THE Orchestra, a big band that occasionally featured big-name guest artists such as Dizzy Gillespie and Charlie Parker.

Some of Potts's recordings of THE Orchestra, including a famous performance with saxophonist Lester Young, have become collectors' items.