In 1947, Heath and his brother Jimmy, a saxophonist, moved to New York, to play in trumpeter Howard McGhee's band. Three years later, both joined Dizzy Gillespie's sextet. In 1952, Percy Heath became a member of the newly formed Modern Jazz Quartet, with pianist John Lewis, vibraphonist Milt Jackson, and drummer Kenny Clarke.

In 1955, Clarke was replaced with Connie Kay; otherwise, the group remained intact for more than 30 years. Performing Lewis's elegant compositions and clad in tuxedos, the group helped to establish modern jazz as concert music.

The MJQ broke up in 1974 when Jackson tired of touring, and Heath began performing with the Heath Brothers, with Jimmy on saxophone and their brother Albert—known as "Tootie"—on drums. In 1981, MJQ reformed, performing regularly if less frequently. After the death of Connie Kay in 1994, Albert Heath joined the group; it stopped performing entirely in 1997 when Percy Heath announced that he would retire.

In the years since, Percy Heath performed occasionally with the Heath Brothers, but spent much of his time fishing. In 2004, he released A Love Song, his first and only recording as a leader.