Mangelsdorff studied the violin as a child and taught himself the guitar; he began his jazz career as a guitarist, playing for American soldiers stationed in Frankfurt after World War II. He began playing the trombone in 1948. In 1958, he traveled to the Newport Jazz Festival with a group of international musicians. While in the United States, he recorded with Louis Armstrong and played with other American musicians.
Starting in the 1960s, Mangelsdorff's style shifted toward the avant-garde; he developed a method of producing multiple sounds with the trombone and frequently played and recorded solo. He performed in several long-running groups and co-led the German-French Jazz Ensemble. In the 1990s, he became music director for JazzFest Berlin.
Mangelsdorff recorded with pianist John Lewis, drummer Elvin Jones, trumpeter John Cherry, and guitarist John Scofield, among others.