Born in Brooklyn to Puerto Rican parents, Barretto was exposed at an early age both to the music of Puerto Rico and the big-band jazz of Duke Ellington and Count Basie. He began to perform in the late 1940s, while serving in the military in Germany, after hearing conga player Chano Pozo perform with Dizzy Gillespie.

A master of the conga drum, Barretto performed both with mambo groups, including Tito Puento's band, and with bebop jazz groups in the 1950s. He recorded with Cannonball Adderley, Kenny Burrell, Dizzy Gillespie, Freddie Hubbard, and Wes Montgomery, among other leading players, and is often credited with introducing the conga to mainstream jazz. In 1962, with his group Charanga La Moderna, he scored a crossover hit with the song El Watusi.

From the 1960s through the 1980s, he directed the Fania All-Stars, an all-star salsa group. Since 1992, he has led New World Spirit, a jazz group. In September 2005, he won the National Endowment for the Arts Jazz Master Award, the nation's highest honor for jazz musicians.