The Brooklyn-born Cohen studied violin at the Music and Art High School in Manhattan, but planned a career in medicine. However, after a stint in the army, where he performed in the symphony orchestra and jazz band, he enrolled at Juilliard, where he studied with Ivan Galamian.
Over the course of his career, Cohen was concertmaster at Pablo Casals' festivals in France and Puerto Rico, as well as for the Little Orchestra Society and the Mostly Mozart Orchestra, both in New York. He joined the Schneider Quartet as second violinist in 1952; he played with the Juilliard String Quartet from 1958 to 1968.
Cohen recorded many albums with the Beaux Arts Trio, which he joined in 1968, including works by Tchaikovsky, Rachmaninoff, Ives, and Shostakovich, and the complete piano trios of Hadyn and Beethoven. He continued to play with the trio until his retirement in 1992.
Among other institutions, Cohen taught at the Aspen Festival, the Curtis Institute of Music, Juilliard, Princeton University, and the Marlboro Music School and Festival, where he was on the faculty for nearly 40 years.