Caldwell graduated from the Curtis Institute in Philadelphia, where he studied with the Philadelphia Orchestra's John de Lancie. He performed with the Puerto Rico Symphony and Chicago Little Symphony before joining the National Symphony in 1965.

During his tenure in the NSO, Caldwell and his wife, cellist Catherina Meints, became interested in early music. He took up the viola de gamba and began to collect period instruments. After Caldwell joined Oberlin's faculty in 1971, he and Meints founded the Baroque Performance Institute, a summer institute at the conservatory, and helped turn Oberlin into a center for early music.

Caldwell's oboe students at Oberlin included Alex Klein, later the principal oboist of the Chicago Symphony.

"James Caldwell was too big a person for just one activity," Klein said. "He was a great teacher and artist, a philosopher, a bonsai aficionado, an avid collector, and a wonderful mentor. He excelled at everything. He opened the door to the world. He saw joy and beauty in every millimeter of life, and he brought his vast knowledge and experience into the practice of his teaching."