Jazz saxophonist Pee Wee Ellis, best known for his work leading bands for James Brown and Van Morisson, has died. He was 80.

Bootsy Collins tweeted of Ellis’ “brilliant arrangements,” calling him a “silent genius.” In a statement on his Facebook page, Ellis’ family said:

“With great sadness we have to announce that Pee Wee passed away last night following complications with his heart. We are working on plans to celebrate his wonderful life and hope you will all take time to listen to his music and continue his legacy.”

Born in Bradenton, Florida on April 21, 1941, Ellis’s career was jump started in 1957 by a chance meeting with Sonny Rollins in New York City. Rollins agreed to train him, and by 1965 he had joined James Brown’s Revue on the recommendation of trumpeter Waymon Reed. He would quickly become Brown’s bandleader, and go on to help him write several of the Godfather of Soul’s signature songs, including “Cold Sweat” and “Say It Loud—I’m Black and I’m Proud.”

Ellis was a member of the band that recorded Van Morrison’s 1979 album Into the Music, sparking a multi-decade partnership as his bandleader and arranger. The next year he was on Morrison’s Common One opus “Summertime in England,” and later appeared on the LPs Days Like This and The Healing Game. Several of his recordings have been sampled for hip-hop records, particularly his work with Brown and his jazz-funk band Gotham. in 2012, he joined Ginger Baker’s Jazz Confusion.