Ray J Says He and Brandy ‘Never’ Had ‘Sibling Rivalry’ Growing Up: ‘I Never Wanted to Work That Hard’

The “Sexy Can I” musician opened up about his relationship with his sister during the Wednesday, June 5 episode of ‘Club Shay Shay’

Ray J and Brandy attend Urban One Honors at The Anthem in December 2018 in Washington, DC. Photo: Brian Stukes/WireImage

Ray J and Brandy weren’t exactly at odds when they were kids.

During a Wednesday, June 5 episode of Club Shay Shay, the R&B singer/TV personality, 43, opened up about whether he had disagreements with his sister when they were growing up.

“Nah, not at all. What’s crazy is like, never,” Ray J, 43, told host Shannon Sharpe. “Because I just never — I never wanted to work that hard. You get what I’m saying? It had to be a little bit more fun. The only time we was, like, going back and forth in the beginning stages was when I was telling her and Rodney Jerkins — who is one of the greatest producers is in the universe — I was telling ’em that Pharrell [Williams] is one of the greatest producers in the universe and the songs that we got is killing everything.”

Ray J and Brandy in West Hollywood in February 2017.Michael Bezjian/Getty

He continued: “And they were just really s—ting on us at the time. With Rodney, you know, it was the new Never Say Never album, it was Michael Jackson — everybody in the world was working with Rodney — and Pharrell hadn’t had his break, him and Chad [Hugo] and The Neptunes. But I had some of the first records, and I just knew that Pharrell was gonna be a game-changer.”

So Ray J ultimately fought with Brandy and the label to drop “Wait a Minute” with Lil’ Kim, which was also produced and co-written by The Neptunes.

“Fast-forward into later, Rodney and I, we did ‘One Wish’…so we made a massive record together later, but the record I wanted and the people I wanted to be around and to be in it was Pharrell and Chad. They thought we was tweaking because it took us 10 minutes to do the song,” the Save the Last Dance 2 star said.

It was on a “Neptunes classic beat,” which Ray J recalled gave Williams’ career a boost, as well as his own.

“It just showed me that there’s never one way to do it,” he said.

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

Ray J and Brandy in Oxon Hill, Maryland in December 2019. Brian Stukes/WireImage

Elsewhere in the conversation, Ray J opened up about the time Brandy, 45, was “pissed” at him for getting her face tattooed on his leg.

While the “gothic” ink on his leg was meant to honor her, it didn’t go over with her well as it portrayed her with bloodshot eyes and a face covered in various phrases like “Best Friends” and “From Ray with Love.”

“I think the tattoo went viral the wrong way,” Ray J admitted. “I think the tattoo was not it… for the world.”