Dubbed by themselves "The Little Men's Club," the group would often meet at a Broad Ripple bar between 2007 and 2009.
According to a source, who spoke on condition of anonymity with Call 6 Investigator Rafael Sanchez, Fogle would often travel within the U.S. with a member of the group to watch his back and keep him out of the media.
"Very few outsiders and, umm, you know, they'd hang out together and they had these common interests that they were all, you know, exploring together. And that was their thing," the source said. "And when it's a small group of people, you have more control over what gets out, who talks about what and whatever. And it's probably in their best interests for none of them to talk."
Click on the video player above to watch the interview.
Members of the club attended Fogle's annual Labor Day barbecues at his Zionsville home.
The source said there was a hope that when Fogle got married in 2010 and started having children things would change.
"I feel like I was duped," they said. "I even defended him early on in all this, and for the truth to come out now, you know, I feel let down and I feel I was made a fool of. I bought his image."