SI detailed accusations made against Richardson that include sexual harassment of multiple women and the use of a racial slur toward a scout who has since left the team.
The NFL has taken over a workplace misconduct investigation into Richardson at the request of the Panthers, who initially announced Friday that they were looking into accusations made against the owner.
Richardson, 81, allegedly made verbal comments about women's appearances, inappropriately touched female employees and made advances to women that included asking if he could shave their legs and for them to give him foot rubs.
Along with the allegation of using a racial slur that led to a settlement with the scout, SI notes comments made by Richardson about black players' appearances and his threat to discipline players who addressed social issues.
According to SI, Richardson and the Panthers reached confidential settlements with complainants that included nondisclosure and non-disparagement clauses. The SI report describes the settlements' value as "significant."
Neither the NFL nor the Panthers have revealed the nature of the investigation. The team declined to address specifics when contacted by SI, but Panthers spokesman Steven Drummond said Sunday that they asked the NFL to handle the investigation for "transparency reasons."
The league will retain an outside counsel to conduct its inquiry.
Members of the organization have been mostly quiet on the situation, although coach Ron Rivera said in a statement he has "enormous respect for the man, but will wait for the results of the investigation before making any judgment."
A Panthers player told ESPN's Josina Anderson on Saturday that he had not encountered racial issues with Richardson but that he supports others for speaking up.
"I respect them for speaking up no matter who it is," the player said. "... Equality is what we are going for."
Richardson was at Bank of America Stadium for the Panthers' home game against the Green Bay Packers, but the windows to his end zone suite were not open, as they typically would be by game time.
Richardson became the first former NFL player since George Halas to own an NFL team when he was awarded an expansion franchise on Oct. 26, 1993. The Panthers began play two years later in 1995.
He has worked primarily behind the scenes during his tenure with the Panthers and rarely does interviews. But he has been active through the years with league matters.
Information from ESPN's David Newton and The Associated Press was used in this report.