Agree that Sterling has "freedom of speech," However, he does not necessarily have freedom of consequences. If I were to spew what he said to a colleague at work, in a workplace that has many African Americans, I shouldn't necessarily expect that friend to hold everything in strictest confidence. Sterling isn't some redneck who owns a bar that caters to rednecks. He has bought an exclusive franchise which by it's nature comprises some 70-80% or so (fill in the correct percentage, I'm just approximating) of the very ethnicity he disparages.
Was he set up? Sure. This "lady" is a gold digger. She trades sex for fancy cars and an expensive condo. Sterling's estranged wife is trying to prevent any more outflow of cash (and she's not exactly a saint, either). The mistress sees the well is running dry and wants to squeeze out more cash and protect what she already has.
And poor Silver is caught between a rock and a hard place. Neither he nor the other owners have any authority to force Sterling to sell the team. Sterling is meeting the financial obligations; therefore the league cannot seize the Clippers. Most Silver can do is fine and suspend him ($1M limit IINM). And expect Sterling to sue in return. This mess has to be solved by sponsors, agents, players and fans. If all the sponsors cancel; if agents and players refuse to sign with the Clippers and those under contract demand trades; if corporations cancel their luxury boxes; if fans stop by tickets...then Sterling will be forced to sell.