In the former ecommerce business I owned, we did about 225K/month in CC billing... about 75% of that was Visa/MC and the rest was Discover/Amex...
If you don't accept Amex you are an idiot! Sure, their rates are higher and they always side with the consumer, but who gives a fuck! In the grand scheme of things, it's just one of many costs of doing business.
However, if you don't accept them, you are turning away free money. Because people that have Amex, use Amex for the exact reasons stated above.
How do you know that 25% wouldn't have just whipped out their Visa and paid with that because you don't accept Amex? I find Amex card holders (and I'm guilty too) will always try to use their Amex because of the customer service, and protection it provides but if you tell people you don't take Amex they go straight to their Platinum Visa.
From my meager e-comm sites I have only had 1 person walk away from the transaction when I told them I don't take Amex. They were probably going to chargeback anyway so no loss there.
Amex holders typically have excellent credit so they will always have a backup Visa/Mastercard on hand.