It was all down to Cost: The Airlines were not making back the money spent on the safety modifications and other upgrades, with some other big costs coming up (tens of millions, before any life extension programme), BA need to write off £84M now rather than £150M in 3 or 4 years. Air France wrote off a large sum of money too.
With the premium first class market non-existant post 9/11, there was no hope of paying back the modification cost to start with, forgetting about any further investment that was required to keep the aircraft in the air. Day to day the aircraft still broke even, but could no longer pay back any big expenditure items, so its days were numbered. It is a sad time, but the inevitable really only came forward a few years from the ends of its technical lifespan.
The development costs of Concorde were around £1.134 billion, which was funded by the UK and French governments. The cost to build the 16 production Concordes was £654 million of which £278 million was recovered through sales returns (this included spares, technical support, etc.). This debt was also funded by the 2 governments.
Because of the premium rate that passengers would pay to fly on Concorde (First Class +20%), the aircraft only needed to be around half full to break even and turn an operating profit.