Current account customers who have unauthorised overdrafts are being charged an average 167% interest a year, according to research.
Business Secretary Vince Cable said high street banks are "ripping off" their customers, adding that they are not transparent when it comes to the fees they charged people.
This is because the industry is dominated by a few major banks, which ensures that customers are losing out, he said.
The research, carried out by BBC One's Panorama show, showed that many people paid interest at a much higher rate on both authorised and unauthorised overdrafts than what the banks advertised because of extra fees they charged.
And the actual average interest charged in penalties is 32% and not the usual stated 19%. This rate is even higher for unauthorised overdrafts, at 167%, once the different fees are accounted for.
However the charges were defended by the British Bankers' Association because it said overdrafts are meant to be used for just a couple of days, adding that working out precise lending rates over the year is impossible.
Additional charges for overdrafts can include penalties for simply making use of an overdraft, fees for the bank paying out direct debits when an account does not have enough money in it and fees for bounced payments.