An ombudsman said that victims of poor service from financial firms should have problems dealt with quickly, particularly during uncertain economic times.
Consumer group Which? People who lost money faced a waiting lottery when they tried to report the situation to their bank.
Average waiting times can range from 16 seconds to more than 30 minutes depending on the bank, tests by Which? It said more priority should be given to distressed victims.
The Financial Ombudsman is the independent arbiter of disputes that can't be resolved between financial firms and their customers.
In the last financial year, it received nearly 9,370 complaints about the response to so-called authorised scams compared to 7,770 the previous year. People who were believed to be a genuine trader or organisation are often tricked into transferring money to a fraudster.
The ombudsman is finding in favor of the customer in three-quarters of these cases, which may require banks to refund the money lost.
In this period of economic uncertainty, it is more important than ever that where problems do arise, they are addressed quickly, said Nausicaa Delfas, interim chief financial ombudsman.
The Financial Ombudsman Service received a lot of complaints about fraud and scams in the year to April.
In the year to April, the service received 165,263 new complaints about financial businesses. The vast majority of 108,065 were about banking and credit.
44% of frontline service staff experienced hostility from customers in the past six months, a rise from 35% during the previous six months, according to data from the Institute of Customer Service.
A quarter of staff who have been facing increased hostility said they believed the increase was due to the increased cost of living and stress this had placed on customers.