The refunds will average £970 but range up to £18,000, the financial watchdog the FCA has said.
The FCA is also imposing a fine of £8.9m on Clydesdale, which includes the Yorkshire Bank brand and is owned by National Australia Bank, for treating customers unfairly when trying to put right their own error.
Clydesdale had installed a new computer system which undercharged borrowers for four years until the mistake was discovered in 2009.
The bank then adjusted payments to make up for the missing money, without telling 22,000 customers with shortfalls that they might not be obliged to to pay it back.
A further 20,500 who paid too much, some of whom paid off their mortgages early, may be able to claim compensation as well.
Clydesdale said the compensation and fine would cost it £42m.
The bank has apologised for episode and said 14,000 of the customers will receive full redress within 48 hours, in many cases reducing the amount they owe.