The sale of about 300 branches was a condition of the £45.5bn bailout of RBS in 2008 during the financial crisis.
RBS struggled to find a buyer, and the UK Treasury has been working on what it called "alternative remedies".
Under the new deal, which the EU has accepted "in principle", RBS would spend £835m to help boost competition.
As a condition for the state support the bank was obliged to reduce its presence in some segments of the market.
However, instead of selling Williams & Glyn, RBS has proposed providing support for challenger banks and funds to boost competition in the sector.
"It will see RBS fund and deliver a package of measures to improve the UK business banking market and is designed to boost competition, helping small and medium-sized enterprises benefit from greater choice and offers on banking services," the Treasury said in a statement.
Under the plan a £425m "capability and innovation fund" will be set up to support other banks who lend to small and medium-sized enterprises.
RBS will provide £275m worth of "dowries" to challenger banks to help them lure SME customers away from Williams & Glyn.
An additional £75m will be made available to cover customers' costs of switching.
'Past to bed'
In total the bank will provide £800m for the scheme, with an additional £35m in running costs.
"[It] looks like a small price to pay to make this longstanding issue go away once and for all," said Laith Khalaf, senior analyst at Hargreaves Lansdown.
"RBS is putting the past to bed, and this new agreement represents another milestone in the bank's long journey back to good health."
The deal will be submitted for final approval by European commissioners later this year.