The couple, who divorced 26 years ago, spent time together at the monarch’s Scottish summer residence on Thursday.
The pair, both 59, travelled to the Highlands separately to avoid being pictured together – after rumours circulate about their relationship.
Fergie landed on a commercial flight on Thursday, while Andy flew to Aberdeen airport on a private jet two days before.
“They went to great lengths to avoid being spotted together arriving”
Their daughters, Princesses Eugenie, 29, and Beatrice, 31, are due to join them for a family holiday.
A royal insider revealed to The Sun: “Prince Andrew and Sarah are at Balmoral as guests of the Queen and are expected to be there for a number of days. They will be joined by their daughters for some of the holiday.
“It’s not the first time they’ve been there as a couple in recent years but it’s significant because of all the rumours about them getting back together.
“The Queen wanted them there as a family and it’s very much her giving the private nod of approval.”
This family holiday is a sign that Fergie has firmly re-established herself in the royal family.
The couple, who share a £13m ski lodge in the posh Swiss resort of Verbier, are determined to keep their personal time under wraps.
The source added: “They went to great lengths to avoid being spotted together arriving.”
In April, ITV’s royal editor Chris Ship was first to reveal Andrew and Fergie were “back together”, but his bombshell story was officially denied by the duchess.
Later in the month, The Sun confirmed that the couple were in a romantic relationship that royal staff branded “friends with benefits”.
A Buckingham Palace spokeswoman said: “The Duke of York’s travel to Scotland was paid for privately. Balmoral is The Queen’s private home so we wouldn’t comment on who may or may not be staying there.”