Prince Philip has supposedly refused to be under the same roof as Sarah Ferguson since her divorce from Prince Andrew in 1996.
He allegedly blamed her for embarrassing the Queen and the royal family after the infamous “toe-sucking scandal”.
Sources at Balmoral said the Duchess of York arrived on Thursday night but then unexpectedly left for London.
Some had hoped Prince Philip and Sarah had patched up their relationship after the pair were spotted attending Princess Eugenie’s wedding last year.
In April, it was reported that Philip, 98, had removed his long-standing objection to Andrew’s relationship with Fergie and “no longer cares”.
But news of her hastened departure from the Scottish retreat – after just two days – suggests there may still be some remaining tension.
Sarah was spotted catching a British Airways flight from Aberdeen Airport yesterday afternoon.
A royal insider told the Daily Mail: “In previous years the Duchess has left the day before the Queen’s husband arrives.
“This year he arrived yesterday, several days earlier than expected, which is rather strange, so the Duchess left.”
She was expected to spend up to a week with Prince Andrew and their daughters Princess Beatrice and Princess Eugenie at the monarch’s summer residence.
Fergie and Andrew, both 59, travelled to the Highlands separately to avoid being pictured together, given the recent focus on their relationship.
Andy flew to Aberdeen airport on a private jet on Tuesday, while Fergie landed on a commercial flight on Thursday.
A royal insider revealed: “It’s not the first time they’ve been there as a couple in recent years but it’s significant because of all the rumours about them getting back together.
“The Queen wanted them there as a family and it’s very much her giving the private nod of approval.”
It could be quite a setback for Fergie, who has done her best to ingratiate herself with the royals.
But she still does have one very important champion – the Queen – who invited Fergie to Balmoral for the fourth summer in a row.
Fergie and Prince Andrew wed in 1986, separated in March 1992 and five months later she was pictured having her toes sucked by Texan millionaire John Bryan in Saint Tropez.
On the morning the photographs were published, Sarah Ferguson was staying at Balmoral with the rest of the royal family.
Fergie hastily left the Scottish holiday home in the morning and drove the 50 miles to Aberdeen airport with her children and nanny.
After four years of separation, Sarah Ferguson and Prince Andrew officially announced their decision to divorce in May 1996.
She has retained her title Duchess of York but ceased being a Royal Highness.
It comes as Prince Andrew was spotted in public for the first time since his paedo former pal Jeffrey Epstein killed himself in a Manhattan jail.
On Friday, court documents relating to his case were unsealed, and included claims the Duke of York groped a woman’s breast and licked the feet of one of Jeffrey Epstein’s sex slaves during a visit to his apartment in 2001.
Prince Andrew strenuously denies any sexual contact with the women.
In a statement, Buckingham Palace said: “This relates to proceedings in the United States, to which the Duke of York is not a party.
“Any suggestion of impropriety with underage minors is categorically untrue.
“Any claim to the contrary is false and without foundation.”