I think I may have found your James Robb

There's an MI in Aberdour kirkyard for the Robb family which has Isobel Stewart dying 2 Oct 1831. A James Robb appears on the same memorial - he died on 10 Sep 1864. The death registration for this James Robb has him as aged 81, a widower (no spouses names), the son of William Robb and Janet Ogilvie. The death was registered in Pitsligo by James' son-in-law, Robert Sutherland (m. Jean/Jane Robb in 1844 in Pitsligo).
This James Robb is with his (presumably) second wife, Mary Stewart/Stuart in Aberdour in the 1841 and 1851 censuses - by 1861, he is a widower. James is noted as being born in Keith, Banffshire, Mary in New Deer, Aberdeenshire. He married Mary Stewart in 1833 - it looks like she died in 1852.
I can't find a
William Robb and Janet Ogilvie, but I can see a
George Robb and Janet Ogilvie, who married in 1769 in Keith, and had a number of children baptised in Keith, so this could well be the family, although James isn't one of the names recorded.
Ruth