This is also from the Times, could this be his death?
The Times, Wednesday, May 29, 1861; pg. 1; Issue 23945; col A
On 25th inst, at 15, Inverleith Place, Edinburgh, the residence of his daughter, Mrs Charles Tennant Dunlop, Ebenr. Fernie, formerly of Tottenham and Kew, Surrey, Esq, J.P. and Deputy-Lieutenant for the county of Middlesex, in his 79th year.
(There is an early entry for wife Mary)
The Times, Thursday, Feb 27, 1868; pg. 1; Issue 26058; col A
On the 25th inst, at Trinity-church, Sloane-street, Chelsea, by the Rev. Richard Burgess, Kirwan Joseph Fernie, Esq, of Haftor House, Orkedal, Norway, son of the late E. Furnie, one of Her Majesty's Justices of the Peace, and Deputy-Lieutenant of the county of Middlesex, Esq, to Ann Caney Stanton, youngest daughter of the late Stephen James Bridges Stanton, Esq, of Chase Lodge, Hendon, Middlesex.
The Times, Wednesday, Apr 30, 1884; pg. 1; Issue 31121; col A
On the 25th April, of apoplexy, at 6, Via Garibaidi, Florence, Marian Witherington Bell, wife of Vernon Bell, MD, and daughter of the late E. Fernie, one of Her Majesty's Justices of the Peace, and Deputy-Lieutenant of the county of Middlesex, Esq.
freebmd.rootsweb.com shows Charles Tennant Dunlop as marrying Janett[ae] Mary Fernie in 1845.
Now, looking at Kirwin (first name often misspelled), he died in Hendon in 1906, aged 86. There's also a 1841 census hit for 'Kerwen' Fernie, 22, b. in Middlesex, living with a Josh Fernie, 55, accountant, born in Middlesex. They're in St. Andrew Undershaft parish, Leadenhall St.
This is probably Joseph Fernie, who died 1844 (his will is on
www.nationalarchives.gov.uk/documentsonline - might mention relatives? He could be a brother to Ebenezer.)
www.a2a.org.uk has a few hits:
Berkshire Record Office: Reading St Giles Parish Records
marriage license, 1812, Ebenezer Fernie of London, gentleman, and Mary Waugh of Reading St. Giles.
http://archiver.rootsweb.com/th/read/SCT-GLASGOW/2006-02/1139858438 - here's someone whose relative married Ebenezer Waugh Fernie.
Very strong links to Scotland in this family!
http://website.lineone.net/~jcn/LND/illust1.htm - !!! This might explain it if you can't find the family in 1851!
http://www.oldbaileyonline.org/html_units/1820s/t18200918-75.html - and another mention of Joseph Fernie of Leadenhall St.