Henry and Mary appear with their family in Sussex in 1841 (consistent with Robert's baptism in New Shoreham); at an address in Medina Road, West Cowes in 1851 and by 1861 they are in London at Gunter Grove in Chelsea.
It looks as if Henry was returning to his roots (consistently given as London, Middlesex) in retirement.
Have you followed through on the witnesses to the 1834 marriage - George and Mary Ann Andrews? Might they be related?
Apologies if you have already considered this, but might Henry have been James Henry at baptism?
Several newspaper announcements for the marriages of his children and his death.
With reference to one of your other threads on the same man, the Ages and Capacities book for Cowes (1839-1922) has the TNA reference CUST 61/179.
I'm also wondering if his father might have also been a Civil servant in some capacity and this is how he came to apply as well.
Nell