Thank you everybody for the helpful suggestions. I apologise for taking so long to reply - I've been unwell. Regarding John Gibson's death certificate, it says his mother was 'unknown'. Usually I've found more information on death certificates of my Orkney ancestors. His son is shown as the informant, so that's very odd.
Regarding the birthplace, I managed to find an image of the original 1851 census which is where the birthplace 'Hoxaybridge' comes from. I only had a transcript when I first posted and I thought the error must have been in the transcription - but no. It definitely says Hoxaybridge, Yorkshire. The 1841 and 1861 censuses just have his birthplace as England. So, following Colin's suggestion, I searched Yorkshire place names and imagined John Gibson with a Yorkshire accent, and an Orkney census taker misunderstanding what he said. I'm thinking the strong 'x' sound must have been in the correct name and wondered about whether the actual birthplace is one of these: Haxby, High Hawsker, High Oxnop, High Stakesby, or Roxby. All of these are in North Yorkshire, coincidentally.
Given that I have no marriage certificate as yet, which might give more details, does anyone have another suggestion as to where I might start with this idea of similar sounding place names? I'm thinking it's a hopeless task, but thought I'd pose the question anyway, just in case.