Is this your Henry in 1861?
Living at Ship Tavern, 4, Great James Street, St James, Bristol, Gloucestershire
Birthplace is given as Bath, Somerset. Wife is Louisa.
Registration district Bristol
Piece number 1721
Folio 28
Page 13
Where did you find the information that he was born in Westbury on Trym?