I note that when Joseph (labourer of Tower Street) and Emma's son Henry was baptised in 1885 Joseph gave his full name as, Joseph Henry Marsh. He was Joseph on all subsequent baptisms for their children (1887 up to 1900) which were at St. Bartholemews Church in Chichester. Having failed to find Joseph on any earlier documentation, one line of thought is that he could have initially been listed as Henry in those earlier censuses and records. FreeBMD however, does not seem to have a birth registered for him as Joseph or Henry Marsh in the Chichester area around 1853, although there is a birth registered in Chichester in 1851 for a Henry MARCH who appears to come from Bury nearby.
There being few other alternatives, you might be interested in the 1851 Chichester census which shows a 26 year old unmarried Charlotte Marsh with her 7 year old son William Leonard Marsh. SFHG and IGI record his baptism at St Martins Chichester in 1843, to Charlotte Marsh, single woman. Charlotte, who had been baptised at St. Peters Chichester in Nov 1824, was the daughter of Thomas & Mary Marsh. You could investigate to see if Joseph was a further base born child of Charlotte's, who may appear in the GRO birth registers under his father's surname (rather than Marsh) and is then entered in the 1861-1881 censuses under the surname of who ever had responsibility for his care.
It is not uncommon in those circumstances to only revert to a parent's surname when finally marrying. For example, there is a double record for the birth of Charlotte's son William Leonard Marsh in FreeBMD 1843, for he also appears in those records as William 'Leonard.' When he marries later in Westbourne near Chichester in 1864 he is back to being William Leonard Marsh. Charlotte and William L Marsh also appear in the Chichester census indexes for 1861, but I do not know whether they were together or if Charlotte had any other children with her. They appear in the reverse order and I speculate William was just in the care of Charlotte's parents by then. Finding that, or those two entries in the 1861 census might be a rewarding line of further investigation.
Roy G