The following is the only possibility I can find born in Norfolk of around the right age:
RG9/435 folio 14 p23
District Foot Guards detachment, North Camp, Aldershot, Surrey
William Bailey, head, married, 30, Private, Commeth - Norfolk
Annie Bailey, wife, married, 28, Burnham - Essex
There are nine Baileys born in Hingham on the 1861 census but none named William:
Charles Bailey b c1837, Elizabeth Bailey b c1841, Joseph Bailey b c 1827, Robert Bailey b c1822, Edmund Baley b c1839, Elizabeth Baley b c1846, Ellen Baley b c1841, George Baley b c1848, John Baley b c1801
All the best
Philip