Were he and Elizabeth maybe of non-Anglican stock and tucked away in the records of the Unitarian / Independent / Methodist / Congregational chapels?
Looking at the Harvey baptisms they and some of the rest of the family were Unitarians.
One of your problems is that there are two George Harveys in 1861 of the same age and born in Bridport, but only one (the son of Charles and Emma, so not yours as the other one is living with Emma in 1861) apparently in earlier censuses (and even then he was indexed as Hervey and Harney).
It may be that your George was born illegitimate but his mother then married a Harvey, so he changed his name. I did find two possible baptisms (George Lockyer and (William) George Hounsell, but the former became a sailor rather than shoemaker and the latter, although marrying an Elizabeth (Jane), was still calling himself Hounsell when he married in 1853 in Yetminster. I haven't treid to find them in later censuses.
Richard