A lot of messages here in the past few hours!
It is possible he was born after the 1841 census. On all censuses apart from 1851 he consistently ages 10 years and is on a 0 at each census, which would make me guess 1841. It makes sense his birthdate is off a bit on 1851 given he wasn't with his family.
Thank you for finding James & Agnes Wylie, that was elluding me. I am now sure that this Agnes is William's mother.