You're right of course, to be cautious about the ages - although I do find the 1841 a bit confusing with it's rounding down and such like, maybe this will swing it for you?
1851 Bristol St Phillip & St Jacob
4 Broad Plain
Thomas Loyd Head 43 Mar Master Iron Founder Employing 9 Men 2 Boys b. Bristol
Jane " Wife 47 " "
Charles " Bro 29 U Engineer "
Thomas " Son 21 " Moulder "
Henry " " 19 " Engineer
Sarah " Dau 18 " Dress Maker
William " Son 16 Bookbinder
MaryAnn " Dau 14
George " Son 9 Scholar
HO107/1954/691/40
So Charles, right age this time, is an engineer as in 1841. Head of family Thomas corresponds with Thomas from your family AND Thomas's son George is most probably nephew George that your G.G.Grandfather lives with when he returns to Bristol.
Still no more information on parents but I'm working on it...
Regards,
Lee