Thanks Allan,
Withers, that is something to go on. That does seem to match quite well with what I have. I have seen the 1851, 1861 and 1871 census but the ages of Thomas, Mary and William do vary, by 2 to 3 years each census. In 1861 William and Sarah are married and live a few houses down from Thomas and Mary Brierwood. I note that in the 1871 Thomas and Mary are living near Charles Felton, who I am thinking could be a brother of Williams as the age fits (but I am still looking for his parents too).
More digging to do...