I think your George may have been married twice as the 1851 census has him and his family below at Water Row, Hermitage House, with a 33 year old wife and a son of 23:
From :
www.freecen.org.uk I'll let you go to the site to check. The 1841 census has what appears to be him with a Mary in the household, but she is only 20 and his son James at the time is 13. Relationships not given on the '41 census.
Surname First name(s) Rel Status Sex Age Occupation Where Born Remarks
ROBERTS George Head M M 53 Woollen Manuf Chief Magistrate Selkirkshire - Galashiels
ROBERTS Agnes Wife M F 33 Manufacturer's Wife Berwickshire - Coldingham
ROBERTS James Son U M 23 Woolen Manufacturer Roxburghshire - Galashiels
ROBERTS Eliza Dau U F 21 Daughter Roxburghshire - Galashiels
ROBERTS George Son U M 9 Scholar Selkirkshire - Selkirk
ROBERTS Alexr - U M 6 Scholar Selkirkshire - Selkirk
ROBERTS John Son U M 5 Scholar Selkirkshire - Selkirk
ROBERTS Edward Son U M 3 Son Selkirkshire - Selkirk
ROBERTS Thomas Son U M 1 Son Selkirkshire - Selkirk
BROWN Jessie Servnt U F 27 House Maid Selkirkshire - Selkirk
LAMBERT Jessie Servnt U F 22 House Maid Selkirkshire - Selkirk
- Catherine Servnt U F 17 Nursery Maid Roxburghshire - Hawick Surname not enumerated.
Regards, Steve