Could this be William in 51 & 61, if he remarried and stayed in Southwell?
1851 census: Westhorpe, Southwell, Notts (Ref: HO107/2134, Folio 398, Page 7)
William Richardson, Head, Mar, 43, Cottager, Epperstone Notts
Jane Richardson, Wife, Mar, 28, Southwell Notts
Job Richardson, Son, 6, Scholar, Southwell Notts
1861 census: Westhorpe, Southwell, Notts (Ref: RG9/2470, Folio 83, Page 10)
William Richardson, Head, Mar, 52, Ag Lab with 13 Acres of Land, Notts Epperstone
Jane Richardson, Wife, Mar, 35, Milk Seller dairy, Notts Southwell
Job Richardson, Son, Un, 16, Farmers Boy, Notts Southwell
This might be the right William if in 1851 his son John was staying with his sister Ann White (formerly Richardson) also born in Epperstone. Ann could have married in 1842 - Warsop is in Mansfield district I think:
Marriages Mar qtr. 1842 (on same page)
White, William Clay, Mansfield, vol. 15: 605
Richardson, Ann, Mansfield, vol. 15: 605