Ok, from 1841:
Parents William and Mary born around 1800-1805, William is a groom. Charles b 1831, William Jr b 1834, Elizabeth b 1836.All born in Nottinghamshire apart from Mary.
From 1851:
At Mill Cottage/ Finningley, nr Doncaster are William (50, Gentleman's servant, born Ordsall Notts), son Charles (20, Farm servant, born Retford Notts) and daughter Elizabeth (15, born ditto). William Sr is married, not widowed, but Mary is elsewhere as is William Jr. There are a couple of possibilities for William, but Mary is harder without a birthplace.
From 1861:
William (59, Groom) working at Barbott Hall, Greasborough, Notts. Listed as unmarried, but suspect widowed.
William Jr is possibly at New York, Brimsworth nr Rotherham, coal miner, with wife Mary Ann.
Mary, Charles, Elizabeth and Thomas C are nowhere obvious.
Looking for deaths:
First name(s) WILLIAM
Last name MILNER
Birth year 1800
Death year 1864
Age 64
Burial year 1864
Burial day 7
Burial month 5
Place FINNINGLEY
Church description HOLY TRINITY & ST OSWALD
County Nottinghamshire
Name: William Milner
Registration Year: 1864
Registration Quarter: Apr-May-Jun
Registration district: Rotherham
Inferred County: Yorkshire West Riding
Volume: 9c
Page: 293
First name(s) MARY
Last name MILNER
Birth year 1801
Death year 1856
Age 55
Burial year 1856
Burial day 18
Burial month 9
Place FINNINGLEY
Church description HOLY TRINITY & ST OSWALD
County Nottinghamshire
Name: Mary Milner
Registration Year: 1856
Registration Quarter: Jul-Aug-Sep
Registration district: Doncaster
Inferred County: Yorkshire West Riding
Volume: 9c
Page: 246
Could this possibly be Thomas Chester Milner?
Name: Tom Milner
Birth Date: abt 1857
Date of Registration: Mar 1918
Age at Death: 61
Registration district: Rotherham
Inferred County: Yorkshire West Riding
Volume: 9c
Page: 953
That's as much as I managed with a quick bit of poking about,perhaps someone else will turn up some more little nuggets?