Sorry I cannot see Samuel in the 1871 census. I tried 1851 Wilton to see if he was there at that time and did not find him then in Wilton either. Here are the Downtons in the Parish of Wilton and Ditchampton in 1851 including William and Martha
HO107/1848Folio 29 Page 7
South StreetPENNY Isaac 68 Gardener Wilton, Wiltshire
PENNY Mary 69 Wife Wilton, Wiltshire
DOWNTON Grandson 17 Ag Lab Wilton,
Wiltshire
Folio 30 Page 11
Mill LaneCHALK Sarah Widow/er 67 Pauper parish Clerks Widow/er Wilton, Wiltshire
HASKETT Sarah 36 Needlewoman Wilton, Wiltshire
DOWNTON John Lodger Widow/er 65 Pauper Carpet Worker Wilton, Wiltshire
Folio 49 Page 11
North StreetDOWNTON George 35 Rug Weaver Wilton Wilts
DOWNTON Lydia 34 Wilton Wilts
DOWNTON Sarah 5 Scholar Wilton Wilts
DOWNTON Nathaniel; 3 Scholar Wilton Wilts
DOWNTON Mary 1 Wilton Wilts
FELTHAM Sarah Lodger 75 Pauper Washerwoman Broad Chalk
Folio 69 Page 1
DitchamptonDOWNTON Isaac 30 Rag Gatherer Wilton Wilts
DOWNTON Diana 36 Haxton ? Wiltshire
DOWNTON Frederick 7 South Newton
DOWNTON Thurza 2 Salisbury
DOWNTON Harriet 3m Ditchampton
Folio 72 Page 6
Ditchampton DOWNTON William 68 Carpet Weaver Wilton Wilts
DOWNTON Martha 63 Wilton Wilts
CLARKE William Grandson 2 Middlesex London
Cheers Kris