Joseph & Charlotte are still there in 1861, but George isn't:
1851
HO107/1716 ed4 folio 252 page 11
Marsworth, Buckinghamshire
Joseph Carter head m 40 ag. Lab. Bucks. Little Ramsdon
Charlotte “ wife m 37 straw plaitter Bucks Pightlestone
George “ son 9 “ “
Henry “ son 3 “ “
1861
RG9/842 ed4c folio 51 page 13
Pitstone, Buckinghamshire
Joseph Carter head m 47 ag. Lab. Bucks. Laylands
Charlotte “ wife m 44 Bucks Pitstone
elizabeth “ daur 22 plaitter “ “
Henry “ son 14 ag lab “ “
Sarah “ daur 9 “ “
Charles “ son 5 “ “
Frederick “ son 8 months “ “
hope this helps
regards