Thank for the information of missing Census data. Yes, it is possible that Benjamin Smith took his family back to Northants, where he was born.
Benjamin and Mary Smith may have had other children in Hertford and I can not find them (such a common name) in 1841 as well.
Just in case, some helpers need further clues, then their POSSIBLE known children were:
Thomas Smith bp. 7 Feb 1808 Datchworth, father trampers;
Moses Smith bp. 12 Feb 1815 Wheathampstead, father brazier;
Sharlender Smith b. 1816; bp. 27 Jul 1828 Datchworth, father tinker;
James Smith bp. 28 Sep 1817 Aspenden, father strangers,
maybe William Henry Smith b. c1820 or 1822 Hertford, alias Tinker Smith sent to Tas 1842;
Sylvester Smith bp. 1 Jul 1821 Bengeo, father labourer
Caroline Smith 9 Oct 1825 Graveley father's details not given
Jemima Smith bp. 18 Jul 1830 Datchworth father tinker
George Henry Smith bp. 17 Feb 1833 All Saints, Hertford, father brazier; address Liberty of Brickendon

?