1841 LAYTON please

[fossey]

I am trying to find a family in the 41 census. 

I have the names of Jane b 1822, Mary b 1827, James  b 1829, George b 1832, Thomas b 1835

These are children to James b 1801 and Mary b 1803 (Layton).  The only thing is on the 41 census I have only James was living with them (along with a John, who turned out to be the nephew is 51).

I was just wondering if these children were living with someone else, or had died by 41, or they weren't the children of this family after all.

Any help would be great.

Thanks

Angela