Don't hang your hat on that family as you need to check a lot of things first...You need to look for marriages and beyond that...
Being that the wife and a few children were in the Canada 1861 you'd have to look at the 2 dates of the census to see if they could be on two at once? then prove they were no longer in Scotland, ie next census......Myself, I think you'll have a wee bit of a problem just for the fact that there are no Anns listed in that family?