I think you really need to explore each member of the 1841 family of Ashworths quite carefully, and maybe see what they put for their places of birth in later censuses.
I have started to do this and have come across something quite strange and Spencer-related!
I looked for the oldest Ashworth brother John in later censuses and although I couldn't find him in 1851, I found this family in the other censuses that looks like the right one:
1861 - Potters Row, Radcliffe
John Ashworth Head Mar 38 Dyer Lancs, Radcliffe
Ann Wife Mar 36 _ Lancs Radcliffe
Robert Son Un 17 Dyer of Cotton Lancs, Radcliffe
Mary Ann Daur Un 12 _ Lancs, Radcliffe
Sarah Daur Un 6 _ Lancs, Radcliffe
Thomas Son Un 2 _ Lancs, Radcliffe
1871 – Back Chapel, Radcliffe
John Ashworth Head Mar 48 Farm Labourer Lancs, Pilkington
Ann Wife Mar 47 _ Lancs Radcliffe
Mary Ann Daur Un 22 Cotton Weaver Lancs, Radcliffe
Sarah Daur Un 16 Cotton Weaver Lancs, Radcliffe
Thomas Son Un 12 Scholar Lancs, Radcliffe
1881 – Bowker? Road, Radcliffe
John Ashworth Head Mar 58 Chemical Labourer (unemployed) Lancs, Blackford Bridge
Ann Wife Mar 57 _ Lancs Radcliffe
Sarah Daur Un 26 Cotton Weaver Lancs, Radcliffe
Thomas Son Un 22 Stamp Maker Lancs, Radcliffe
I was puzzled as to why I couldn't find them in 1851 as they would already be married by then and their elder two children Robert and Mary Ann would be 7 and 2. Then I think I found them:
1851 – Radcliffe Hall, Radcliffe
John Spencer Head Mar 28 Dyer Lancs, Radcliffe
Ann Wife Mar 26 _ Lancs Radcliffe
Robert Son Un 7 Scholar Lancs, Radcliffe
Mary Ann Daur Un 2 _ Lancs, Radcliffe
Alice Hughes Visitor Mar 25 Weaver Lancs, Radcliffe
Mary Mitchell Hughes Visitor Daur Un 1 Lancs, Little Bolton
Now, am I going crazy or does this look like the same family??? And they are called Spencer.
This is the same census where my ancestor James is living in Radcliffe and calling himself Spencer before changing to Ashworth in 1861.