Sillgen: I did say TRY, but perhaps it should have been, "consider!"
The problem with places like Worthing is that it can be listed as a person's birthplace, when they actually mean, "IN THE VICINTY OF.... !" Having found no hits on Worthing I considered the surrounding area and alternative parents. Places like Broadwater, Tarring, Findon, Goring, Angmering, Patching, Clapham, Salvington, etc.
I did find a Charles Mitchell who married a Sarah Pullen in Broadwater in 1808 (IGI) but the IGI showed no sons being born to them, only 5 daughters (1809-1817) Could they have relocated to somewhere also in that area that has yet to appear on the IGI or the Sussex Family History Group Baptismal index?
Sandym: It looks like him in Cranbrook Union (workhouse?) in 1851, but have you tried to find your particular Joseph on the 1841 census? Both suggested fathers are listed as living in Sussex, but neither have a son Joseph with them. If we knew your Joseph's occupation, it may also help towards clarifying whether there was any connection with milling?
Lastly, there are also four people on Genes Reunited that have your Joseph 1823 listed in their ancestries. Two correctly give his birthplace as Worthing, the other two have it (mistakenly?) as Strood in Kent where he once lived with his family.
Roy G