Right
I have come back from the Swansea archives and I have made some progress
Whilst I didn't find William's baptism or his father, I learnt that he died in 1883 in Merthyr Tydfil and he was not born or baptised in Cadoxton Juxta Neath. The William born in Cadoxton is a different William in the 1851 census
He was born in Neath, but not Cadoxton.
Does the marriage between him and Ann Abraham give any mention of a father for William?