You can call yourself anything you like as long as there is no intent to defraud.
I've seen baptisms where an illegitimate child does have the father named (sometimes as "reputed father of") but those are only a small % of illegitimate births, and I assume in such a case the relationship between the parents was well established and recognised, even if they weren't married.
Sometimes illegitimate children were given the father's name as a middle name, which can be a clue, but not if there's no sign of a father! eg my gtgtgrannie was baptised Augusta Upton Marshall; a year and a half later her mother, Fanny Marshall, married Augustus Upton and little August becamse Augusta Marshall Upton. I conclude that Augusta was likely to have been Augustin's daughter (DNA has backed this supposition).
Back to your family - James could have been Joseph's genetic son, but you've no evidence of this. I find that step children often were given the new stepfather's surname on the censuses (in fact I'd say the majority that I've seen have done this), only reverting to their original name when they married (if ever).
