Who did James Palmer marry? Or didn't he?
Interestingly, there's a Philip Palmer (23) and Harriet (22), with children Margaret (3) and Leonard (6m) in Wolverhampton in 1881. They say everyone is b. Wolverhampton, but I'm having a hard time finding a trace of any of them except for Leonard's birth.
There are a lot of possible Harriet(t) Bills in the 1861/71 censuses, so it will be difficult to pick one as being the mother of your James without more information.