There is this from a past RootsChat post, it does'nt mention James but may give some clue's...
"What is interesting is in the 1841 and 1851 and 1861 censuses, Mary Ann Walder and her family lived a couple of cottages down from the Rayward family in Warninglid, Sussex. Several of the Rayward children Abraham and Thomas wed in Brighton in 1856 and 1862 then returning to the Slaugham area. They were grocers by trade and one of the Rayward brothers lived in Viaduct Terrace in Brighton in the 1871 census, but also resided mainly in Clayton several miles north. What is ironic is that before Thos Roberts moved to London in 1864, he lived at Viaduct Terrace in Brighton.
The Raywards must have been good friends of the family as another sibling, Henry H Rayward registered the death of Mary Ann Walders mum, Mary Ann Walder wife of John in 1845. Henry was in Brighton in 1851 and died in the Slaugham area in 1860. So these Raywards did fluctuate between Brighton and Slaugham. I reckon maybe Mary Ann Walder met Thomas Roberts through the Raywards."