Hi
I wonder, if your NAYLOR family was aware of the James NAYLOR a Quaker who was a Preacher man of apparently some note (enough to have poetry written about his last words according to google), well more than a century before your ggrandfather? And somehow a bit of a family joke about the same name and piety got warped along the descent of generations?
This certainly happened in my family, though the space of years was not so great - my gg-grandmother had the same surname, and the same father's name as that of a wealthy shipping company owner near whom she used to live with her own family. The story came down the family that she was the shipowner's daughter, and he had one stage disowned her for marrying my gg-grandfather (though why she should then be practically living on said shipowners doorstep not so long later.......). Anyway, I was able to disprove the theory entirely, no connection whatsoever. Her real father had died when she was very young. I can imagine her making up some silly joke/tale to appease/tease the questions of little ones, which got passed on, twisted and remembered as a "family fact".....
Of course perhaps your James was indeed a preacher of some sort, but perhaps something to keep in mind
http://www.geocities.com/quakerpages/1694fox000c.htmCheers
AMBLY