Yes, illegitimate means born to an unmarried woman.
There are two people named Joseph Hey, both born in 1856. If you look at the births index on
http://www.freebmd.org.uk/you will see that one was registered in Hunslet district in June quarter. This is the child born to Ann Hey in Burley (which is in Hunslet registration district, near Leeds), who was unmarried at the time Joseph was born, and is still unmarried at the time of the 1881 census. His baptism record shows his birthdate as 29 June 1856.
This Joseph Hey is the man who married Ann Ellen Fox. In 1891 he is aged 34 living in Leeds with his wife Ann and children. His occupation is shown as 'forgerman' and he names Leeds as his birthplace.
The other Joseph Hey birth was registered in Dewsbury district in September quarter. This is the child born to William Hey and Ann (maiden name Goldthorpe), who married in 1855. His baptism record shows his birthdate as 17 July 1856.
He appears in 1861 with his parents in Dewsbury, and in 1871 with his grandmother, Mary Goldthorpe in Dewsbury.
In my previous post I said that this Joseph Hey married Mary Ann Pollard in 1879 - and I think that this is correct, Joseph Hey aged 24 born in Dewsbury appears in the Pollard household in 1881. I don't have a definite sighting of him after that.
Without knowing what your starting point is, I am unsure whether your man is the one who married Ann Ellen Fox, or the one who is the son of William and Ann (Goldthorpe) Hey? However, I don't think that they are the same person.