Can't see a naturalisation for him (not unusual to see "naturalized" on census but have no paper trail to prove it, though - lots of people didn't actually bother naturalising). Unfortunately without further information about where in France he came from you will have a hard time tracing him back.
You might try tracking down early directories for Portsmouth/Portsea to see if you can see him as a watchmaker.
1881 (Bathia transcribed as "Martha")
https://familysearch.org/ark:/61903/1:1:X31R-D7QBathia I think is probably the "Bathia Douglass" death on freebmd.
It occurs to me that if his age in 1871/1881 is about right, he could have been a young French soldier or sailor. There were many French prisoners of war from the late 1700s up to 1814, and many were held in the Portsmouth area (in May 1814 there were more than 20000 in Portsmouth). I found one source indicating that "boys under 12" if captured were not held prisoner but sent back, which suggests that a teenage Joseph could have been among this number. There are records held at the National Archives in Kew, but it would require a visit in person I believe.
