I haven't really got anywhere with the Youngs in 1841.
If it is the same Elizabeth on those baptism records in 1835 and 1840, then presumably William Martin born 4 August 1835 had died.
James was born 2nd February 1832
Where does the 2nd February date come from? Do you know when he died, and the age given?
How reliable is the Woolwich place of birth?
There is a James Martin born 3 Feb 1829 and baptized St George the Martyr, Southwark, 25 Feb 1829.
Parents John and Elizabeth, abode Kent Street, father a Labourer.
The same couple may then have gone on to have a daughter Maria, born 18 Jan 1831, who was baptized at St Mary Newington, 16 Feb 1831 (Kent Street, father Labourer)
I would imagine though that there were a few John and Elizabeth Martins about.
The one navy record I can see that might be your man, UK, Naval Officer and Rating Service Records, 1802-1919, on ancestry
quote from records - his ship
Occupation - HMS Medina
James was a stoker (a fireman) in the Navy on the HMS Medina
James Martin
Served on the Medina from 8 May 1854 (age 22) to 24 January 1859
Again from 25 Jan 1859 to 11 March 1861
Then on the Alacrity, 1861-63 (later rating on there "Musician")
On to the Phoebe, 1863-66, Victoria, 1866, finally the Royal Oak, 29 Sep 1866 to 7 May 1867 (Painter 1st Class on all those)
If him, what was he doing before 1854? Was he not in the Royal Navy?
Could James have been in the merchant service?
Incidentally, there is James Martin, born 1830, St George, Surrey, in the merchant seamen's records (BT113)
But I don't have access to this at the moment, so I can't say if it might be relevant to anything.