I think that we who are trying to help are going to need some more information to be of use.
You show his birth 1830 as 'Plymouth?'
This might well be correct but not necessarily the home of his parents who may well have been visiting there (from Frome perhaps?) or perhaps his father may have been 'ex Frome' and a military man (Marine or Sailor) stationed there and married there.
On the other hand he may well have been the 1st born in his family in which case,as was commonplace, he may have been born in his mother's parish (Plymouth?) and his subsequent siblings in Frome or thereabouts.
The significance of the battle of Trafalgar at this stage of your quest may be a dead end as the medal may well have been found or acquired from a shop or similar as a 'curio' and for the time being perhaps it should be placed on the backburner.
There was a Trafalgar 'Commemoration medal' and a Trafalgar 'Campaign medal' and you could identify which one you have by 'googling' and let us know.
The 1851 census will give place of birth and there is living at Frome;
Karen Dole widow 58 'Keeper of quarry' born Herfordshire and her grandson James Dole 21 born Devonport (half a mile from Plymouth) a quarryman which is a good bet for your James born Plymouth 1830.
Going back to 1841 census (where no precise birthplace or birthdate is shown) we have living at Westbury workhouse;
Elizabeth Doel 60 aglab 'not born in county' and James Doel 12 aglab 'born in county' which might indicate that his parents had died and he was with his grandmother.
I cant help but thoroughly concur with suzard's input.
joboy