Hi G -- try putting a post on the Devon board (with a link to this one to avoid duplication!) -- DOB7 there has access to parish records for Devon (which is notoriously hard to research pre-1837 because records are not available on line and must be purchased) and should be able to give you a hand at looking for James in that county.
... Actually, I see that even familysearch has a couple James Giles-s baptised in Devon in the mid-late 1790s, so identifying yours, even if he was from there, could be impossible.
But the one born in Broad Clyst is still there in later censuses, leaving one baptised in 1796 in Stoke Damerel (the Plymouth area) which would be a very reasonable source for someone who went to Nfld, as it is a seafaring/naval place. However, father's name Peter isn't reflected in your James's children's names.
https://familysearch.org/pal:/MM9.1.1/N5ZF-BH5Peter and Elizabeth show as having one other child, Nancy, 1798, baptised at Morrice St Wesleyan in Devonport (also generally the Plymouth area):
https://familysearch.org/pal:/MM9.1.1/NPW2-2HDand there's a 1794 Nancy Hibly(?) Giles
https://familysearch.org/pal:/MM9.1.1/FQZ5-BY3also recorded as Nancy Shirbly(?) Giles
https://familysearch.org/pal:/MM9.1.1/JQYJ-9D9(so it looks like the first one died)
... and there is another, Mary Ann 1789
https://familysearch.org/pal:/MM9.1.1/FWGR-8XWhttps://familysearch.org/pal:/MM9.1.1/JMJN-P6VUnfortunately, since they are girls, there is no point looking for them in 1841 as they would have different surnames.
It's always possible that your James went to Nfld as a child with family; you could try looking in Nfld records for any of those names too.
It would be a matter of trying to rule out a death/marriage/other presence in England later on, for any possible you identify.
And of course, as a quick search at familysearch.org shows, there were several James Giles-s born in Kent in the mid-late 1700s ...
(If your Mary Ann was from Nottinghamshire, we might be related,
(see my sig line) -- unlikely as a source of someone in Nfld unless her father was military maybe.
-- But, if that mangled name happened to be Sibl(e)y, well, we could still be related. The nest of Siblys from St Cleer, Cornwall, are mine, and I see my grx3 grandmother's brother Samuel is in Devon in 1851, and seems to have married in Stoke Damerel in 1795 ... and as a matter of fact, their sister Elizabeth born 1772 seems to have gone missing from marriages in Cornwall ... heh heh)