I see that a James Stuart and Janet McGregor were married on 4 June 1803 in the parish of St Machar, Aberdeen. Are these your James and Janet, I wonder?
I also see that there seem to be two records of a marriage of Peter Stewart to Margaret George - one in the parish of Banff on 21 June 1840, and one in the parish of Kirkmichael on 18 November 1841. Normally, you get two records, at roughly the same time, if the couple lived in different parishes, but these are 15 months apart, so are they two different couples with the same names?
Pursuing this line of thought, there are 8 references to baptisms of children of Peter Stewart and Margaret George
Margaret, 5 January 1842, Gamrie
John, 11 January 1842, Kirkmichael
Margaret, 18 October 1844, Kirkmichael
James, 20 February 1845, Gamrie
Janet, 22 May 1849, Gamrie
John, 17 May 1851, Gamrie
Mary Anne, 10 December 1854, Gamrie
Jean McPherson, 10 December 1854, Gamrie
Noting the dates of birth of both the first John and the first Margaret in 1842, it is clear that they are indeed two different couples.
Now, to the census; In 1861 your ones were at Alton of Melrose in Gamrie: Peter, 53, shepherd; Margaret, 46; John, 9; Jean and Mary, 6. I also see that this is where 'Shore of Gleady' came from, and that the transcribers were doubtful about this interpretation.
See
http://www.geograph.org.uk/search.php?i=71201963 for some photographs of this Melrose (not to be confused with the larger and much better known town of the same name in the Borders).
In 1851 the family were at Greenskares, Gamrie.
See
http://www.geograph.org.uk/search.php?i=71201997 for some photographs of Greenskares. The family in 1851 were Peter, shepherd, 43; Margaret, 36; Margaret, 9; James, 6; Janet, 1.
This census says Peter was born in the parish of Glenelg, which makes far more sense than 'Shore of Gleady'. Glenelg is the most southerly mainland parish on the west coast of Inverness-shire. See
http://www.genuki.org.uk/big/sct/INV/Glenelg for a description of the parish and
http://www.geograph.org.uk/search.php?i=71201918 for some photographs of Glenelg.
Unfortunately the parish register of Glenelg seems to be a bit patchy, according to
https://www.nrscotland.gov.uk/files//research/list-of-oprs/detailed-list-of-old-parochial-registers-of-scotland.pdf, which says that there were only 11 entries before 1805. So the chances of there being any surviving record of Peter's baptism, or of any of his siblings, seem small.
I also wonder whether the 1803 marriage can possibly be that of Peter's parents rather than of another couple with the same names? Glenelg is 180 miles from Aberdeen, through some rugged terrain, which in early 19th century terms is a very long way - several days' journey, in fact on foot or horseback or in a cart, or even by sea round the notoriously stormy north of Scotland.
(BTW James Stewart and Janet McGregor must have been born well before 1797 - they could hardly have had a child when they were both just 11 years old.)