FreeCEN shows the following:
1841 at New Duffus, Moray
Andrew Grigor, 25, journeyman shoemaker
Helen Grigor, 25
Elspit (sic) Grigor, 3 months
- all born Morayshire.
1851 at a close off 174 High Street, Elgin
Andrew Grigor, 36, journeyman shoemaker
Helen Bower or Grigor, 36
Elspet Grigor, 10, scholar
Ann Grigor, 6, scholar
William Grigor, 3
John Grigor, 1
- Elspet and Ann born Duffus, the others in Elgin.
Looking on Moray Council's Libindx site, there are details of an earlier Andrew Grigor, shoemaker. He might be the father of your Andrew (your Andrew's death certificate should clarify it). Just in case, the details from Libindx and FreeCEN for the earlier Andrew are:
Libindx -
Andrew Grigor, shoemaker, born Elgin 2-7-1785, son of Robert Grigor. Married Anne Laing at Elgin. Died 24-9-1860 aged 77 (actually 75) Bede House, Elgin.
1841 census, Little's Buildings, Elgin
Andrew Grigor, 55, shoemaker
Ann Laing or Grigor, 50
James Grigor, 20, shoemaker
Elspet Grigor, 15
Charles Grigor, 12
Margaret Younie, 4
- all born Moray
1851 census, Littlejohn's Close, Elgin
Andrew Grigor, 69 (sic), shoemaker
Ann Grigor, 63
Clementina Grigor, grandchild, 3
- all born Elgin
NB in your post you mentioned "Shanbryde". The parish is actually St Andrews-Lhanbryde (the two parishes merged in 1781). Lhanbryde is the village.