I suspect that James Glover might be an even harder nut to crack, even assuming that Mary was being truthful when she told Alexander about his father.
I don't imagine that there are many diamond cutters in or near Buckie! So presumably Mary was working away from home when she fell pregnant, and came home to have the baby. I know there is a diamond cutting industry in London, but I am not aware of there being any nearer Buckie, so that would be a topic to look into. A quick search online is not helpful because it produces lots of results for present-day diamond cutting services.
There is no mention of a James Glover in LIBINDX
http://libindx.moray.gov.uk/mainmenu.asp.
I had a look in the Rathven Kirk Session records but found nothing. It's possible there might have been a paternity case in the courts, but so far it doesn't appear in
https://www.scottishindexes.com/default.aspx, though more cases are being added all the time.
If he didn't leave the country, he should be in the 1881 census. According to FindMyPast there are 229 James Glovers born between 1825 and 1865 in the 1881 census. It would be a longish but not impossible job to look them all up and see what their occupations were, and I think that would be the best way to go about looking for him.
I see that in the 1901 census he was married to an Elizabeth; that Alexander Glover and Elizabeth Jemim* Ross were married in Edinburgh in 1900, and that Elizabeth Jemima Ross or Glover died in Edinburgh in 1920 aged 39. Do you have a copy of that 1900 marriage certificate? What does it say about Alexander's father?