The modern lists of if-your-name-is-xxx-you-are-a-member-of-Clan-zzz are largely a load of hogwash invented in the 19th and 20th centuries by the Brigadoon industry as a means of generating sales. Certainly there are some - even many - nuggets of truth lurking among the rubbish, but the clans were a social feature of the Gaelic-speaking Highlands that did not extend into the Lowlands. Most Scots lived in the Lowlands and did not belong to any Highland clan.
You are absolutely right to question this if your grandfather was born in the southern extremity of the country, as far from the Highlands as it is possible to get and still be in Scotland.
Usually, these lists say that Adamson is a sept of Clan Gordon. This is based on the fact that yes, the given name Adam was common among Gordons, and yes, there were Adamsons who belonged to Clan Gordon. But the name Adam is of biblical original and was used all over the Judaeo-Christian world, so there must be countless sons of Adam who have no connection whatsoever to Clan Gordon.
I'm a bit surprised that you say that Touch not the Cat bot a Glove is a Macdonald motto, because it is generally said to be the motto of Clan MacPherson or by extension of the Clan Chattan confederation, which included Clan MacPherson, Clan MacIntosh and others. I've always understood the Clan Donald motto to be Per mare per terras ('by sea and by land') but it's quite possible that a branch of Clan Donald might have used Touch not the Cat bot a Glove.
It is quite likely that somewhere among the forebears of your gt-gt-grandfather George Adamson there were Gordons or MacDonalds or members of Clan Chattan, that his ancestors had moved south at some point, and that the family story of the clan links was handed down over generations.
I suggest that you concentrate on finding out definite facts, and put any possible clan connections on the back burner unless and until you come up with a definite link.