Hello and welcome!
There is a very long article trying to sort out fact from fiction as to how old he really was, that was run in the Banffshire Journal and General Advertiser 28 January 1868.
I've taken a few notes from it, pertaining to anything that might help you see if your Gardens are, in some form or another, related. They may not help, but as he only had one son to pass the name down, who knows!
Peter was, according to relatives (long after he died, so family lore), born on the estate of Muireak, parish of Tariff. From a branch of the same family as Peter Garden of Delgaty, (stocking merchant of Turiff that did a lot of trade with Holland.) Also related to Garden of Troup.
Peter was married twice. The second wife was the widow of a Mr. Cruikshank.
He had one son (first wife), George, who died age 80, lived at Darley, had four sons and one daughter.
One son was a clergyman, another a baker in London, another a barber in Peterhead, and another a labourer at Peterhead and Pitaligo. The daughter married in Turriff to a gardener, Mr. Cruden who at one point had the farm named Barnyards of Delgaty.
Peter took up permanent residence in Auchterless prior to 1700, a holding named Ardley.
There are no direct descendants in Turiff or Auchterless; there is a family in Auchterless of close kinship.