Yes, that is exactly the point. What a pain!
All five of James McHarg's children were baptised in Portpatrick. The Rev Andrew Urquhart was minister of Portpatrick from 1832 until the Disruption in 1843, so he would have baptised at least two of Elizabeth Wither's McHarg children and both of Minnie Adair's.
It is not credible that James McHarg could have set up house with Minnie Adair, either bigamously or unmarried, without the minister and at least some of the elders knowing about it, and hence that a child of his and Minnie's would have been baptised at six days old without some sort of record in the Kirk Session records.
Therefore I agree that it is unlikely that the Elizabeth Wither in Leswalt is the same person as James McHarg's wife.
The Portpatrick Kirk Session minutes from 1833 to 1853 are in the Virtual Volumes on Scotland's People. Go to
www.scotlandspeople.gov.uk > Search our records > Advanced people search > Kirk Sessions > Search record creators > and type "Port Patrick" (two separate words) in the search box, and then click on 'Search'. Select the volume with 1833 to 1853, and read through the records from 1834 (baptism of Elizabeth Wither's youngest child) to 1839 (baptism of Minnie Adair's first) and see what you find. Pay particular attention to the accounts as they may contain a mortcloth payment in respect of Elizabeth Wither or McHarg that would settle the question.