Those are all valid points, terianne; that's why it's important (and usually necessary) to search for as much corroboration as possible to prove we're following the correct line.
Andrew Ker was not an uncommon name. Just because there was one baptized in Hawick in 1814 doesn't automatically mean he was the same person as the Andrew Ker who was born in Jedburgh circa 1808 and died in 1876. As you pointed out, some people were non-conformists and were not necessarily baptized in the same town or city where they were born; in addition, not all church records have survived. So the 1814 baptism might be for Violet's father but it might not. (The 1808 baptism most definitely is not, unless there were two Andrew Ker + Mary Blackbel/bill couples who were having children at around the same time.)
Consulting the 1876 death record and the will would be a good start and should be helpful in proving the relationship between the Andrew Ker who was the father of Violet Ker (who married James Grahamslaw). At least that's the hope.
The death record should indicate the names of Andrew Ker's parents; if they're the same parents as the ones in the 1814 baptism, then that's a good reason to spend time researching that family; if they're someone else, then that's a good reason to spend time researching them. As you stated, records can contain erroneous information, so the hope would be to find other documentation that would confirm or correct the information that has already been found. Sometimes that is more straightforward than at other times, depending on the records that are still available.
Regards,
Josephine