Also I wonder if he really was born in Banff as I can't find any record of his birth in the OPR for Banff Parish. He may have just said that for the census as he had to say something.
It certainly looks as if he believed he was born in Banff in 1820/1 because he was consistent in his replies throughout the census. It is far more likely that he was genuinely born in Banff and that for some reason his baptism was not recorded, or if it was the record has not survived. This is common enough to be the most likely explanation. I have hundreds of people in various families born in the first half of the 19th century who are missing from the records.
The Statistical Account of Banff says that at the time of writing (1797) there were in Banff, besides the Church of Scotland parish church of St Mary's, a Relief church with a congregation of 400, an episcopalian congregation of about 300, and about 100 Roman Catholics.
I take it that you have checked the Roman Catholic records at Scotland's People as well as the OPR? If there are extant records of the relief or episcoalian churches, they may be in the National Archives of Scotland
www.nas.gov.uk under catalogue No CH3/
According to an obituary that I have for him he apprenticed as a silversmith prior to becoming a blacksmith. I roughly calculate that he was only 11 or 12 when he started that apprenticeship. So he may have left home very young or may have been orphaned at that time??
That's possible. Or he could have been illegitimate, or his family could have been very poor.
If he was illegitimate, there might be some record in the minutes of the Banff Kirk Session, which are in the National Archives of Scotland
www.nas.gov.uk under catalogue no CH2/1109. You need to go to Register House in Edinburgh or one of a small number of local archives which have access to the digitised verson of this.
There's no point thinking about the Poor Law records because they don't start until 1845. Before that, the relief of the poor was the responsibility of the parish, and the Kirk Session records may record payments to pauper families, or other expenditure such as purchasing clothing or boots for pauper children. I suggest that that is your next move.