Possible, especially given that Charles wasn't born in Moray. Aged 79 in 1841, he would have been born 1761/2. I wondered if he could be the one baptised in Boharm on 2 November 1761, son of Alexander Phin and Jean Young. Part of Boharm is in Banffshire, so this would make sense.
I note with interest that in 1841 a Charles Phin, aged 15, is in the household of Janet Young, 65, and her family Margaret, 45, grocer; James, 35, schoolmaster and John, 35.
In 1851 Janet Young is listed as 79, born Cromdale, and Margaret (57) and James (50), parochial schoolmaster, were born in Dallas. John Young (48), flesher, is in School House, Dallas with nephew Charles Phinn, 25.
In 1861, James, 60, is schoolmaster, and in the household are his brother John (58) flesher and tenant of land; sister Margaret (67), grocer; nephew Charles Phinn (34), born Knockando; visitor Ann Phinn, widow, 63, born Dallas and George Gammie, visitor, aged 5, born Mortlach. I just happen to know (because this George Gammie married a distant cousin of mine) that George Gammie's mother was one Margaret Phin.