I see that the information in LIBINDX came from the Register of Poor. The Keith Registers of Poor are fascinating, because they contain a vast amount of detail, and the Inspector's often blunt comments on the people involved. If Charles was illegitimate, there's a better than even chance that the record will say so. You should be able to get a copy of Charles Mann's record from Moray Local Heritage Centre through the online shop at LIBINDX, unless you are near enough to go to Elgin and look it up yourself.
If he does turn out to be illegitimate, and he was born in Keith, it would be worth taking a look at the Keith Kirk Session records, because the Keith Kirk Session would certainly have taken a close interest in an unmarried pregnancy, and the record, if it exists, would probably tell you where the parties lived, and may even provide other interesting details.
The Keith KS records are in the National Archives of Scotland. They have been digitised, but are only available in General Register House in Edinburgh and a few other Scottish archives - Glasgow, Aberdeen and Inverness, and probably a few more. If you can't get to any of those, you might need to hire a professional searcher.
The 1851 census shows Charles and Lillie and three children in Mid Street, Keith. It gives Charles' age as 36, which implies a birth year of 1814/1815. In 1861 he's a flesher, aged 49, implying a birth in 1811/1812. The 1881 lists him as 70, which implies a birth year of 1810/1811, same as the death from LIBINDX. There are two other Charles Manns in the 1881 census of roughly similar age. One was born in Marnoch and is aged 71. The other is aged 70, was born in Keith, is married to an Isabella and has a family born in Aberdeen, so is evidently a different one.
Whose death certificate have you actually got? Have you got the one for 'your' Charles?