Thanks, Jack.
I like the consistency in his age across the censuses and at his death, which point to him being born in 1786. And although the Calverley connection after he left the Army is a strong one, that might just have been because his wife came from that area (see the 1851 Census where her birth place is Calverley and his is Bradford). My instinct would be to trust the census entries where his birth place is shown as Bradford, and then look at the enlistments which match those two moderately reliable facts (place and year of birth) and draw up a short list of possible regiments. Since he was a Chelsea pensioner, and the Chelsea records are fairly complete, they would be my starting point, and then try and reconstruct his military career from that angle. So for instance that rules out the Royal Marine (not a Chelsea pensioner), the 53rd Regiment / 4 Garrison Battalion man from Clayton (wrong place and year of birth), and the man born in Calverley c. 1788 who went into the 3rd Guards (and in any case was not discharged until 1826), and so on.
Also, while I agree that it is most likely that he joined a Yorkshire or West Yorkshire Regiment, don't discount other Regiments. All the Infantry were recruiting hard at that time and I have no doubt many will have gone to the densely populated area around Leeds / Bradford / Halifax on recruiting tours so he might have been persuaded to join one of them. What I don't think he will have done is to travel hundreds of miles to join a regiment which was only recruiting, much further away, say in Scotland or Ireland. However do not be misled where it says he was attested at such and such a place. That will probably be the depot of the regiment and not the place where he was actually recruited. The Chelsea records frequently don't mention where a man was recruited.
At the end of that process you still won't be certain you have the right regiment, but it might allow you to start digging deeper into your candidate regiments, for instance by researching the various Regimental museums, histories and associations to try and find further confirmation to support your working hypothesis. Also keep in mind the muster rolls I mentioned before. They won't contain any additional identifying information to confirm or displace your theory, but they may just throw up a detail which might lead you to another source. For example, although leave was not very common, if he was granted leave the roll might just refer to where he went on leave.
And if you haven't done so already, might I suggest trying to track down an image of the marriage register for St Wilfred's in Calverley for 17 Jan 1821. It is just possible that the actual register might contain some detail about his service career which the transcribers ignored. The baptismal records for the first two or three children might also reveal something, although I suspect that, as you mention, the only record will be of his occupation as a wool comber.