Noting from the 1851 census that James was 34, a weaver, born Glasgow, I wonder if this is his family in 1841 at Elba, Barony Parish (Glasgow)
James Yule, 55; Robert Yule, 25; Dorothea Yule, 25; James Yule, 20; William Yule, 15; Thomas Yule, 15. All the males are cotton hand loom weavers and Dorothea is a cotton yarn winder, and all were born in Lanarkshire. Adults' ages in the 1841 census were supposed to be rounded down to the nearest 5 years, so James could have been any age from 20 to 24, which fits with his age in 1851 and later censuses.
If this is him, and James Yule aged 55 is his father, then his baptism record has not survived - there are four James Yu(i)les baptised in Glasgow in 1816/1817, none of whose fathers' given names was James.
I see that one James Yuile was baptised on 22 September 1817 (not 1816) and that his parents were William Yuile and Janet Crawford. According to the 1851 census this James Yuille was a tobacco spinner, unmarried, living with his widowed mother and siblings including Rebecca and Sarah, both very uncommon names at that time, in Glasgow. So he cannot be your James Yuile.
Where did you get the birth date of 22 September 1816 for your James Yuile?
In 1851, he's in the parish of Ardrossan: James, 34; Mary, 22; Mary, 2; William, 0.
In 1861, Ardrossan: James, 45; Mary, 12; William, 10; John, 8; David, 6; James, 3; Jane,1.
In 1871, Ardrossan: James, 55; Mary, 42; William, 20; John, 18; David, 16; James, 13; Catherine, 10; Agnes, 7; Robert, 5; Jessie, 2.
You can view the originals of all these censuses at
www.scotlandspeople.gov.uk using the credits you have left over after viewing James' death certificate.
I think Dorothy Yuille, who died in 1880, may be the Dorothea in the 1841 census.