Hi Danny,
It was fairly common for youngsters of 14 to be servants at that time, and indeed much later (My father's elder sisters both went 'into service' at around that age in the 1920s) so it certainly doesn't indicate to me that his father was dead. I think you need to bear in mind that a lot of young men at that time were moving away from terrible poverty in Scotland to seek work or a new life abroad too.
Have you tried looking on Scotlands People? Of course registration didn't start in Scotland until 1855 so you would only be able to find a baptism if one survives. I know exactly where my great grandfather was born at about the same time as your man but there's nothing on record to confirm. I have had a quick look and there appears to be only one record of a baptism for Andrew Ellis between 1840 and 1854. I don't have any credits so didn't look at the record to find out details. The date is 30.10.1844 which is a little earlier than you might expect but ages can vary quite a bit on census records. Having said that it didn't like the suggestion that his father was William. It might be worth you spending a couple of pounds on the site to check it out. Likewise there appears to be only one Andrew Ellis on the 1851 census aged between 2 & 8 years (he's 5) but he doesn't appear to be with a William.
If you have the names of his children this might give you a few clues as to other family members. Most Scots were fairly good at following family naming patterns and if you find a surname as a middle name that will be a clue too.
On the down side the Scots are also extrememy good at using random variants of a name (partly due to the Gaelic/English thing) For example one of my great great grandmothers is recorded on various official documents as Eppy, Elsie, Elspet, Ephemia Euphemia and Phemie.
That probably doesn't exactly clafiry things but might give you some food for thought.
Sheena