Hi Joel,
The reason I asked about the childrens names was because the Scots quite often but not always used their father and mothers names for their children. I believe the Irish may have followed this tradition too.
http://www.scotlandspeople.gov.uk/content/help/index.aspx?561 This link explains all
I also note James has a middle initial A. This could be a name he picked up and started using as an adult so don't be put off a potential James in Scotland just because he has no middle name.
Unfortunately, if James did go to Ireland prior to crossing the Atlantic it's highly unlikely you will find evidence. Boats travelled back and forth between Scotland and Ireland without passenger lists being kept.
There are a few potential James McIlwraiths (the McElrath spelling must have came about in US) in Scotland - 2 Ayrshire born boys and one Renfrewshire born - but from the quick look I've had they seem to be still appearing on 1861 and 1871 Census - so they can't be your man. Bang goes my living in Glasgow theory, as the Ayrshire boy in Glasgow appears on 1861 Census.
However there is one lad, slghtly older than your ancestor who disappears of the Scottish Census after 1851. He could of course be dead or have moved to England, but he was living in the ideal location to nip over to Ireland.
This boy is James McIlwraith
https://familysearch.org/ark:/61903/1:1:XYMG-XJ2 born to Gilbert McIlwraith and Martha Murray at Minnigaff Kirkcudbrightshire on 16 Sep 1832
He is living in Minnigaff with his family in 1841.
By 1851 the family is in Penninghame Wigtownshire approx 25 miles from Stranraer - the main port for boats crossing from Scotland to Ireland.
They are farmers and they are living at Glenvosie (probably farm name)
MCILWRAITH Gilbert Head M M 42 Farmer 700 Acres Wigtownshire - Penninghame
MCILWRAITH Martha Wife M F 46 Farmer's Wife Kirkcudbrightshire - Minnigaff
MCILWRAITH James Son U M 18 Farm Servant Kirkcudbrightshire - Minnigaff
MCILWRAITH John Son U M 17 Farm Servant Kirkcudbrightshire - Minnigaff
MCILWRAITH Elizabeth Dau U F 14 Farm Servant Kirkcudbrightshire - Minnigaff
MCILWRAITH Martha Dau U F 12 Kirkcudbrightshire - Minnigaff
MCILWRAITH Susan Dau U F 10 Kirkcudbrightshire - Minnigaff
MCILWRAITH J Son - M 6 Kirkcudbrightshire - Minnigaff
MCILWRAITH Peter Son - M 3 Kirkcudbrightshire - Minnigaff
I think Gilbert and Martha remain in this area for the 1861 and 1871 Censuses.
I couldn't find James anywhere.
I liked that your James had a daughter Mattie and wondered if she was named in honour of Martha .
Of course the rest of your ancestors' children do not have names that match this family. And this boy may not be your ancestor. But I thought I would draw him to your attention.
Good luck with your research
Best wishes,
Looby