It's definitely worth tracing James and the other children through all possible census returns, to see if they ever give a more detailed place of birth. The later returns often show better detail..
Also worth checking on any Higgins or other Mayo relatives sharing the house of your family, or living nearby, and following them if possible through English census and civil records.
Sometimes RC marriages can mention more details.. possibly even a note of a letter of freedom from their parish of baptism...
Shane