Hi,
I have been doing a bit of digging for you, in 1901, there were only three houses in stump of the castle. my house was built around that time so is not listed. None of the families that lived there were Kavanagh's or Doyle's. However, in 1911, there was 7 houses, with the 4th (next to mine) having a family name of Doyle. All these houses would have been labourers houses beside a forest. most, except 1 have since been renovated. Doyle is a common name in the area, but there is every reason that they would have been related. Good luck with the search