I wish I could give you just one source to confirm your question.
There is a lot of evidence which points towards John Tilley married to Elizabeth Fraser being the factor on Bunce Island.
They married on 01 February 1804 at St. Mary at Hill Church, Billingsgate, London in the presence of John Anderson, Alexander Anderson, George Wilson and Charlotte Ednistoun (or similar).
John Anderson, nephew of Richard Oswald who originally owned Bunce Island was married to Susanna Fraser, sister of Elizabeth Fraser. John and Susanna were also the grandparents of the third wife of Sir John Tilley K.C.B., Susannah Anderson Montgomerie.
Alexander Anderson, nephew of Richard Oswald was married to Christian Oswald, grand neice of Richard Oswald.
The third Anderson brother and nephew of Richard Oswald, James, lived on a plantation in Georgetown, South Carolina named Ramse(a)y Grove, presumably after the wife of Richard Oswald.His son was named Richard Oswald Anderson who died in Bath in 1852 while visiting his sister, Elizabeth Webb Browne nee Anderson. Who married in 1810 in St. Mary at Hill Church, Billinsgate, London in the presence of Alexander Anderson, Richard Oswald Anderson, George Wilson and Alexander Wilson.
In his book, London to Tokyo, Sir John Tilley, Diplomat made mention of a housekeeping ledger from the counting house of Richard Oswald. This was passed down through the family.
I hope this has made sense to you but as you can see, the same names crop up time and time again.