Just came across your post - John William(son) and Caroline Mitchell are my ancestors too, through their daughter Mary who married William Oughton in 1802 in Burnham Thorpe. I haven't been able to get back any further either :-)
You might not have seen these mentions:
When Lord Nelson dug a pond, it was in the shape of HMS Victory, the ship on which he was to die in 1805 at the Battle of Trafalgar. Along with a gardener called Williamson, Nelson designed and constructed the ship-shape pond in the gardens of the Norfolk rectory where he was born in 1758.
https://www.theguardian.com/money/2003/jun/29/observercashsection.theobserver7A memo in the Church register, written in 1907 by the Rev. Hon. J. Horatio Nelson, who at eighty-two years of age came to preach in the Church, tells how, when he was Rector of Belaugh from 1857 to 1872 he was repeatedly told, by an aged parishioner named Williamson, this story:
That he, Williamson, when a boy, used to clean the knives and boots at Burnham Thorpe Rectory, and that he also "helped dam the rivulet ... sufficiently to fill up a small portion of ground which Lord Nelson had made into the size and shape of a man of war so as its possible to float a model ship in it".
http://www.nelson-society.com/html/Five_Years_On_The_Beach.html