The fact that this Anthony was a gardener, (now proved) not only he could have moved South to Kimbolton Huntingdonshire after marriage 1769 in Pontefract, but what you don't mention is he had already moved some distance from his home parish of Aston parish (Rotherham ) to Pontefract at some point before marriage. With his trade as a gardener, both moves would fit your theory of working for a wealthy household as a servant and moving about the country with his employers household. (His thought Grandson Charles Handley b 1818 Hatfield son of Benjamin b 1770 Kimbolton Huntingtodonshire the saddler, was also gardener at some point in the census years near Mexborough)
After Anthony's death 1779, Esther and children could have continued being cared for by the estate house hold, son Benjamin starting his working/learning life within the household-?, mother Esther may have been working in the household, infant Ann taken care of in the within the staff ongoings. The other option is she was sent to her home parish of birth or Anthony's home parish of birth for support from the parish overseers or grandparents or remarried.
Boys them days from 10 years onwards followed their education into trade or work! not the mothers apron strings till marriage of their own. Therefore a parting with mother widow Esther and son Benjamin being his work or education into work is logical and daugher Ann Handley b 1778 went with her mother Esther to a place unknown, and seeing Ann Handley b 1778 poping up in another part of the country at a later date of marriage would have no bearing on Benjamin's life or places in his later life.
No parish information in Huntingdonshire after 1779 of these Handley's till Benjamin Handley pops up in Dronfield marriage 1812 parish in Derbyshire, Dronfield Parish boundary line with Aston parish in Yorkshire that abodes in the parish's did overlap. The Rotherham & Doncaster general area Handley's family trades, along with Darwent families were Farmers, Shoemakers. Tanners, Saddlers, Blacksmith/tanner combined trade, Malster /brewer, Innkeepers & Saddlers some combined trades.
Benjamin Handley marriage 1812 Dronfield ( thought son of Anthony Handley b 1730 in Aston Parish) being a saddler by trade would have spent years learning that trade and that is a hard fact !!!!
Its sparse but likely logical with locations & family trades etc