Your suggestion that Drury relocated to West Walton is a good one but unfortunately becomes less likely as Ann nee FILLIHOE almost certainly outlived him.
The Sparham PR gives Anne Layht/Liyht/Leight, widow, d. 1825 age 92.
Born 1733 (Sparham) was Ann PHILIPPO, dau Robert and Mary - the family name appears well established across Norfolk, but a tall order for the transcriber.
One other detail:
Settlement certificate: Late, Drury, his wife Ann and child Mary, Feb 1757. Certifying parish: Great Witchingham. Parish of residence: Sparham. There's also a 1755 removal order from Sparham to Witchingham for John Drury and wife Ann - could be relevant?
Afraid this doesn't take you much further forward in explaining what happened to Drury after 1757, but helps with context. There's no mention of Ann dying a pauper.
Of course Drury could have abandoned the family in Sparham and started a new one with Alice in West Walton - Ann would still be described as a widow. But there's certainly room to explore your and mowsehowse suggestions that the Drury Layt/Leights in West Walton/Cambridgeshire were cousins - did James d. 1675 have brothers? They must be related in some way and born 1711-1729 suggests the same generation.
Helen