I wonder whether in fact he grew up in the Pimlico/Westminster area, thinking (mistakenly) that he had been born there.
In 1871 we see this family among many at 6 Carteret St, St Margaret Westminster:
James Hardy Head Mar 41 Carpenter unemployed Southwark
Jane E do Wife Mar 41 no occupation Knightsbridge
Mary E do Daur 7 Scholar Westminster
James do Son 5 Scholar Lower Sydenham
RG10/127/55/9
Likely birth registrations for the children:
Mary Emma Hardy, mmn Sutcliffe, Sep qtr 1863 Westminster
James Hardy, mmn Sutcliffe, Jun qtr 1866 Lewisham