In the Scottish lowlands, you had agricultural labourers working for one farmer or "free-lance" ones working for whoever required them (often just listed as "labourers"). It was quite common for a married farm worker eventually to rent a small croft of land. Either they worked that exclusively or, very commonly, worked at a local farm, coming home every day, or often staying at the farm and coming home on their day off. The lease might not be very long. These folk are sometimes called "crofters", sometimes "farmers", sometimes just "labourers" in censuses. Some enumerators will call a man with two acres a farmer, others will call someone with fifteen acres a crofter. There is no great consistency.
If your man is in Scotland, you can check the valuation rolls from 1855 onwards and see the name of the landlord and what his annual valued rent is.