'Old duffer' implies an element of direct military action in the old man's (and it is man, not person) past that old buffer lacks - see meanings around 'duff up'. I'd have no problem with the former being applied to a retired non-comissioned officer in any service. 'Old buffer' carries (for me) an implication of past uniform or office without the associated element of action - to buff is to polish so perhaps batman or steward? Either implies a certain lack of brains and initiative coupled with rigidity of personal habits based on past military service, and neither quite works with 'Lord X' imho.