I think the idea of paternalism comes from seeing people harm themselves by undertaking actions they were manipulated into taking, and confusing that with actions they'd have done on their own, independently.