To steal a physical key you need to find a person and make him give you the key (and tell a PIN code if it is protected). It is hard to do on a large scale and you probably get beaten or go to jail soon.

And passwords can be stolen remotely using trojan software. You won't even know whether your password was compromized.

I agree the recovery can be a problem but there probably are ways to solve it.