Eventually, sure, but I have crappy old fridge magnets that are still going strong after 30 years. What's different about little neodymium rods that they fail in 5?
I think it has more to do with that your body builds up scar tissue around the magnet that makes its wiggles inside your body less palpable, than that the magnet itself weakens.