If I have to go to a thing and push a button, I'd rather the button do the thing I wanted in the first place. Voice assistants are for when my hands are full or I don't want to get up. (I wrote more about my home automation philosophy in another comment[1]).
Also I have all my voice assistant devices mounted to the ceiling
What if you have two things? You'd then need two buttons.
The push button is a perfectly viable option, it just needs to be in a form factor that's works. Could be as simple as a tiny low-energy Bluetooth board with a coin battery that will last several months.
Yeah, a touch-sensitive com-badge mounted on your chest would work.
Actually, I would think a small coin-sized button & transmitter that did nothing but emit a signal that your assistant (or phone) interprets as 'start listening' would be pretty useful. In your pocket, on a watch band, etc.
Also I have all my voice assistant devices mounted to the ceiling
1. https://news.ycombinator.com/item?id=47399909