You couldn't control everything the chromecast can do with simple controls, such as selecting which content to play.

That said you could pause, rewind, fast forward, and unpause, I suppose. But you'd still need some other device to select content.