Yeah, you lose the full-stack benefits of Django pretty quickly once you start swapping out components. If I wanted a more customizable Python stack I'd pick one that was designed to work that way from the beginning.