I don't get the slavery connection from "overseer", either. That's still a common term, and not closely tied to slavery. It just means someone who oversees, a manager/supervisor/foreman. Dictionaries (the ones I checked at least) don't even mention slavery, though I did notice a lot of slavery-related "related searches" on dictionary.com.

Obviously what you see is what you see, so you're not wrong, but I think the connection isn't something most people would see.