You also have: - People jaywalking without looking at the street - Bicycles running stops and red lights - Dogs running onto the street - People leaving stuff on the road - Trees falling on the street - People standing in the middle of the road and talking - Kids running on the street after the ball - Things falling of other cars - Bicycles driving at night without lights - Deep potholes

World is unpredictable, and city environments are full of surprises. Machine needs to be aware of rules and follow up, but also be able to deal with rules being broken by other users. General rule "approach with caution and be ready to react" doesn't work that well with machines - problem is very very tricky. There's a reason why Waymo had self driving cars for 10 years already, but are still in very limited beta.