The existence and popularity of sites like gasbuddy.com suggests that some number of people are willing to go out of their way to find the lowest price. I personally know people who will make a run to Costco for gas, even if they're not going to go in for groceries, because the price is good. There are at least a half dozen gas stations on the way to Costco.
Half the people driving out of their way to get gas are bad at math. The other half simply don't assign a dollar value to their time.
If your car gets 30 mpg, has a 16 gallon tank (that you refill at 1/4 tank, so you're buying 12 gallons), and you drive an extra 5 miles to pay $3.93/gallon instead of $4.00/gallon, how much did you really save?
I'll give you a hint: It's less than a nickel.
Meanwhile, you've probably driven at least 10 minutes that you didn't need to drive. 10 minutes to save a few pennies.
The math only gets worse as the gas prices go up and your fuel economy goes down. You need a greater delta to make the drive worth it.