To immediately address the question of the Taurus vs. the VW and gasoline polluting more per mile driven.
Right now, that is not the case, as that Taurus meets SULEV standards and the VW is only required to meet ULEV numbers. So, while the Ford burns 2.5x the amount of fuel, the VW emits:
5.5x the amount of NMOG (non-methane organic gasses)
2.1x the amount of CO
3.5x the amount of NOx
I agree with your reasoning though - and applaud the thought - but your example ends up being a numbers game for emission standards. Soon, diesels will need to be SULEV emission capable, and you will see a corresponding drop in fuel economy. It will still be higher than most gasoline powered cars, but the gap will decrease enough that you might think twice about buying a diesel.
Diesel fuel in the U.S. is the same low sulphur content as in Europe, our cetane is a bit lower - but ULSD is now the only legal diesel fuel for use in licensed motor vehicles.
Why fuel economy sucks these days:
1. You, the consumer:
The tempo mentioned before:
52 Horsepower. The 90-hp 2.3L gas engined car went from 0-60 in a blistering 13.0 seconds. I can't find data on the diesel, but if it is like my old Jetta diesel, it was probably close to 16 seconds.
Cars today - even econoboxes - get shit upon if they don't reach 60 mph in under 10 seconds. Higher power is needed for that, so fuel economy goes down.
Tempo: curb weight of roughly 2500 lb.
Chad's Fuckus: 3000 lb.
Just under 250 kg of extra mass that needs to be accelerated and decelerated. The Fukus is also capable of 8.0 seconds 0-60 and has heavier brakes to slow the additional mass - and in much a shorter stopping distance than the Tempo.
Everyone wants a 5-star crash rating, the government has imposed airbags to protect the stupid, . Fact is, we all want to have the best chance of walking away from an accident or want to protect our family the best we can. All of that adds mass. More energy is required to get the extra shit moving.
Try to find a vehicle w/o air conditioning, power door locks, power steering, power buttplug, etc - it's hard. I'd imagine less than 10% of the market is equipped with manual windows / door locks / whatever.
Luxury items from 20 years ago are standard on today's cars.
Sound deadening, 20-speaker infotainment systems, et al all add mass to the vehicle AND increase the parasitic loads on the engine by increasing the electrical demand on the alternator, adding a power steering pump and an A/C compressor.
My old man had one of the first Tempos with a 2.3L M/T no air, no power and an am muthafuckin radio and was able to get 42 MPG highway in that thing.
Oh, I almost forgot - That Tempo was sold when the national speed limit was 55mph and now you assholes demand 40mpg driving 75mph. Wind resistance has that fukin' exponent on the velocity term, i.e. it takes over twice the amount of energy to push a car through the air at 80 mph than it does 55 mph.
2. The government & its silly little regulations.
Emissions standards and all the "trouble" they create are actually quite a good thing. Emission control systems operate near 100% efficiency - and are not too restrictive (gasoline systems). Their drawback is added cost for more componentry and the significant contribution to the vehicle's weight.
Diesel particulate filters are a bit restrictive, but the overall downside of those (including added mass of the aftertreatment system) is that you use "non-torque producing" fuel to burn off / clean the filter at regular intervals.
The reason why having emission standards is a good think is once you scale up pollutants coming from a manufacturer's fleet. That's a huge number of vehicles and the output has to be low, as we're trying to offset all the old vehicles operating with old, broken or deleted systems - AND trying to make the systems we equip today last 150,000 miles. (Yep - that increases your vehicle's price too)
Passive safety and 4,6,10,20-airbag systems that come with 5-star crash ratings, emission systems and all the "features" the customer "has to have" in a vehicle increases mass. Mass goes up, fuel economy goes down.
Customers will not settle for a 15-second 0-60 automobile, so bigger/more "featured" powertrains also increase mass.
It is hard to believe, but the vehicle mass has the most impact on today's fuel economy numbers. There is no conspiracy. Seriously, WTF!? First people say the Volt was the government forcing GM to build a fuel-efficient car, and now shit's swung full opposite scale.
So, you want to go fast, be safe and have all the luxuries of a Cadillac AND get 50 mpg?
There's no conspiracy, just ignorance.