Now, I'm not saying the technology didn't exist, since it did, but making a bigger engine with crappy parts to keep the emissions down was a cheaper option than engineering a highly efficient smaller engine.
Not to mention the buying public didn't want small engines. They weren't popular back then. This was Merica - land of the free and home of the V8 and we're better than everybody else because our cars thunder, float, and get 6 mpg because we can. That was the general post-WW2 attitude. Thus, big V8s.
But now, we live in the era of the eco-weenie, where everybody thinks we small humans have a greater role in the environment's condition than, say, the sun. Most people, even us arrogant and ignorant Americans, are at least somewhat environmentally
weeniefied conscious and so they're willing to cut back for that reason. Also, gas isn't hilariously cheap anymore. That's probably the biggest reason for trending toward more efficient engines.
But honestly, people and car companies these days must not be too worried about efficiency. While cars average much better mileage numbers, they do so with a lot more power than they did back in the day. And I firmly believe that the average weight of vehicles on the road - especially here in the States - has gone up considerably. So while today's cars to run cleaner and use less fuel, they also boast way more power, are way faster, and heavier too. Imagine how efficient they would be if they didn't have way more power, weren't way faster, and weren't heavier. It would be Europe, land of the at-least-30-mpg car.
It's not that the car companies can't do this or can't do that, or somebody has better technology or whatever, it's just that they're businesses, and businesses tend to sell products that make them money. The higher the profit margin the better.