If we’re to retire in comfort, we need to be deadly serious about saving money for perhaps three decades. That leaves a little wiggle room: If our careers span four decades, we might have a decade or more when we can be a little less focused on making and saving money.

The question is, when should this “goof off” period be? Conventional wisdom has its answer: We should pursue our passions in our 20s, traveling the world and dabbling in this and that, before we’re burdened by mortgage payments and young mouths to feed.

I think this is nonsense.

From the reactions I’ve received, I realize my opinion is an unpopular one. For some reason, we—as a society—believe youth ought to run wild, but middle-aged folks should be content to grind it out day after day in their cubicle. Here’s why I think conventional wisdom needs to be turned on its head:

1. In our 20s, we’re certain we know what we want—and we’re often wrong. We pursue careers, only to find we hate our chosen profession. We spend money on all manner of things, only to grow quickly dissatisfied and perhaps even regret our purchases.

But time helps: As the years pass, we come to understand ourselves better. What to do? Maybe we should spend our 20s and 30s making and saving money, while we figure out what we really want from our lives.

2. Pursuing our passions grows more important as we age. In our 20s, the work world is novel and exciting. We’re anxious to learn the rules, prove our worth, and collect promotions and pay raises.

None of this seems nearly so exciting in our 40s and 50s. By that point, we have had plenty of career successes and pay raises—and discovered they deliver only fleeting happiness. Instead, we hanker to spend our days doing work we love, rather than the work our bosses demand. To use the psychological lingo, we become less extrinsically motivated by the carrots and sticks of the work world, and more intrinsically motivated by what’s important to us.

Read: How to live out your childhood dreams in reality

3. If we save when we’re young, our entire financial life will be easier. Let’s say we want $1 million at age 65 and our investments earn 4 percentage points a year more than inflation.

If we listen to conventional wisdom, we might pursue our passions for the first 10 years out of college. Finally, we settle down at age 32 and save in earnest for the next 33 years. If we sock away $1,215 every month, we’d have $1 million at age 65.

What if we didn’t goof off for those first 10 years? Consider an alternative scenario: We sock away $821 a month for 33 years, from age 22 to 55, and then stop saving and simply leave the money to grow for the final 10 years before retirement. Those final 10 years could be our “goof off” years, when we pursue careers that maybe aren’t so lucrative, but which we find more fulfilling. Result: We still have our $1 million at age 65 and we still save for just 33 years—but our required monthly savings is 32% lower.

Starting to save early doesn’t just mean more help from investment gains. It also means a lifetime of less stress. By starting young, we quickly get ourselves in good financial shape—and we’re spared the money worries that dog so many Americans throughout their lives.

But what about living in the moment? As always, the greatest temptation is to make our current self happy. But what about our future self? We will almost certainly become that person: If we’re age 20, there’s a 90% chance that we’ll live to age 60, according to the National Center for Health Statistics.

Put it all together, and I’d argue it makes more sense to pursue our passions in our 50s than our 20s. There are obvious exceptions: If you have athletic talent, it wouldn’t be wise to delay your NBA tryout until age 55. But for those of us who aren’t 7 feet tall (or, for that matter, 6 feet), saving first and pursuing passions later makes a ton of sense. My advice: Spend your 20s and 30s getting yourself in great financial shape—and one day your current self will be filled with gratitude for your old self.

This column first appeared on HumbleDollar.com. It has been reprinted with permission.