The Machine Learning Reason You're Stuck in the Wrong Career
How the path with desirable difficulties could make you irreplaceable.
When you were young, you had all these dreams and goals you wanted to achieve. Football player, astronaut, filmmaker, skier, F1 driver, writer, space cowboy (ik it might just be me).
But as you grew older, most of these dreams started to fade, feel unrealistic and out of reach. Perhaps a few were, but even the ones that weren’t - they’re just gone.
Let’s understand this pattern:
Maybe you opted for a major in school, or life drifted you onto a path. But eventually, you found something to study in school, specialise in, get a job in, and get enough expertise to make a decent career out of.
This is the path the majority of us take, and many of us are in this process right now.
Have dreams and interests → Grow older → Get convinced they’re unrealistic → Find something “attainable” in a college career counselling session → Dedicate your life towards it.
And honestly, this works for some people.
They find out really early what they want to do, and work on it for a better part of their lives. But for a lot of people, they don’t get to choose their route - they stumble onto it.
They might be good at it, making decent money and progressing position-wise, but since they didn’t opt for that path, they feel stranded and caged, like their career has stalled and the work feels forced.
And I say “they didn’t choose” because you only get to choose when you’re aware of the available options. But for the people following this pattern, the options didn’t exist.
So all this happened, and you can see yourself in this pattern.
But is there any realistic way to start working towards the dreams we once had right now, or is the headstart advantage gone, and should we stick to the current route or else we risk losing that too?
The Overfitting
Most of the time, we tend to cling to something and refuse to experiment for two reasons: unfamiliarity and the fear of looking bad.
When you’re familiar with one path, you try to stay on it instead of trying out different unfamiliar routes.
This mindset settles in from a really early stage. Since school, we’re told to opt for one of the available options.
Science, Arts, or Commerce? Pick one. Opting for a major in college? Pick one. Which job position suits you? Pick one.
And you try to become proficient in the only path you were exposed to.
The majority of these decisions we make really early on, and people expect some 18-year-olds to make perfect decisions and stand by them, which leads to that caged feeling later in their careers, even if they’re apparently doing well.
We have a concept similar to this in Machine Learning (ML) called Overfitting.
In overfitting, an ML model is trained on a particular dataset. For instance, a model is trained to differentiate between a cat and a dog.
The machine performs outstandingly in training, sometimes even scoring 100% accuracy. It might seem like a great model to you, but there’s a catch.
Let’s say someone provides the model with a different dog breed than the ones provided while training - since the model wasn’t trained on that breed, it will break, and that 100% accuracy will mean nothing in the real testing environment.
That is overfitting.
Initially, the specialised model would perform better than a model trained on a wide dataset, but in a practical environment, a more diverse model would always perform better, even if its accuracy is 60%.
Now connect this to our subject.
A person who abandons all their different curiosities to go in a direction that they didn’t choose and becomes proficient might score better early on.
But if a person explores multiple interests, chooses a few, and aligns them with the direction they want to take, they may seem to be failing initially, but in a practical environment they will perform better, even if they’re not 100% in a domain.
In the book Range, David Epstein explained this through the context of “Match Quality”.
‘Match quality’ is defined by economists as the degree of fit between the work someone does and who they are (their abilities and their interests).
Epstein mentioned Ofer Malamud’s work on match quality:
“Malamud studied English and Scottish students. English students had to specialise before college so that they could apply to specific, narrow programs.
In Scotland, however, students were actually required to study different fields for their first two years of college, and could keep sampling beyond that.
Malamud analysed data for thousands of former students and found that college graduates in England and Wales were consistently more likely to leap entirely out of their career fields than their later-specialising Scottish peers.
And despite starting out behind in income because they had fewer specific skills, the Scots quickly caught up.
With less sampling opportunity, more English students headed down a narrow path before figuring out if it was a good one.”
The overfitting doesn’t allow you to find your match quality because you wouldn’t have enough samples to play around with.
The same can be mapped with the fear of perceived failure.
The Desirable Difficulty
When people are really good at something, they try not to risk that reputation by doing something they could fail at, even if that short-term failure could lead to long-term growth.
The reason is simple:
When you’re consistently praised for something, and then you try something else and look below average at it, you consider it a failure, even though this was how you were supposed to look.
There’s a study by Robert Bjork and Nate Kornell targeting this.
Kornell found that strategies with the strongest long-term retention often feel harder, produce more mistakes during practice, and lower confidence during learning.
Yet they improve retention, transfer, and long-term performance. Bjork calls this Desirable Difficulty.
A desirable difficulty is a learning challenge that initially makes learning feel harder, but it forces deeper processing and creates stronger memory and understanding later.
Simply put, looking bad initially is the price you pay for growth, and not everyone is rich enough to pay it.
I caught Desirable Difficulty in action. Once you see the pattern, you’ll be able to identify it in real time as well.
I don’t know about you, but whenever I need to do something I’m not familiar with, my mind always tells me, “It’s not that important.” “I don’t like doing it.”
Even if, in the back of my mind, I know this could lead to growth.
I understand this pattern now, but for a long time, I believed in it. Until recently, when I tried something unfamiliar and realised that I actually like doing it.
Then I understood that it was my brain’s safety mechanism protecting me so that I don’t jeopardise my reputation as someone “smart & competent” and fail at something.
But now I’ve established that 100% doesn’t always mean success, and 60% doesn’t represent failure.
This realisation always leads me to this proverb:
“How big would you dream if you knew you couldn’t fail?”
Get Messy
We’re creatures of habit.
We prefer predictability, clarity, and defined next steps. We avoid taking a different route home, switching our perfume brand, or brushing our teeth with a different hand.
This is ingrained in us, but it is a very mechanical way of thinking about your career.
Career progression is rarely predictable, especially in the current times of uncertainty.
We need to get messy ourselves and explore beyond the path we’ve stumbled on to find our match quality.
Even if that means looking bad in public, starting slower, sampling through various unrelated things, or not being 100%.
The goal is to avoid overfitting because in the real world, you’d never be asked to identify the same dog as you learned in training.
P.S.: I got the idea of exploring this subject from the book Range by David Epstein (highly recommended).
This is the longest, probably the only, break I took from posting on Substack. But I think I’m at a stage where I want to focus on the quality of the posts and allocate my time better after posting every week for over a year.
Will stay connected. 👋