Engineer to Director: How to skyrocket your Machine Learning career in 6 months [Guest]

How a Machine Learning Engineer got himself promoted to director leveraging 6 months of work.

Hey, it’s Devansh 👋👋

Our chocolate milk cult has a lot of experts and prominent figures. In the series Guests, I will invite these experts to come in and share their insights on various topics that they have studied/worked on. If you or someone you know has interesting ideas in Tech, AI, or any other fields, I would love to have you come on here and share your knowledge.

If you’d like to support my writing, please consider buying and rating my 1 Dollar Ebook on Amazon or becoming a premium subscriber to my sister publication Tech Made Simple using the button below.

p.s. you can learn more about the paid plan here. If your company is looking for software/tech consulting- my company is open to helping more clients. We help with everything- from staffing to consulting, all the way to end to website/application development. Message me using LinkedIn, by replying to this email, or on the social media links at the end of the article to discuss your needs and see if you’d be a good match.

---

While Jack Blandin is a newcomer to the AI Content Creation space on LinkedIn, I find his posts to be insightful, genuine, and approachable. Jack has many great posts regarding career progression, conducting better interviews in AI, and how Engineers can upskill in AI.

After stalking his page for a bit, and having a brief conversation to get to know each other better, I knew that I wanted to have Jack share his knowledge with y’all. In the upcoming article, Jack will share his experience with a very harrowing but ultimately rewarding project. This project enabled Jack to go from a Machine Learning Engineer to the Director of Data Science at his former organization. I’m sure you will really appreciate all that Jack has to share, I know I did.

In addition to fueling growth of the company, this project also significantly accelerated my own career advancement. Senior leadership saw the results that machine learning could have when allocated properly, and so they decided to integrate it into other areas of the business. And because they’d seen me lead the earlier ML development, they naturally saw me as the person best fit to lead those new initiatives. I found myself promoted to Manager by the summer of 2019. And one year after that, after my team successfully launched other ML initiatives, I was promoted to Director.

-What you have to look forward to.

PS: I came across Jack through a fellow cult member. If there’s someone who you like, don’t be shy about making the introduction. I don’t care how many followers they have, about their job title, etc. As long as they have interesting things to say, I would love to feature them.

---

About me

Hey all, my name is Jack Blandin. I’m currently the VP of Data Science at Fi––a NY-based startup that makes smart dog collars. Previously I’ve worked as an Associate Director of Machine Learning at Wayfair, and various other roles at Trunk Club and GoHealth.

Until relatively recently, I have avoided any kind of content production. But lately I’ve decided to change that, and share insights I’ve learned over my nine-year career in Machine Learning, Data Science, and Software Engineering.

My content is a bit different. I don’t focus purely on technical topics. Instead, I emphasize what I call the “softer side” of tech, and how these topics interweave with technology. The general message I try to preach is that being successful with ML, or any other tech, is not purely about technical prowess. That is just one piece of the puzzle, and there are many others that are often overlooked. I like to tell stories from my past that led me to this line of thinking. And to make these stories engaging for readers, I tell them with very little filtering.

Why I'm telling this story

When I speak with candidates when I am interviewing for open roles on my team, one of the most common reasons they cite for their search is that they feel like they have hit a career plateau in their current role. But when I dig into why they feel that way, more often than not I realize that they are just not aware of what’s within their control.

Early in my career, I thought that my advancement was entirely dependent on my manager. I believed that if I did a good job and got good performance reviews, then my manager would automatically recommend me for promotions. However, as I have progressed in my career, I have come to realize that this is not the case. Advancement is not a one-way street. It requires the support of many people, including your manager, your peers, and your senior leaders.

In this article, I’ll share a story about a machine learning project that helped advance my career, ultimately leading to two major promotions within two years. But the story I tell is not the picture-perfect story about the technical novelty of the algorithm, or the fancy new framework we used. Instead, I tell the whole story. I talk about both the good, and the bad. I don’t hold back. In this article you’ll see what’s under the hood of a successful ML product launch.

Without further ado, let’s get into one of those stories.

Development

In the summer of 2018, I was hired as a Senior Machine Learning Engineer at GoHealth. The company operated a call center where people would call in and GoHealth’s sales agents would help them enroll in health insurance. The call center frequently experienced times of high volume where there were not enough agents available to answer them. I was tasked with deploying a machine learning model to prioritize incoming leads so that agents could focus on the most important ones.

The model was called the “Lead Scoring Model”. And while I can’t give too many specifics since the model is proprietary, I can share that the model was a relatively simple binary classifier that predicted the probability that an incoming lead would enroll in a new insurance plan. The model’s output would be used to prioritize the incoming call queue, so that calls made by leads with high intent would be prioritized first. While the model was relatively simple, the deployment constraints and development process were anything but.

When I was assigned the project, the model existed as a collection of Jupyter Notebooks. The best way I can describe them is that it looked like someone vomited a bunch of SQL and Python. Very little documentation. Queries with very complicated joins without clear naming conventions. The feature transformations didn’t use any pipeline objects (e.g. pd.get_dummies() was used to onehot-encode the categorical inputs), and the excessive grid search space led to the process taking 6 hours to train. To make matters worse, the model was using features that were not available at prediction time. As a result, I ended up having to rewrite most of the model training pipeline from scratch.

Photo by Tim Gouw on Unsplash

The project was further complicated by a hard requirement from the Engineering organization that the model be deployed in a Java Spring application. For those not familiar, Spring is essentially the Java web application equivalent of Python’s Flask/Django. This meant that the web application could not read in a pickled Python object, so I’d have to find a way to either (a) find a Java library that could build a machine learning model, or (b) find a way to convert the Python pickled model into something that the Spring application could interpret. Knowing that eventually I’d try and move this entire application into an AWS-hosted Flask application, I opted for option (b) so that I could keep the model development process in Python. My manager and I found a library that converted a pickled object into something called a PMML, which is a markup language similar to XML that Java knows how to interpret.

If you’re thinking – this sounds like a recipe for bugs — you’re right. It was probably the most buggy software I’ve ever built. To understand why, imagine trying to represent a fitted random forest or a neural network as an XML object. It works well when the model is extremely simple, but starts to break down if any advanced feature engineering or model adjustments are implemented.

While this strategy was suboptimal, it was the quickest path to getting a working model in production. And given that we were short on time, this was a tradeoff we had to make.

And while the technical challenges created a fairly stressful environment, they were nothing compared to the non-technical challenges.

Photo by Justus Menke on Unsplash

First was the pressure created from the significance of this project. The output of this model would have a major impact on the business because it decided how we would prioritize incoming leads, so any modification or mistake here had a significant impact on the business’s top line revenue.

As a result, all eyes were on this project—I had people checking in on the model’s progress from many aspects of the business. Also, this was the first major piece of software produced by the newly formed data team. So if this didn’t go well, it wouldn’t bode well for us.

There was also the challenge of having a fixed, and very tight, deadline. This was because we needed a software freeze several weeks prior to the start of the Annual Enrollment Period (AEP), which is when GoHealth conducts most of its business. So I had to deploy the model and ensure everything worked properly by August 15th, which was just 6 weeks away.

Easy right? Hardly. I spent the next six weeks glued to my computer rebuilding the model training environment, setting up the deployment pipeline, harassing Data Engineers to set up the necessary ETL pipelines, and haggling with the DevOps team to grant me the necessary permissions needed to deploy the software. My one saving grace was that I had a very capable and hardworking manager who made themselves available to help with the integration debugging, which was the part of the process I was the least familiar with.

But with literally hours to spare, I finished the deployment in time, and all integration tests passed.

Launch

Despite all of the challenges, my hard work and technical preparation led to a very smooth rollout of the model. Everyone applauded me for all my hard work, and I was cheered during our company meetings.

Sound too good to be true? You’re right. In reality, total chaos erupted.

Photo by Ricardo Viana on Unsplash

As soon as I deployed the model I had to start playing whack-a-mole with bugs that popped up. I was on-call 24–7 since I was the only one that knew the full deployment stack. And so I was frequently woken up in the middle of the night when some piece of software decided to stop working. For instance, I remember the Java app frequently running out of RAM due to the massive PMML object, as well as a small memory leak (that I discovered later on).

But technical bugs were actually the least of my concerns. The bigger issue was that people started freaking out about whether the model was making the right decisions. Or, more accurately, why the model wasn’t making the decisions they wanted it to make. Let me explain.

Like any successful sales business, GoHealth had results-oriented leaders that worked hard to generate strong lead demand. So, inevitably, there were some people in the business who’s personal interests conflicted with the objective of the model. For instance, the model started to “learn” that certain marketing campaigns tended to have higher conversion rate than others, and so it prioritized leads from those higher conversion campaigns. This would upset people who managed the lower conversion campaigns because the leads they generated never made their way to sales agents and didn’t result in sales. To put it simply, the model created tension, and much of that tension was unsurprisingly directed at me since I was responsible for the model.

Between the 6 weeks of intense development, and then the emotional roller coaster of technical debugging and stakeholder tension, I was mentally and emotionally drained. I remember thinking–––what have I gotten myself into.

Impact

But I’m happy to say that it was worth it–––no joke. I consider myself very lucky to have been given that opportunity. Here’s why.

During all of the commotion in dealing with the model, what was not yet apparent to me was that all of the chaos came about as a direct result of the model’s tremendous success. This wasn’t a minor improvement. It was a HUGE improvement. Conversion rate increased by as much as 35% during times of high lead volume. This had a direct impact on top-line revenue.

The Lead Scoring Model wasn’t just a win, it was a game-changer. It differentiated GoHealth from its competitors, and so it became the focal point of senior leadership as they strategized more ways to leverage the model. And because I was the person who best understood the model, I started getting asked to join discussions with leadership to field questions that popped up. By the end of that year, I was regularly included in meetings with many of the senior leaders at the company.

In early 2019 when the busy season had ended, the focus shifted to developing new applications for the upcoming year. Many improvements and expansions were requested to expand the influence of the Lead Scoring model. We created variations of the model to better capture nuanced aspects of the business. We collected more data so that we could add more informative features. We also started building other intent-based models in other parts of the business.

In addition to fueling growth of the company, this project also significantly accelerated my own career advancement. Senior leadership saw the results that machine learning could have when allocated properly, and so they decided to integrate it into other areas of the business. And because they’d seen me lead the earlier ML development, they naturally saw me as the person best fit to lead those new initiatives. I found myself promoted to Manager by the summer of 2019. And one year after that, after my team successfully launched other ML initiatives, I was promoted to Director.

Reflection and Insights

Looking back on this time, I can now articulate the specific aspects of that 6 month period that had the biggest impact on my career advancement. Let me walk through a few of them.

Hitting deadlines

None of the Lead Scoring Model success would have occurred if I hadn’t built the model in time. It was extremely stressful, but that stress paid out 10-fold after the fact.

Don’t get me wrong–––stress, deadlines, working late, all of that totally sucks. I set boundaries between my work and personal life and don’t work overtime unless absolutely necessary.

But even now, after another 5 years of experience, there are still times where I’ll buckle down and do the dirty work to get something done in time. I do this because I understand the importance of having a reputation for getting things done and not making excuses. Having that reputation gives me leeway in doing things my own way, including prioritizing and planning my team’s development process. At GoHealth, for instance, I was given the autonomy to rebuild the model deployment stack using the framework of my choosing, rather than using the Java Spring framework. (I opted for AWS Sagemaker + MLFlow).

Gaining exposure by becoming the point-person on important projects

My advancement was made possible because I had the support of leadership across the business. They’d witnessed me first-hand demonstrating my expertise with building production machine learning models and their application within the business. So when the company wanted to invest in more machine learning projects and hire more MLEs and Data Scientists, it was not a hard sell to put me at the helm.

My exposure to leadership came about because I was the “point person” on a major project. My expertise on both the technical and business side of the application made it so that I was very valuable to have in strategic discussions that involved the lead scoring model, which often involved senior leadership. So I didn’t have to go out of my way to engage with leadership. Instead, I was simply invited.

Here’s a surefire way to become the point person on a project: become an expert on the non-technical aspects. In other words, learn about the business-facing side of things. In most cases, the technical team will only have a high-level understanding of the business requirements. But if you understand both the technical and non-technical aspects, you will be in a unique position to contribute to planning discussions. And who attends those discussions? Senior leadership.

Demonstrating clear, quantitative impact

Nothing speaks louder than numerical results. To this day, I always ensure to bake this into the development process. It’s the hardest thing to argue against and it’s the easiest thing to leverage in promotion discussions. You can be the best programmer at the company, or the hardest working, but if that’s not leading to results then none of it matters.

How do you demonstrate value? A/B tests are your safest bet. For the Lead Scoring Model, for instance, I developed an A/B test strategy that would show the conversion rate of the ranking system with and without the existence of the model. We built a Tableau dashboard that showed the impact on conversion rate throughout each day, and how it would spike during times of high inbound lead queues.

Selectively embracing pressure

The months of June-December of 2018 were the most stressful months of my professional career. At times, the pressure seemed insurmountable. But, as the saying goes, pressure makes diamonds. The end product was a huge success and much of that success was a result of the extraordinary pressure applied.

I do not recommend seeking out high-pressure environments, but if you can strategically place yourself in the right ones, you may find yourself in a position of great importance. This is because pressure often comes with responsibility, and important projects often have the biggest impact.

Wrapping up

If there’s anything I hope you took away from this story, it’s these three things.

First, you have more control over your career advancement than you might think. And while in this story I was lucky enough to be given a high-impact project, there are always ways to seek them out.

Second, the most successful tech projects are guaranteed to have an uglier side that only the core developers know about. So don’t feel like you’re the only one dealing with tech debt or inserting the occasional hack to get something out the door.

Third, being a successful Data Scientist, Machine Learning Engineer, Data Engineer, or any developer role is not purely about understanding technology. The softer skills are just as important in order to succeed.

—

Thanks for reading. If you liked this story and want to hear me talk about this type of stuff, follow me on LinkedIn.

---

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. If you like my writing, I would really appreciate an anonymous testimonial. You can drop it here. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

Check out my other articles on Medium. : https://rb.gy/zn1aiu

My YouTube: https://rb.gy/88iwdd

Reach out to me on LinkedIn. Let’s connect: https://rb.gy/m5ok2y

My Instagram: https://rb.gy/gmvuy9

My Twitter: https://twitter.com/Machine01776819