Product Bulb

The biggest challenge for data scientists / managers in decision making capacity and one with the biggest consequential outcome for both the business and the data team, I feel, is the part where you say yes / no / let’s modify – to a new data project idea from leadership, or even starting a new project within the data team. 

You see 100s of articles on why X% (looks like 80% is the consensus so far) of data science projects fail. Now this is not an intellectual exercise, right ? There is a real consequence to these projects failing, which is its effect on the runway of the data team. Data Science teams generally have a set budget for the year (apart from people costs), and with that, you need to plan your  infrastructure and operational costs. This could be tools or compute resources or data or training or inference or any kind of data platform and managed machine learning services costs. The foundation to all these costs is going to be the kind of projects you pursue as a team, that’s gonna incur these costs.

Now we can divide data science projects (note: not data engineering, that’s a whole another topic)  into 3 different categories:

1. Core product data science projects
2. Support and Consulting projects
3. Research  projects
4. Core product data science projects

These projects are directly related to your company’s product and hence its revenue. So if you are building Machine Learning or AI services as part of the company’s product, those projects fall under this category. This is probably the most valuable part of the data team from a short term perspective, purely in financial terms. An example could be the Amazon Lookout Metrics which is an ML based anomaly detection service that any AWS customer could avail.

2. Support and Consulting projects

These are projects where a data science team works to support another team, mostly an internal team like sales, marketing or people (HR). These could be projects like Sales Analytics where you predict customer churn or upsell or cross-sell probability, or a Marketing analytics project where you assist in some brand study or campaign effectiveness. So the data team essentially consults for other internal teams.

3. Research projects

These are projects where the data team works on cutting edge problems, that may not necessarily tie to immediate revenue generation, but is an important part of long term roadmap. This is where the company invests generally to stay ahead of the competition and nowadays a way to avoid the FOMO, considering all executive leadership teams wanting to explore how LLMs (Large Language Models) can benefit their company. 

Unfortunately the teams working on bleeding edge research projects are generally among the first ones to be axed when the economy takes a bad turn, as we have been seeing recently, likely followed by support and consulting teams. The core product data teams seem to be a relatively safe bet during downward economic trends.

Things to keep in mind:

For Data Science Individual Contributor: 

If you work at a core product level data science function, and you enjoy working on the intersection of production level software engineering as well as statistical and machine learning research, good for you – keep it going. This role may not give you a lot of opportunities for publishing research papers or conference talks, or at least not as much as a research data scientist. If you are working in the other two areas, i.e. as a consulting data scientist or in deep research, you have got to be more proactive in managing your professional growth. The ideal set up would be where you get experience in a mix of all three, while developing gradual expertise in core product level data science, if you are planning to grow as an individual contributor. The reason I’m saying this is because at the end of the day, a company is going to support you till bring in value in terms of revenue generation. Many data professionals who are super smart and bring in amazing value in terms of moving the research needle forward or showcasing their value to adjacent teams as supporting actors, don’t get recognized (in most average sized companies) when leadership talks about ROI (Return On Investment). Make sure that your data projects are tied to tangible quantifiable business results.

For a Data Leader/ Manager:

As a Data Science Manager, it’s imperative that you proactively manage your team’s (each and every member’s) professional growth by encouraging them to distribute their efforts towards a mix of data projects that makes sure that every individual contributor can showcase their value in measurable terms at every annual review. I have seen  managers who do the project distribution almost in an auto pilot mode where every individual simply carries on working in the same type of projects as the previous one they had worked on, thereby choking their professional growth. If you cannot afford a separate Engineering Manager for overseeing the employees’ professional growth and wellbeing, it should become a job requirement for the data science manager / team leader, in order to build an effective long standing data team.

As a data science leader, deciding which data science projects to pursue requires careful consideration of various factors. Here are some steps you can follow to make informed decisions:

1. Align with Business Goals: Start by understanding the overarching business goals and objectives of your organization. Identify the areas where data science can provide the most value and contribute to achieving those goals. Prioritize projects that align with the strategic direction of the company.
2. Assess Feasibility: Evaluate the feasibility of each project by considering factors such as data availability, data quality, and technical resources. Determine if the necessary data is accessible and if the required infrastructure and expertise are available to execute the project effectively.
3. Potential Impact: Analyze the potential impact of each project. Consider the magnitude of the problem being addressed, the potential benefits to the organization, and the potential return on investment. Projects with high impact and the potential to generate significant value should be given priority.
4. Resource Allocation: Assess the resource requirements of each project, including human resources, budget, and time. Evaluate the availability and allocation of data scientists, analysts, engineers, and other team members. Consider the trade-offs and determine if the resources can be allocated effectively to execute the project successfully.
5. Risk Assessment: Evaluate the risks associated with each project. Consider factors such as technical complexity, data privacy and security concerns, regulatory compliance, and potential ethical implications. Identify potential roadblocks or challenges that could arise during project execution and determine if they can be mitigated.
6. Stakeholder Input: Involve key stakeholders from different departments or business units in the decision-making process. Understand their priorities, pain points, and requirements. Engage in discussions and seek feedback to ensure that the selected projects align with their needs and expectations.
7. Proof of Concept (POC): For projects with higher uncertainty or where the value proposition is not clear, consider starting with a small-scale proof of concept. Conduct a preliminary analysis or experiment to validate the feasibility, potential impact, and value of the project before committing significant resources.
8. Continuous Evaluation: Regularly evaluate the progress and outcomes of ongoing projects. Use metrics and key performance indicators (KPIs) to track the success and impact of each project. Based on the evaluation results, adjust priorities, allocate resources, and make decisions about continuing or discontinuing projects.

The decision making process is ultimately an iterative one, where the data leadership works with the data team to understand the ideal team composition for the business requirements, and the evolving business requirements standing as the guiding light for choosing the data projects that get the final green light. Iterate, adapt and lead your data teams to success.

(Originally published by the same author @ harini.blog)