Let’s take a look at AI and what the state of the tech means for today’s busy Product people.
As a Product Consultant it won’t surprise you to hear that a lot of clients are seeking to respond to Board requests to ‘investigate what we could be doing with AI’. To help you answer the same question yourself, here is the AI + Product story so far – what is possible with today’s tech and how to approach applying it to your products.
The story so far.
Firstly, let’s just take a breath, because we’ve seen changes like this before. Consider some of the advances you have seen in your life: a computer on every desk (if you are as old as me), the internet, email, smartphones and cloud. I can remember being wowed, intimidated and initially resistant to each one, so when faced by machines that feel like they may soon become smarter than we are, it’s reasonable to have a similar reaction to AI. But it is important to keep this latest step forward in perspective and to evaluate clearly how the tools might translate into real use cases in your products.
‘AI’ is just the latest step in the evolution of data science, which has been busily underway since the 1950s. The 18 months since ChatGPT launched has been exciting and importantly very visible to the public, but it is a step change (not a big bang) nonetheless.
So what has occurred so far? We had rule engines back in the 80s, but for 2 decades we were limited by storage and compute. Enter the cloud and hardware improvements, and by the Noughties data science teams found they could finally store and process the mountains of data they needed to make meaningful progress in Machine Learning. Neural networks emerged, and pattern recognition boomed.
Recommendation engines arrived, so Netflix and Spotify started suggesting the next flick or the next song. Large language models emerged, identifying words that often go together, and chatbots with canned responses became commonplace. Then the latest step arrived – Transformers. No, not Optimus Prime and friends, but an advancement in neural networks that add the concept of ‘attention’ to analysis of sequential data. Transformers process data sequences faster and can handle more complexity, so have fuelled the emergence of GenAI (Generative AI) such as OpenAI’s ChatGPT and Google’s Gemini. And here we are, in the moment Marketing teams are calling AI.
Current state of the tech
We’ve now got a healthy toolset to play around with in this space: the capability to store data in datalakes and analyse the ripples (patterns) produced, Machine Learning algorithms to identify patterns far beyond what humans are capable of, and GPTs (Generative Pretrained Transformers) that utilise LLMs (Large Language Models) to summarise existing content and even generate new text, images, video and audio. It feels like we have arrived, that anything is possible. The pace of advancement is certainly growing and many fear the rise of Skynet.
But remember that this is technically only the world of ‘narrow AI’ – intelligence that is limited to a specific field or data set. Machines trained to a specific purpose, be it recommendations, document summaries or content creation. This is not AGI (Artificial General Intelligence) – that comes later, when the models begin to think and reason. That’s when things will get really exciting and you finally get that hoverboard you’ve been eagerly awaiting.
What is possible today?
So how are people starting to use AI and specifically, how are Product people using it? The short answer is that we are experimenting with exciting new tools. Some fledgling use cases have emerged, but as Pendo recently found, 49% of people are ‘just playing around with AI to see what is possible’.
We are running around with a shiny new hammer, looking for nails.
As you’d expect, the benefits we seek boil down to efficiency gains, cost reductions and new opportunities.
So how are people putting the new tech to use today? Let’s group what we’re seeing as follows:
1. Doing more with data
This is where the quick wins are. Businesses are drowning in data and many are taking the time to do more with what they already collect, or to define new metrics and to gather data for today’s use cases. Almost every digital system will be recording logs and a range of other data. You may have a data lake and centralised Data team, or have evolved to a data mesh with distributed access.
2. Improving existing products
PMs are getting excited here. As I write this, Notion AI is making suggestions I’m ignoring. GitHub CoPilot has had a huge impact. Jira and others are close behind. Adding a chatbot was exciting a few years back, but now teams are looking to add recommendations to it for customers to self-serve, and then there is analytics. Oh yes, machine learning has finally (along with big compute and big storage) placed truly delightful analytics within reach. Users will delight! There are gains here both internally by identifying previously unknown trends in operational or user data, and externally by adding Machine Learning-driven insights to digital products, or GenAI-powered suggestions when text input is common.
3. Building new products
Some are even building completely new products (like Mackmyra whisky, the Swedish startup who trained a model to suggest recipes and saved decades of trial and error). Generally this is machine learning, though GenAI-based products are being touted, few have reached MVP. And where businesses have ‘built our own AI’, they are generally an application of a public GenAI model to an internal use case. Right now, we see AI value actually being delivered by doing more with data and training a Machine Learning model on your dataset. GenAI will come, but still suffers from ‘hallucinations’ – providing accurate-looking answers that cannot be verified as correct.
4. AI assistants
This is what the Board are excited about and this is where the hype is. ‘Automate your life!’ the pundits say (and implicitly, deliver more work with fewer people). This seems the easy win and has leaders across the land itching to explore it. But as you will likely know from testing tools out, beyond searching, summarising documents and limited Q&A, GenAI is not quite there yet. Yes it is exciting, evolving rapidly and well worth testing, yes GitHub CoPilot will be already helping your Engineers, but tools beyond the basic are likely not yet ready for the primetime. Yes, early gains have been made in content generation for Marketing departments (the gains are real and the conversion rates will surprise you), but be cautious of jumping in too quickly and filling the internet with slop. Optimise processes before replacing them wholesale with GenAI-powered automation.
For example, at MTP London recently, Lindsey Jayne shared the story of The Financial Times improving semantic search by building a proof of concept RAG (Retrieval-Augmented Generation) model to test and prove the search and summarise use case. The RAG technique improves LLMs by incorporating external data sources and the proof of concept was loved by the initial test group so they plugged in GPT Anthropic Claude in order to scale it. This delivered an excellent search and summary tool on the site. GenAI did not begin to write articles for the Financial Times, but did make their vast library of content far easier to access and learn from. In this way businesses are tapping into what they already have and are augmenting existing processes using today’s AI toolset.
5. As a learning aide
Yes, we learn a lot in Product and it takes time we just don’t have. So using GenAI to get answers quickly or to understand a topic an Architect or a Customer just mentioned is a godsend. This is where I use GenAI a lot. Call me old, but I remember when Google first arrived and how it just seemed like a better way to search. I feel the same now when using Google Gemini Advanced or Perplexity (to access OpenAI in an anonymous way and to see sources used in results). I also love tools trained on author content such as Lennybot – for answers that I know come from a reputable Product source. I would not however suggest copy-pasting the responses into your work. Treat them as blog content from an unknown author – as interesting ideas to be verified before use.
What might be possible tomorrow?
The generalists will get smarter. They will graduate from ‘high school’ (today) to ‘university’ and eventually ‘post-doc’ levels of answers on an ever-widening number of topics. They will power search both on the web and in-app.
The specialists will also advance. This is the exciting area for many Products folks. Ever more useful off the shelf models will be deployed and trained on app data, and will converse with both internal users and customers in ways indistinguishable from humans. Cue cost savings and efficiency gains galore.
PM tooling will improve. in ways that PM platform Fibery have only just begun to hint at and you will spend ever less time on admin. That’s right, PRD writing, preparing roadmaps for reporting, OKR cycles, story writing and bug describing all receive AI assistance.
You’ll have the data. I say this because though our job interviews decry it, few PMs outside of big tech truly have access to useful data today. In future it will become ever easier to access data – and in-built AI helpers will serve you up insights galore. Pendo are scratching the surface of this now.
Analytics galore. We will see a range of businesses incorporate not only GPTs but also machine learning to predict, forecast and recommend in ways that we will find obvious and will frankly forget we ever lived without. This area has huge potential for Product Managers. Cue doing more with data and even, the rise of truly data-driven decision making and (gasp) data-driven organisations.
Put the Management back in PM? Indicators are that you will put all this spare energy into the Management of your Products and look up, outwards and ahead for your insights and ideas. You will spend more time thinking, planning and communicating with internal and external stakeholders, and less time explaining why data is necessary in the first place.
Where to start
- Start with why. Yet again, Simon Sinek was right. As with any big idea, start with the need, the nail, then build the hammer (while filtering out the hammer salespeople and those who believe we will never need hammers).
- Pick a use case, then take an iterative approach to testing hypotheses via proof of concept, MVP and limited release cycles – just like any exciting new feature.
- It all starts with data. Our conversations tend to start with ‘the Board wants to know our AI strategy’ and move swiftly on to ‘data strategy? We don’t really have one but we have lots of data’. If you remember one thing, it is this: garbage in means garbage out. So take the time to improve the inputs. You will likely be surprised how many easy wins you discover in data you already have.
- Be honest. Build transparency into your AI-powered features so that users can see the logic behind the suggestion. Minimise number of both bugs and angry users caused by AI failing silently and hallucinating answers. De-risk by exposing steps made to achieve an outcome, so users can understand the logic and perhaps ask in another way. Perplexity shares sources, and Google Gemini lists steps, formulae and python code used during calculations (example: estimating a retirement pot).
- For more background, have a listen to this episode of the 101 Ways podcast where Jon and Emma’s take us through the state of Data & AI and how to apply it.
Things to be aware of
- Test everything, in cycles. More than any other new technology, the output of models must be tested. My colleague Jon Parish recommends preparing a set of test data to compare model-generated results against, to mitigate against AI failing silently.
- Always include a disclaimer. Google Gemini does this well.
- Consider privacy and data governance when choosing a tool. Read the fine print, else your data may be used to train the model.
- Avoid bill shock. As Citi have done, consider fair usage when rolling out AI-powered tools to internal teams. Queries on paid tools carry a cost and some may overly rely on AI assistants.
In summary
That was a long one, but this felt like an important story to tell. We as Product people are heralded as the custodians of user value, the people who should learn about new toys early, so that we can apply them to our products. The reality is that the Board may kick start your AI investigations in this case, so it is important to educate yourself on the history of data science, the technology types in use, and emerging tooling you might put to use internally and in your products.
Finally I’d urge you to take a leaf out of your favourite helpful Engineer’s book and learn in public. Share what you try and enjoy. Just as with the migration to Cloud or emergence of DevOps, this exciting new era involves new tools, ideas and ways of working. Share your ideas to help others and in return learn from others in the tech community.
Thank you for reading this post, written by a synthetic Product consultant (just kidding, but for how long).
Photo by Austin Rucker on Unsplash