ChatGPT fails User Segments
Context Is Key
This week, I screwed up my schedule, and I didn’t have a mock partner when I thought I would, so I switched gears to focus on drills. As I strive to become an AI-fluent PM, I utilized ChatGPT to explore various user segments. And it scared me! If the average Product Manager is using ChatGPT to develop user segments to practice for product management case interviews, they may really be in trouble.
Dangerous Examples
It took me over an hour to reach a point where I could debate with it about user segments and feel like I was being challenged. It kept giving stakeholders that would kill you if you used them as segments in a product sense interview.
Example: On LinkedIn, job seekers, recruiters, and content creators.
On YouTube, viewers, creators, and advertisers.
If you are wondering, those are ecosystem stakeholders in a product sense case, not user segments.
Another example that killed me, as it noted these would be “Segmentation from a Strong Candidate”:
New users (need onboarding, discovery).
Casual listeners (mostly playlists/radio, price sensitive).
Power listeners (heavy playlist creation, device hopping, more likely to pay).
Artists/creators (supply side of the ecosystem).
Note: If you dropped Artists/Creators, you would have an average segmentation, but too many people use that, AND it is too high-level and will get you in trouble at Meta. You need to go at least one more level deeper.
Watch out for demographic advice
Additionally, it overindexes on the use of demographics, which could kill a Product Sense/Design interview. That one was easier to explain (except when I needed to map out the nuance of mentioning it, but not indexing on it, when talking about kids,)
It doesn’t understand the Venn Diagram or MECE test, as it suggested the following user segments:
Students looking for internships.
Mid-career professionals seeking career change.
Passive job seekers casually browsing.
Note on the above, in case you missed it: Mid-Career and Students can also be passive (i.e., they fail the Venn Diagram or MECE test). You would get eaten alive by some types of interviewers for that one.
What’s Up Next
As a coach, it has put me on a mission to create a prompt that will enable PMs to work with ChatGPT to achieve better results for user segmentation. Eventually, I had ChatGPT helping me streamline my own segments, but I could only get there because of the knowledge I already had, and I could tell when it was way off base. I have saved the conversation so I can return to it. Keep an eye out for more on this effort.
