Research Questions
- How do LLMs perceive and interpret user personas?
- How does cultural context (particularly user profiles from India) shape LLM interpretations?
- Can LLMs reconstruct demographic profiles based on persona descriptions?
Results
- GPT-3.5 and GPT-4 showed strong alignment across the three India-centred personas analysed.
- The highest scores were assigned to Consistency (GPT-3.5: 6.34, GPT-4: 7).
- The lowest scores were given to Credibility (GPT-3.5: 5.67, GPT-4: 6.33), reflecting limitations in perceived realism.
- GPT-4 consistently reconstructed demographic traits such as age, income, tech proficiency, and occupation.
- High agreement was observed between the two models’ outputs.
Findings
- Consistency:
- LLMs achieved the strongest performance in capturing internal coherence among persona attributes.
- Credibility Challenges:
- The low realism scores (“Does this persona feel like a real person?”) reflect an expected limitation.
- Demographic Reconstruction:
- Models were able to infer demographic profiles from persona descriptions.
- Persona B (Dependent Family Talker): estimated to be aged 50+, with low-to-middle income and low tech proficiency.
- Persona C: predicted to be a small business owner or entrepreneur with medium-to-high tech proficiency.
- Model Agreement:
- GPT-3.5 and GPT-4 produced highly similar ratings, with minimal divergence across evaluated dimensions.
Scores
- LLM Models: 5
- Synthetic Data: 2
- Method: 4
- Speed: 1
- Ethics: 2
- Accuracy: 4
- Demographics: 5
If you would like to explore this research in more detail, click here to read the full paper.