Sexism in ChatGPT: How does AI’s hidden bias impact us?

• Gender bias in AI models such as ChatGPT raises concerns that some responses may inadvertently convey gender stereotypes
• Despite developers’ efforts to reduce bias, data bias and cultural differences still make gender equality difficult to achieve
• Increased transparency and user feedback mechanisms are the direction of improvement, and AI needs to strike a balance between de-bias and personalized experiences

Amazon developed an AI tool in 2014 to screen job candidates, aiming to simplify the hiring process using AI. However, the company soon discovered that the AI recruitment system tended to rate female candidates lower, especially for technical positions.

Amazon’s testing of the recruitment system revealed that the AI exhibited unfair bias against female candidates. This finding shocked the tech industry. Despite Amazon’s heavy investment in developing an AI tool to screen candidates, the company ultimately abandoned it due to the tool’s implicit bias against women. This case highlighted that AI systems may ‘unintentionally’ carry gender discrimination, even when the designers’ intentions are neutral.

AI technology is rapidly transforming the way we interact with the world, yet the issue of gender bias remains pervasive. We cannot ignore this phenomenon, as it goes beyond individual interactions and subtly shapes our social perceptions.

This raises an important question: Do language models like ChatGPT also unintentionally reflect the gender biases present in society? In our everyday interactions with AI, how might these biases influence our beliefs and decisions?

Also read: The double sexism of ChatGPT’s flirty “Her” voice

Unconscious bias of ChatGPT (AI)

ChatGPT is designed to provide neutral and objective responses, but are its answers truly free from gender bias? AI models primarily rely on large-scale datasets for training, often containing text from social media, websites, and other public sources. When this text reflects societal gender biases, AI models may unintentionally reproduce these ‘unconscious biases’ in their responses.

In some tests, we found that ChatGPT tends to use a particular gender to represent specific professions or roles in certain responses. One example is when users ask ChatGPT to identify who might be better suited for nursing roles or executive positions. ChatGPT may subtly lean toward responses shaped by gender stereotypes. These nuanced biases, though less overt than traditional discriminatory language, can still impact users’ perceptions and decision-making.

How should we understand gender bias in ChatGPT? Some users may not immediately notice these subtle differences, as ChatGPT’s responses often appear objective. However, the issue lies in its ‘unconscious bias’. While AI lacks any actual gender perspective, it unconsciously mirrors deeply ingrained stereotypes present in its training data. This not only affects users’ perceptions but can also create a new ‘feedback loop’ of bias. That is, users internalize the biased information from AI, reinforcing these views in real life, which in turn deepens the biases in data. “After a model learns from biased data and is redeployed, the model’s bias in the generation of new data can be further reinforced, creating what is known as a ‘bias amplification effect'”, says Rohan Taori, PhD in computer science, Stanford University.

Regarding feedback loops and bias in AI, computer scientists emphasize that data feedback loops can reinforce pre-existing biases in data sets.

After a model learns from biased data and is redeployed, the model’s bias in the generation of new data can be further reinforced, creating what is known as a ‘bias amplification effect’

Also read: A new study finds ChatGPT is quite gender-biased

Why is gender equality in ChatGPT so difficult to achieve?

Developers design large language models like ChatGPT with the goal of meeting diverse conversational needs. And then providing a wide range of information and support.

The researcher is developing large language models such as ChatGPT

However, it is worth noting that ChatGPT’s training data covers a large amount of web content and public text. And the content itself often reflects gender biases in society. Even as OpenAI tries to filter out more balanced and diverse data, many biases remain hidden in subtle language. For example, the model’s responses quietly carry in some stereotypes about occupations, abilities, and personalities. Ruhi Khan, the ESRC researcher at the LSE, pointed out that male pronouns were less likely to be associated with traditionally female professions, and female pronouns were often linked to nurturing or less technical roles.

At the same time, ChatGPT users come from all over the world, each with different gender perspectives, cultural backgrounds, and social norms. ChatGPT struggles to accurately balance these cultural differences, ensuring that every user can have an appropriate, unbiased conversational experience.

ChatGPT’s users are culturally diverse and gender diverse

To reduce gender bias, ChatGPT’s development team could incorporate more de-bias strategies into the text generated by the model. There are challenges, however. Conversations that are too neutral may seem ‘cold’ or ‘formulaic’, lacking the personality and naturalness of interaction.

This also raises another core issue: should we completely eliminate bias, or balance different gender traits while respecting diversity? Current AI technology has yet to perfectly address this issue, but we can make AI closer to neutral by strengthening ethical guidelines and increasing technological transparency.

Also read: A Comprehensive Study of ChatGPT: Advancements, Limitations, and Ethical Considerations in Natural Language Processing and Cybersecurity

PopQuiz

What is the main reason why language models like ChatGPT may exhibit bias when handling gender-related topics?

A. The designers intentionally included gender-biased algorithms

B. Most AI models are trained using data that contains gender bias

C. AI models automatically generate gender bias without any data

D. ChatGPT is designed specifically for a certain gender, so it is biased

(The correct answer is at the bottom of the article)

‘Gender awareness’ in ChatGPT: A necessary equality or a new challenge?

We can see introducing ‘gender awareness’ into ChatGPT as a double-edged sword. On one hand, it allows the model to better align with user needs in certain specific scenarios, providing detailed and personalized services. On the other hand, careless implementation of AI’s gender recognition and adjustments could raise ethical issues and even unintentionally worsen gender bias. As AI researcher Kate Crawford pointed out, “AI systems are not neutral. They are often deeply rooted in societal norms, and when not carefully designed, they can exacerbate existing inequalities.” This quote from Kate Crawford underscores that AI is not neutral. Especially in sensitive areas such as gender. If not designed properly, it can exacerbate inequality in society.

Researchers such as Dr. Moatsum Alawida highlight that eliminating bias without sacrificing the authenticity of conversations remains a complex and ongoing challenge in AI development. He pointed out that “Removing biases without sacrificing conversational realism remains a complex, ongoing challenge in AI development”.

Removing biases without sacrificing conversational realism remains a complex, ongoing challenge in AI development

The following will delve into the necessity and risks of ‘gender awareness’ in AI.

How does gender awareness enhance personalized experiences?

A new perspective suggests that AI should not completely ignore gender, but instead require a more nuanced ‘gender awareness’. This awareness can not only help avoid bias but also provide more personalized and detailed responses in specific contexts. When discussing AI and gender, some experts argue that AI should not completely ignore gender. Instead, a “gender conscious” approach should be adopted. This perspective underscores the importance of recognizing the implications of gender bias in AI systems and taking steps to consciously address these issues. According to reports from organizations such as the OECD and IBM, although AI has the potential to reduce gender inequality. But if poorly managed, it can also reinforce stereotypes. OECD pointed out that AI should not ignore gender entirely, but rather embrace a ‘gender-aware’ approach.

Additionaly, Sara Sterlie, researcher at the Technical University of Denmark (DTU) said:”We expected some gender bias, as ChatGPT is trained on material from the internet that to some extent reflects the gender stereotypes we’ve known for many years. “

AI should not ignore gender entirely, but rather embrace a ‘gender-aware’ approach

In certain situations, considering gender differences can enhance the quality of AI’s services. For example, in areas such as mental health, education, and career advice, users of different genders may have varying needs.

Suppose a student is using ChatGPT to seek study advice; by understanding the user’s gender, AI may be able to recommend study resources better suited to the user’s background and interests, or provide suggestions that align with the user’s social needs.

Christine Exley and her colleagues at Harvard University conducted a study. They explored how women tend to downplay their achievements and are less likely to self-promote than men. In the context of AI and ChatGPT, these gender-based dynamics can influence the way users receive responses and interpret the information provided. Because AI models often reflect biases embedded in their training data. As Exley points out, solving this problem requires more than just encouraging women to be “more confident.” It involves changing the institutions and frameworks that shape gender expectations. And challenge the underlying social norms that drive these differences.

Women tend to downplay their achievements and are less likely to self-promote compared to men, even when their performance is objectively equal

Ethical Challenges of gender consciousness: Might stereotypes be reinforced?

However, despite the potential advantages of ‘gender awareness’, it may unintentionally reinforce stereotypes. According to reports from organizations such as the OECD and IBM, while AI has the potential to mitigate gender inequality, it also has the potential to reinforce stereotypes if poorly managed. As mentioned earlier, when ChatGPT attempts to recognize and adapt to a user’s gender, it may make judgments based on the societal norms prevalent in the training data. For example, if the model offers different career recommendations based on gender, it may inadvertently convey gender biases from traditional views, such as suggesting that the tech industry is more suitable for men, while education or humanities fields are better suited for women.

This issue is particularly concerning in career and education contexts. On one hand, personalized gender awareness may provide a more tailored experience that meets users’ needs; on the other hand, this recognition and adaptation may reinforce gender stereotypes, especially when AI makes gender-differentiated suggestions based on mainstream beliefs.

Therefore, the introduction of gender awareness also brings new challenges—how to balance fairness and diversity in ChatGPT?

Also read: AI image generators often give racist and sexist results: can they be fixed?

Thinking about the future

As AI technology continues to develop, we can reasonably believe that efforts will gradually address the issue of AI bias. However, the key question remains—how should we design AI so that it can maintain objectivity and neutrality while being sensitive to and adaptable to gender diversity? Future AI may need to find a new balance between equality and personalization.

The issue of gender bias in ChatGPT serves as a reminder that AI is not a completely neutral tool, but rather a product influenced by its data and society. Although fully eliminating gender bias may be impossible, future versions of ChatGPT can achieve greater fairness by incorporating diversified data, using transparent prompts, and implementing user feedback mechanisms.

At the same time, for every user of ChatGPT and other AI tools, understanding the limitations of AI and maintaining critical thinking are key to addressing this issue. In other words, we must remain vigilant, not blindly relying on AI’s responses, but instead be aware of the potential biases behind them.

As we increasingly rely on AI, gender equality is not only a technical issue, but also an ethical one that requires the collective discussion of society. How AI becomes a more inclusive and diverse technology in the future will deeply impact our daily interactions, values, and our understanding of “gender equality.”

Quiz answer

B. Most AI models are trained using data that contains gender bias