On Generative AI
39 The Degenerative, part 1
Generative AI, as a deep learning tool, has inherited all the ethical and social fallouts of machine learning models.
Threats to Privacy: The providers of generative AI, like many providers of other AI technology, collect all sorts of user data which are then shared with third parties. OpenAI’s privacy policy concedes that it deletes user data if requested but not user prompts, which can themselves contain sensitive information that can be traced back to the user1.
There is also the risk that people reveal more sensitive information in the course of a seemingly human conversation, than they would otherwise do2. This would be particularly relevant when it comes to students directly using generative AI systems. By being so successful in imitating human-like language, especially for a child’s grasp of it, this technology “may have unknown psychological effects on learners, raising concerns about their cognitive development and emotional well-being, and about the potential for manipulation”3.
Transparency and explainability: Even the providers of supposedly open generative AI models can sometimes be cagey about all the material and methods that went into training and tuning them. Moreover, as deep models with millions of parameters, the weights assigned to these parameters, and how they come together in bringing about a specific output, cannot be explained3.
Both the form and content of the output can vary widely, even where there would be little difference in the prompt and user history2. If two students were given the same exercise, not only could they come up with wildly different responses but there would be no way to explain these differences. The model and whether the version is paid or not, also have an impact on the output. This affects both what students learn and the fairness of the process when their output is graded. But banning their use is also problematic, since it will widen the gap between the learners who can access it at home, and those who cannot1.
Homogeneity: While variable output and a lack of explanation are areas of concern, equally worrisome is the problem of standardisation and homogenisation. As models trained from internet data, generative AI systems promote certain views and cultural values above others, limiting learners’ exposure to diverse perspectives and their capacity for critical thinking3. “Every data set, even those that contain billions of images ie, text pairs scraped from the internet, incorporates some world view, and slices the world into categories that may be highly problematic”4. For example, Wikipedia, a popular recourse for training datasets, has predominantly male content creators5.
As foundation models6 designed to be adapted to all sorts of tasks downstream, this tendency towards homogenisation is stronger than in other machine-learning models. However, how they are adapted seems to have a big role in whether the homogenisation is heightened, weakened or