Other Writings

Validation Using AI

How do you use generative AI to research when it is known to have biases and "hallucinations"? Even beyond that, how can AI be trusted in apophatic philosophical topics when it is necessarily positivistic? How can AI be trusted with cause and effect when there is no a priori test for causes and effects (ala David Hume)? 

A way I am using AI is to validate my research and ideas, particularly on topics that are more obscure or where there is no clear consensus. The process of validation enables a comparison between what I have found and what the flawed database of all knowledge, along with the hallucinogenic additions of generative AI, claims to have. Where it does not support my position, more consideration and better research are needed. In the slide deck below, I have the output from AI validating minor and major points in the AGB and Redeeming Asimov.

ChatGPT is the first one I've tried. Historically, most of what I'm getting back seems plausible if not correct, as can be seen by the Anshar entry on the first slide (1). A task like translating to and from Sumerian cuneiform is a bit more mixed (2). Even in basic stuff like matching a sign to a cuneiform symbol, it is making a lot of easy mistakes. When asked to do a specific part again or reconsider, it produces something correct or at least plausible. But, if I didn't know enough to ask more specifically, I would have wrong or misleading information.

And then there are truly bizarre cases, such as when it was given 6 lines of Sumerian transliteration and told to translate into English. It would be generous to describe the translation as lazy. It was like a kid who doesn't want to do his homework, so he fakes it, writing any old thing just to turn something in. There was maybe a 5% correspondence between the transliteration and ChatGPT's translation. When asked to make a "literal translation", it produced something credible. I didn't have to ask for the translation to be "literal" the first time, and it did a decent job. The lesson I'm learning so far is that you can't trust the first answer.

One way of validation that I have found ChatGPT fairly useful is to ask a general, but edgy factual question that forces ChatGPT to choose between presenting fact or opinion. Then, as a form of argument, press on all the soft parts of the factual and non-factual synthesis it produces with follow-up questions. ChatGPT is mostly a fair arguer and, to please, may give way too easily to the interlocutor. If I can prompt ChatGPT to change its disagreeable opinions to more acceptable ones with just a few factual questions, that is validation. Because most AIs are programmed to please, prolonged questioning to get acceptable opinions should not be seen as a success. The PDF file links to the right or below contain chats of various lengths, where I was able to direct ChatGPT to my way of thinking within a couple of questions of a disagreement.

I have some concerns about security and privacy, so most of the AIs are not ones I'm logged into. Perplexity AI is the exception. It usually avoids outright wrong answers on the first try by being less ambitious and hedging. Perplexity AI is not very sensitive to chat flow, and the quality and context of the sources it cites sometimes leave something to be desired. This can lead to contradictions down the line. For instance, after establishing that the genocide of Gaza by starvation narrative has no empirical support and that media outlet Al Jazeera is systemically pro-Palestinian and anti-Israel, Perpelexity AI goes on to cite Al Jazeera as a primary source when asked for media sources with more balanced reporting! Furthermore, all of the sources it cites as "balanced" forcefully carry the genocide narrative as an unquestionable fact! 

On conditions as they impact social issues, Perplexity seems to have a whichever way the wind blows sort of feeling. I prefer Perplexity AI when searching for quick, uncontested facts. ChatGPT seems better for validation and hypothesis testing of bigger ideas.