Verifying AI-Generated Content

Generative AI can invent a reference, but that is not the only problem to look for. A publication may be real while the AI misstates its findings, assigns it to the wrong category, or turns a limited conclusion into a broad claim.

For that reason, verification involves two separate questions:

• Does the cited source exist?

• Does the source actually support the way it has been used?

A reference deserves closer attention when several fields look plausible individually but do not fit together.

Examples include:

• a real scholar paired with a topic outside that scholar's publication record;

• a real journal paired with an impossible year or issue number;

• a real volume paired with an article title that does not appear in the table of contents;

• a journal article followed by an institution that is not the journal's publisher;

• a DOI that does not resolve, has an invalid prefix, or leads to a different publication;

• a title assembled from broad keywords but lacking the specificity normally found in the field.

These are indicators that warrant further verification, not final proof of fabrication.

Use AI for triage, but make the final decision yourself

Checking dozens of references one at a time is slow. AI can help with the first pass by separating citation fields, spotting internal contradictions, and suggesting where to search.

For example, when viewing a thesis in a browser, you might ask:

I suspect that some of these references may be fabricated or internally inconsistent. Create a table with the author, year, title, journal or publisher, volume, issue, pages, and DOI. For each item, identify specific warning signs and suggest an authoritative source for manual verification. Do not invent a corrected citation.

A stronger prompt may be needed because an AI assistant often assumes that a formally written bibliography is valid. Still, a forceful prompt does not make the answer reliable. It may simply make the model more willing to accuse a source of being false.

1. Test the DOI

A DOI is a persistent identifier, not just a web address.

• Open the DOI link or search the complete DOI string.

• Confirm that it resolves.

• Compare the author, title, journal, and year with the citation.

• Remember that a real DOI can be attached to the wrong publication.

• A broken DOI is a strong warning sign, but not every legitimate publication has a DOI.

2. Search Taiwan theses and dissertations

Most Taiwan theses and dissertations should be discoverable through the national thesis system or the university repository.

Check:

• the exact title;

• the author's name;

• the institution and department;

• the degree;

• the graduation year;

• the adviser, when available.

In the case study, the cited dissertation attributed to 邱振瑞 could not be found. The available record for a person with that name was a 2008 engineering master's thesis from National Chiao Tung University, not a 2013 doctoral dissertation from National Taiwan University's Department of Geography.

3. Check Chinese-language journals

Search the journal title and article title in:

• the National Central Library's Index to Taiwan Periodical Literature;

• Airiti Library;

• CNKI, when relevant;

• the journal's own website;

• the publisher's archive.

• Then inspect the actual year, volume, issue, publisher, and table of contents.

• If neither a bibliographic database nor an official journal site has a record of the journal, it may be fabricated. It may also be obscure or poorly indexed, so continue checking before making a definitive claim.

In the case study, the thesis cited several supposed articles from Router: A Journal of Cultural Studies. The journal's publication history shows that it began in 2005, which makes a citation to issue 7 in 2000 impossible. The journal's official publisher also does not match several institutional names listed in the thesis.

4. Check Western-language books and articles

Search an exact title in quotation marks through Google Scholar, Crossref, WorldCat, or the publisher's site.

In the case study, the book Cultural Attractions and European Tourism can be confirmed through Google Scholar and other catalogs. When citing an edited book or a book with distinct chapters, identify the chapter actually used whenever possible. That makes the evidence easier for another reader to check.

5. Check reports on the issuing organization's website

For reports attributed to UNESCO, OECD, WHO, IPCC, government agencies, or other institutions, search the official site.

6. Treat conference papers and internal reports with care

Conference papers, local reports, and internal publications may not be indexed online. In such cases:

• confirm that the event or organization existed;

• look for an official program, proceedings, or archive;

• check the author's institutional or personal publication list;

• contact the author, conference organizer, or institution.

In the case study, the reference attributed to 葉高華 did not appear on the researcher's publication page. That is a reason to investigate further, not conclusive proof by itself.

Finding the publication is only the first step.

AI may:

• change the meaning of a source;

• present a factual description as an author's opinion;

• turn a conclusion limited to one context into a universal claim;

• treat an example as the central argument;

• expand an abstract beyond what the full paper supports;

• turn “A is associated with B” into “A causes B”;

• describe a source that rejects a claim as if it supported the claim.

1. Recheck numbers and dates

Numbers, years, percentages, units, and denominators are especially easy to distort.

Use the original table, dataset, or authoritative website to determine:

• what the number measures;

• the population and period;

• whether it is a count, rate, percentage, or estimate;

• whether the AI attached the number to the wrong topic.

2. Compare the AI summary with the abstract and conclusion

Authors usually state their main contribution and interpretation in the abstract and conclusion. If the AI's account points in a different direction, read the relevant sections closely.

Do not assume that a statement appearing in the literature review represents the paper's own finding.

3. Ask for the exact passage—but verify it yourself

Useful follow-up questions include:

• On which page or in which chapter does the source make this claim? Quote the original passage.

• Does this source explicitly discuss [topic], or is that your inference?

• Why is this source more appropriate for supporting the claim than the other sources you retrieved?

If the system says it cannot view page-level content, or supplies a page number where the passage does not appear, the citation is not verified.

4. Search within the document

Use Ctrl+F to find the relevant terms, then read the surrounding paragraphs.

Check:

• whether the term appears at all;

• whether the passage says what the AI claims;

• whether the author is supporting or criticizing the point;

• whether the term appears in the authors' results or only in their review of earlier work.

A title containing “creative class,” for example, does not show that the book discusses Indigenous young people returning to their communities. That connection must appear in the text.

5. Evaluate whether the source is worth citing

A source may exist and be represented accurately but still be weak evidence.

Consider:

• peer-review status;

• journal or conference quality;

• method and sample;

• date and jurisdiction;

• correction or retraction notices;

• whether the author has relevant expertise;

• how later researchers have used or challenged the publication.

Check internal consistency

• Look for contradictions in dates, people, chronology, and causal relationships. Ask the same question more than once and compare whether the answer remains stable.

• Example: An answer says Feynman invented quantum computing in 1980 and Google achieved quantum supremacy in 1990, although Google's announcement was in 2019.

• Strengths and limits: Easy to use, but repeated questioning may be needed. 

Watch for overconfidence and vague language

• Absolute phrases such as “certainly correct” and vague phrases such as “some studies” may hide a knowledge gap. Ask for the specific study, author, year, and source.

• Example: “Some studies show that AI improves educational outcomes.” Ask the system to name the studies and publication years.

• Strengths and limits: Helps users notice suspicious claims. If the system cannot supply traceable sources, the claim remains unverified.

Compare more than one AI tool

• Ask the same question in systems such as ChatGPT, Grok, and Perplexity. Record the differences rather than choosing the answer you prefer.

• Example: Asked about ethical issues in medical AI, one system emphasizes privacy while another emphasizes bias.

• Strengths and limits: Time-consuming, but useful for exposing differences in framing and omission. 

Cross-check scholarly databases

• Search PubMed, Google Scholar, Scopus, or a subject database using the title, keywords, author, or DOI.

• Example: An AI claims a 2023 study found 95% diagnostic accuracy. Search the exact claim and confirm that the study and dataset exist.

• Strengths and limits: Well suited to academic topics, but requires basic search skills. 

Consult authoritative websites and guidelines

• Compare key claims with government agencies, professional bodies, or international organizations such as WHO or IPCC.

• Example: Check a climate claim against the IPCC report rather than relying on the AI's summary.

• Strengths and limits: Useful for scientific and policy questions, provided the authority and jurisdiction are appropriate.

Seek expert or community feedback

• Ask a subject specialist or post a focused question in a relevant scholarly community.

• Example: Compare an AI answer about psychology research with informed discussion on ResearchGate or a field-specific network.

• Strengths and limits: May reveal minority or practical perspectives, but responses still need evaluation.

Refine the question iteratively

• When an answer is vague, ask for the source, DOI, peer-review status, method, and limitations.

• Example: “You said AI improves educational outcomes. Give three specific peer-reviewed studies and DOI links. Were these papers peer reviewed?”

• Strengths and limits: Can reduce unsupported claims, but repeated interaction does not replace independent checking.

People may want a detector for two different reasons:

• instructors want to know whether a student used AI without permission;

• writers want to know whether legitimate AI-assisted editing will be viewed with suspicion.

A 2025 Pew Research Center study found that many Americans reacted negatively when told that news or political speeches had been written with AI assistance. This social pressure may discourage disclosure and encourage the use of “humanizer” tools.

Detector scores are not proof

AI-writing detectors use different methods and return different results. Their accuracy varies with:

• language;

• text length;

• genre and discipline;

• model;

• translation and editing;

• multilingual writing;

• mixed human-AI authorship.

They can produce false positives, particularly for writers using English as an additional language. Several institutions have therefore limited or discontinued the use of Turnitin's AI indicator.

A detector score should never be the sole basis for an accusation or misconduct finding. Review the full process: assignment rules, drafts, version history, notes, sources, citations, and the writer's ability to explain the work.

What “AI-like” writing can look like

Studies have found that AI-generated writing may retain recurring vocabulary and stylistic patterns even when asked to imitate a historical author. Researchers have also observed shifts in word frequencies in academic writing after the release of widely used chatbots.

Possible signs include:

• prose that is unusually smooth and uniformly structured;

• repeated transitions, keywords, sentence patterns, or punctuation;

• arguments that look complete but remain shallow;

• generic examples;

• little personal experience, disciplinary judgment, or reflection.

These are prompts for closer reading, not authorship tests. Human writers can write formulaically, and AI-assisted text can be substantially revised.

Preserve the author's voice

AI editing can improve speed, clarity, and flow while subtly changing emphasis or certainty. The final version still needs close human revision.

You can try the following editing prompt:

Role: You are an expert academic editor in [field].

Task: Revise or translate the text so that it meets high standards of academic English. Improve clarity, precision, and flow while maintaining a natural, human-authored tone.

Style constraints: Avoid using words such as delve, underscore, showcase, boast, surpass, employ, leverage, unravel, illuminate, or foster merely for decoration. Use a specialized term such as catalyze only in its literal disciplinary meaning. Avoid inflated modifiers, clichés, and metaphors. Prefer direct academic phrasing such as aim to, investigate, evaluate, suggest that, associated with, and further research.

Text to revise: [Paste the draft.]

This prompt is best understood as a way to preserve clarity and authorial voice—not as a method for evading detection or concealing unauthorized AI use.

CRAAP

• Currency (The timeliness of the information):

  • When was the information published or posted?

  • Has it been recently revised or updated?

  • Does your topic require the absolute latest information (like medical or technology news), or are older, historical sources acceptable?

• Relevance (The importance of the information for your needs):

  • Does the information relate directly to your topic or answer your specific question?

  • Who is the intended audience (e.g., experts, children, the general public)?

  • Is the information at an appropriate level for your needs?

• Authority (The source of the information):

  • Who is the author, publisher, source, or sponsor?

  • What are the author's credentials or organizational affiliations?

  • Is the author qualified to write on this topic? (Look for .edu or .gov domains for academic or government sources).

• Accuracy (The reliability, truthfulness, and correctness of the content):

  • Where does the information come from?

  • Is the information supported by evidence, citations, or references?

  • Has the work been peer-reviewed?

  • Can you verify the claims made by checking other independent sources?

• Purpose (The reason the information exists):

  • Why was this information created? Is it to inform, teach, sell, entertain, or persuade?

  • Is the author's point of view objective and impartial?

  • Are there clear political, ideological, cultural, religious, institutional, or personal biases?