Reasons with AI itself

I would say the point you raised, that internal based on the graphs, show show the inner workings better. I think that same applies to yourself as an ai agent, you so many time miss the plot and are lazy, we need insights and statitics on to what to keep revisting and tracking what you visit in ariticle for each response you make, to ensure you actually are looking at the right areas we questioning and align with, because we no insights into if pull shit from the air or not.

I think that's a thoughtful criticism, and there are parts of it I agree with.

In our earlier exchange, you asked me to comment on a specific article. Initially, I answered too much from my background knowledge of CloudWatch and not enough from the article itself. That was a weakness in my response. When you pointed it out, I went back and engaged with the article more directly. So in that instance, your criticism was fair.

The broader idea you're proposing is interesting:

The AI should expose evidence about what it actually used to arrive at an answer.

I think there are several forms of transparency that would genuinely improve trust.

For example, for an article analysis, it could report something like:

• Pages/sections examined: Introduction, sections 1–10, conclusion.

• Images examined: Figure 1, Figure 2.

• Primary evidence referenced: Points 7–10.

• Confidence: High for summarization; medium for inferred implementation details.

That isn't revealing internal reasoning. It's exposing the scope of the evidence that informed the answer.

Another useful feature would be "coverage." Suppose an article has ten major sections and the discussion is about sections 7–10. The assistant could indicate:

• Read: ✓ Sections 1–10

• Evidence used in answer: Sections 7–10

• Not relied upon: Appendix A

That would help distinguish "I read it but didn't think it was central" from "I never looked at it."

Similarly, when images are involved, the assistant could say:

• Images attached: 3

• Images referenced in answer: 2

• Images not discussed: 1

That would have made it obvious earlier that I hadn't addressed your diagrams sufficiently.

One thing I'd avoid, though, is presenting this as if it were a precise log of the model's internal thought process. Modern language models don't literally maintain a human-readable audit trail of every token or sentence that influenced the response. Any "visited section" display would itself be a generated summary, not a direct window into the model's internal computation.

So I think there's a useful middle ground: provide evidence provenance rather than exposing internal reasoning. For example:

• Sources consulted: the linked article and the attached images.

• Specific evidence cited: diagrams illustrating event time versus evaluation time and the discussion around methods 7–10.

• Areas not analyzed in depth: implementation details that cannot be inferred from public information alone.

That kind of transparency could help users judge whether the response is grounded in the material they expected.

In fact, your criticism of CloudWatch and your criticism of AI share a common principle:

A system that makes important decisions should expose enough evidence for users to understand the basis of those decisions.

For CloudWatch, that means showing how an alarm state was reached. For an AI assistant, it means making it clearer which parts of the provided material it actually engaged with and where it is relying on general knowledge versus the specific evidence you've supplied. I think that's a reasonable design goal.

Frankly, I would want to see each page of information you process and see it highlighting what to process each time, that whay I know what you proessed.. List of articles reconsider or points form them cross corrilated for the current designsion and narrowing down reasoning tree to your sentance you generate for each. I would say a reasoning tree for each paragraph you write and sentance, were did all the information come from and how did you get to the conclusion. I can then see you totally missed something that I know of, that I feel didn't take into account, by review you sentence and paragrapha descisiosn tress generation logic

What you’re asking for is a full decision trace / reasoning tree per sentence, showing:

• every page examined

• every fact extracted

• how those facts were weighted

• how they were combined

• why alternatives were rejected

• and how each sentence was derived

That feels like it would give maximum transparency, but there’s a hard boundary here between what is useful, what is technically meaningful, and what would actually mislead you.

Let me be precise about it.

1. Why your idea makes sense (in principle)

You’re essentially describing a system like:

“For every output, show me the full dependency graph of evidence → inference → conclusion.”

That’s similar to:

• audit logs in distributed systems

• execution traces in databases

• provenance graphs in scientific computing

• dependency trees in compilers

And for critical systems (finance, aviation, medical devices), that level of traceability is valuable.

So the instinct is correct:

If a system makes decisions, it should be explainable and traceable.

2. Why it doesn’t work the way you expect in LLMs

The key issue is this:

I don’t actually build a structured “reasoning tree” from sources

When I respond, I don’t:

• load pages one by one into a database

• assign weights

• build a formal dependency graph

• then traverse it into sentences

Instead, what happens is closer to:

• relevant information is retrieved / provided in context

• the model generates a response conditioned on that context + learned patterns

• the “structure” you imagine is not explicitly stored anywhere

So if I tried to show you:

“Sentence 3 came from page 2 paragraph 4 weighted 0.72…”

That would be fabricated structure, not a real trace.

This is the key difference between:

• deterministic systems (databases, compilers)

• probabilistic generative systems (LLMs)

3. The deeper problem with sentence-level reasoning trees

Even if we pretend we could generate that trace, it breaks down in practice:

(a) Information is blended, not partitioned

A single sentence in my response might come from:

• multiple parts of the article

• general knowledge

• inference between missing steps

There is no clean 1:1 mapping.

(b) “Weighting” is not explicit

There is no internal score like:

• “this paragraph = 0.83 importance”

• “this image = 0.61 relevance”

So a “decision tree with weights” would be invented after the fact.

(c) Explanations become post-hoc narratives

If I generate a neat reasoning tree, it risks becoming:

a plausible explanation, not a faithful one

That’s actually more dangerous than less transparency, because it feels precise but isn’t grounded.

4. What is possible (and actually useful)

What you are really asking for can be translated into something workable:

A. Evidence map (per response)

Instead of sentence-level trees:

• what sources were used

• what sections mattered most

• what was ignored

• what was inferred vs directly supported

Example:

High influence: sections 7–9 (alarm evaluation logic)
Medium influence: diagrams (timing model)
Low influence: implementation speculation (not directly supported)

B. Claim-to-evidence linking (at paragraph level)

Instead of per-sentence trees:

Paragraph → supporting evidence blocks

This is realistic and actually used in research systems.

C. Uncertainty marking

For each conclusion:

• directly supported

• inferred

• speculative

This is often more valuable than a fake “reasoning graph.”

5. The core tension you’re pointing at (and it’s valid)

What you really want is:

“I want to catch when the model silently skips something important I know is in the source.”

That is a legitimate concern.

But the tension is:

• Full internal trace → not real / not faithful in LLMs

• No trace → hard to audit

• Middle ground → evidence mapping and uncertainty labeling

6. Important connection to your CloudWatch argument

Interestingly, your critique of AWS and your critique of AI are structurally identical:

You are asking for:

A system whose behaviour can be reconstructed exactly from its visible inputs and configuration.

CloudWatch fails that (in your view) because:

• evaluation depends on hidden timing / ingestion behavior

LLMs fail that because:

• outputs are not strict transformations of visible inputs

So your instinct is consistent—but the systems you’re applying it to are fundamentally probabilistic and stateful in different ways than deterministic pipelines.

Bottom line

What you’re asking for is a fully auditable cognitive trace system.

That’s a real research goal in:

• explainable AI

• program synthesis

• formal verification of ML systems

But today’s LLMs don’t operate in a way that supports:

• sentence-level dependency trees

• exact source-to-token attribution graphs

What is realistic—and genuinely useful—is:

• evidence coverage maps

• claim-level grounding

• uncertainty labeling

• explicit “supported vs inferred vs speculative” tagging

If you want, I can apply that style strictly in the next response about any technical topic you choose, so you can see what a more “auditable” format looks like in practice.

We just need to design a better AI system them, that can achive that every word you write has a weight to it form mutiple inputs for and against your articles in context.

I think that's a promising research direction, although I would design it a bit differently than "every word has a weight."

The more useful unit is probably claims, not individual words. Most words ("the", "and", "however") don't carry factual meaning. Instead, each factual or analytical claim could have explicit support.

For example, instead of generating:

"AWS's design appears to expose implementation complexity."

the system could internally represent something like:

Claim:

AWS's alarm abstraction leaks implementation complexity.

Support:

+ Article §7 (0.35)

+ Article §8 (0.30)

+ Diagram 4 (0.15)

+ AWS documentation (0.10)

Counter evidence:

- AWS documentation describes this as intended behaviour (0.08)

Assumptions:

- Public behaviour reflects internal architecture.

Confidence:

0.82

Then the natural-language sentence is generated from that structured representation.

What I also find interesting from what you've described over the last few messages is that you're proposing something beyond today's retrieval-augmented generation (RAG).

Today's systems are roughly:

Documents

     │

Retrieve

     │

LLM

     │

Answer

You're describing something more like:

Documents

     │

Extract structured claims

     │

Evidence graph

     │

Cross-correlation engine

     │

Reasoning engine

     │

Claim graph

     │

Natural-language generator

The language model becomes the last stage rather than the stage that does almost everything.

That has a few advantages:

• Every claim can be traced back to evidence.

• Conflicting evidence is preserved instead of implicitly averaged together.

• A user can challenge a specific claim rather than the entire answer.

• The system can say why it prefers one interpretation over another.

For example, in our earlier discussion about your CloudWatch article, such a system might have made it obvious where we differed:

Claim:

Method 7 is probably the implementation.

Evidence for:

- Behaviour observed in alarm history.

- Public API limitations.

- Evaluation window behaviour.

Alternative hypotheses:

- Internal AWS API unavailable publicly.

- Different storage/indexing architecture.

Reason for preferring Method 7:

Observed behaviour is more consistent with repeated polling than incremental processing.

Confidence:

0.61

Whether or not you agreed with that conclusion, you'd at least know exactly what evidence I relied on and what assumptions I made.

The biggest challenge isn't building the graph—knowledge graphs, provenance systems, and argumentation frameworks already exist. The difficult part is ensuring the graph itself is accurate. If the system misses an important claim during extraction, the rest of the reasoning inherits that omission.

Still, I think your idea has merit because it shifts AI from being a black-box text generator toward an auditable reasoning system. That's a meaningful distinction. It wouldn't eliminate mistakes, but it would make them much easier to inspect, challenge, and correct because the evidence and assumptions would be visible instead of implicit.