What we ask LLMs to do

[Bootstrap]

for fun I fed our debates into a few llm's and asked them who won, who was consistent and what the primary point of each user was.

These were specifically stripped of identifiers so they had no bias reviewing it.
Gemini likes Bootstrap
Claude likes me
Chatgpt hedged

Often, the differences are less about the LLMs than what we ask them to do.

Notice that you asked “who won,” which frames this as a debate. In my experience, anything that signals taking sides strongly nudges an LLM to do the same. It guesses what kind of answer the user wants and cheerfully provides it. I do not know what you have been asking Claude previously, but that history also influences the results, it may already know which views you agree with.

I asked Claude 4.5 to recommend some prompts for summarizing articles objectively, and here are some that it recommended.

Copy-Paste Ready Prompts

Ultra-short:

Code: Select all

Summarize this objectively without evaluation.

Balanced (recommended):

Code: Select all

Provide a neutral summary of this article's arguments, distinguishing factual claims from assertions.

With structure:

Code: Select all

Outline this article's main arguments objectively, noting which claims are supported by evidence.

Academic style:

Code: Select all

Summarize this article as a critical reader would, identifying the thesis, supporting arguments, and any unsupported claims.

Journalist style:

Code: Select all

Summarize this article's key points neutrally, as if for readers who will form their own opinions.

With skepticism trigger:

Code: Select all

Summarize this article objectively, flagging generalizations or contested claims.

Simple and direct:

Code: Select all

What does this article argue and what evidence does it provide?

For controversial topics:

Code: Select all

Summarize this article's position without endorsing it, noting where claims are debatable.

Any of these will work better than an unqualified "summarize this" when you want neutrality.

[barnhart]

When I see something like "I fed this question in AI" in my mind it registers as "I shook the magic 8 ball until this answer turned up."

[Bootstrap]

When I see something like "I fed this question in AI" in my mind it registers as "I shook the magic 8 ball until this answer turned up."

You mean that people have an answer they want, and they keep asking AI questions until they see it? Yes, I think that's probably what most people do. Often without knowing it - you have to think carefully to avoid that.

I spent a little time this morning trying to ask a set of objective questions about the article that we were discussing in the other thread. It may be a bad example because it's pretty nerdy, but it also shows how careful prompting can bring out information that did not come up in that thread. I used Claude 4.5 for all requests, mostly because it's the same LLM that Soloist used.

If you look at the prompts I used, I was trying to avoid introducing my own bias and encourage it to consider things from multiple points of view.

This is a Google Doc with several tabs:

https://docs.google.com/document/d/1kec ... sp=sharing

[Soloist]

for fun I fed our debates into a few llm's and asked them who won, who was consistent and what the primary point of each user was.

These were specifically stripped of identifiers so they had no bias reviewing it.
Gemini likes Bootstrap
Claude likes me
Chatgpt hedged

Often, the differences are less about the LLMs than what we ask them to do.

Notice that you asked “who won,” which frames this as a debate. In my experience, anything that signals taking sides strongly nudges an LLM to do the same. It guesses what kind of answer the user wants and cheerfully provides it. I do not know what you have been asking Claude previously, but that history also influences the results, it may already know which views you agree with.

I asked Claude 4.5 to recommend some prompts for summarizing articles objectively, and here are some that it recommended.


Any of these will work better than an unqualified "summarize this" when you want neutrality.

What you don't seem to understand is that using LLM's is a waste of time and most people just jump over a forum post made by them.

I did some of these things as a point in itself. As for the who won, I know how LLM's work and I know how to as I said, strip it from bias. Then it pattern matches based on the discussion itself as to who it believes won. You can try it for yourself. Open a new account, ask the questions and it pattern matches without the bias of the user as it has no indication as to which user is actually asking.
The facts though, the response is not consistent and will change and this wasn't really productive. The point was the lack of productiveness.
You are open about using Chatgpt for your responses, I did the same and it appears to have bothered you.

I had not discussed any of Paines work with it prior and the majority of my discussions with it have been centered around little coding projects I'm using to learn code.
You can also do the same by creating a project and outside of your "main" memory it isn't connected to your chats at least on Claude. Memory function has been integrated with Chatgpt much longer and you are much more likely to get a bias response using GPT then Claude.

That being said, Claude is more of a coding LLM and Chatgpt is a generalist.

When I see something like "I fed this question in AI" in my mind it registers as "I shook the magic 8 ball until this answer turned up."

I agree mostly. Google searching is painfully full of ads and sometimes you can find what you need quicker with the right prompt. I can ask random questions such as how to program a specific function and compared to searching for the information, its much faster.
But to use it for forum posts, for religious conversations, debates, its just going to give you what you put in mirrored.
Its just a robust auto complete. You feed it intent, and it spits out its auto-corrupt version.

[Bootstrap]

What you don't seem to understand is that using LLM's is a waste of time and most people just jump over a forum post made by them.

Then why did you trash a thread by dumping raw output from LLMs into it? That's not what I was doing in the thread.

Why intentionally flood a thread with something you yourself see as garbage?

[Soloist]

What you don't seem to understand is that using LLM's is a waste of time and most people just jump over a forum post made by them.

Then why did you trash a thread by dumping raw output from LLMs into it? That's not what I was doing in the thread.

Why intentionally flood a thread with something you yourself see as garbage?

Because the assumption is you are doing so.
Also, you refuse to actually engage in any points someone makes or you hand wave them away.

Besides, it wasn't that I viewed it as garbage, but to argue over something of that volume when you wouldn't even engage with the points anyway saved a great deal of time and effort when it basically pointed out the flaws you were ignoring again and again. You kept shifting the argument and didn't engage in the discussion on the point I was making.

I do think though that it would be helpful if you don't make posts from chatgpt edited or not. I'll read your words, but I'm not going to bother if you make a post that is just AI.

Besides, I didn't say it was garbage. I said it was a waste of time. Distortion of my point.

[Soloist]

Besides, it would appear that your objective neutral review of her article broke it apart on very similar grounds to me. It didn't use Japan like I did as that portion of getting forced open into a maritime power, embargoed heavily and then getting crushed before prospering under occupation completely gets ignored by it. I'm assuming those details aren't commonly discussed and then it didn't identify it from training sources.

[barnhart]

When I see something like "I fed this question in AI" in my mind it registers as "I shook the magic 8 ball until this answer turned up."

You mean that people have an answer they want, and they keep asking AI questions until they see it?

Yes, that's one angle but also included is the category of asking the wrong questions. It's like looking in the Bible to find justification to kill my enemy. The type of questions you should be asking of AI is fact sifting, how many times have the borders of Switzerland shifted since 1800? How many Japanese civilians died in WW2? How many states have legalize marijuana consumption? Which supreme Court cases in the last 10 years deal with parental rights? Not "What is the meaning of life?" or "Are my arguments better than my opponent's?" You may as well use a numeric code to type these questions into a calculator, this is simply beyond the boundaries of AI even if it pretends to give an answer.

[Bootstrap]

I do think though that it would be helpful if you don't make posts from chatgpt edited or not. I'll read your words, but I'm not going to bother if you make a post that is just AI.

Let me push back on two things here.

That thread started by pointing to an article, with a summary that I wrote with help from AI, but the summary was not "just AI". Then I quoted two paragraphs that summarized what I was focused on, directly from the article. I don't think it is fair to claim that I outsourced my thinking.

And I don't agree that I am refusing to engage or hand-waving away points. I tried to identify specific claims you made and respond to them directly in multiple posts. We may have different interpretations of the article or different understandings of how the world works, but that's the whole point of engaging with people who have different perspectives. It's not a wrestling match to force the other person to agree.

[Bootstrap]

Yes, that's one angle but also included is the category of asking the wrong questions. It's like looking in the Bible to find justification to kill my enemy. The type of questions you should be asking of AI is fact sifting, how many times have the borders of Switzerland shifted since 1800? How many Japanese civilians died in WW2? How many states have legalize marijuana consumption? Which supreme Court cases in the last 10 years deal with parental rights? Not "What is the meaning of life?" or "Are my arguments better than my opponent's?" You may as well use a numeric code to type these questions into a calculator, this is simply beyond the boundaries of AI even if it pretends to give an answer.

I think this is a very useful direction.

I think it can be useful for more than fact sifting, but I agree that it is good for the first set of questions you list, and bad for the second set. I think there are some questions in between that it can do well at, and others that it does badly at.

I think AI is best at what we call "constrained tasks" that have:

• External grounding
• Clear truth conditions
• Independent verification

And those are also areas where we can verify whether it got it right with available information. Here is a list of such tasks - I got this directly from ChatGPT, but it matches my experience so far. These are things LLMs can do well if used correctly:

• Fact retrieval
• Enumeration
• Classification
• Pattern summarization
• Source aggregation

AI is especially bad at things like:

• Deciding who is morally right or wrong
• Answering big “life meaning” questions
• Declaring who won an argument
• Acting like an authority when people don’t actually agree on the standards