Large Text Pattern Analysis with Prompted Models
Feed large batches of text into a single context window to extract overall patterns and sentiment across many posts, enabling scalable, non-sequential analysis while monitoring for hallucinations.
One capability that stood out to me early on with GPT-3—and I don’t think it was explored as fully as it could have been, and in many ways still feels underexplored with these models—is that we’re used to interacting with them sequentially. I ask a question, I get a response, I ask another question, and so on.
But these models can do several things at once. You can give them a list and say, “Point out five things and respond to them.” Or you can give them a large chunk of text and ask for an overall read.
Early on, I was asked to work with a social media company to see whether you could tell how a basketball game was going by looking at a lot of tweets—let’s just call them posts. One way to do that would be to take each individual post from accounts you’re following, ask the model how the game is going, and then average the results.
But you can also do something much more direct: you can put 40 or 50 of these short posts into a single context window and say, “Tell me what’s going on.” And it was actually pretty good at that. It could look at a big conversation and give you an overall opinion in one shot. It worked with message boards and other places, too.
I was even asked to explore applications for the intelligence community—basically, ways to quickly assess what was happening in forums: what the overall sentiment was, not just from one person but across multiple people. You can also use this approach to find patterns. For example, if you suspect people are posting under multiple accounts, you can feed in a bunch of posts and ask, “Who feels the same?” or “Who sounds similar?” Models are actually pretty good at that kind of mass analysis.
Of course, there were upper limits. If you pushed too far, hallucinations would start to creep in. But the interesting part of this wasn’t, “Here’s a lot of text—give me some big, complicated structural analysis.” It was more like: “Here’s a lot of text—look for patterns.” That’s what it was good at: spotting familiar patterns.
And even when the words look different to us, in embedding space some phrases and intents are closer than you’d expect, and the model can often tell you something about how people were thinking. I still think this is an area that isn’t explored as well as it could be: taking a bunch of examples that seem unrelated and asking the model, “Is there a pattern here?”
It’s not magic. It’s about looking beyond spelling and surface form and trying to pick up intent. Someone might have a habit of ending paragraphs a little angrier than they start. The words might change, but the tone is consistent. Or someone might consistently use more visually descriptive language rather than abstract language. Models can be surprisingly sophisticated at picking up those kinds of patterns.