Completely OT - It's always important to eat your own dogfood.
For those who don't know, this is developer talk for consuming your own product/code so that you suffer with the same issues that your users do and know better.
As you know, one of my AI projects is a AI patenting lawyer (actually, it is a multiagent legal patent firm with more than two dozen personas/agents interacting with each other, let's assume it is a monolithic application). This system serves as both my in-house counsel and as my research testbed for some of my ideas.
I have 4 different patent families running simultaneously, each family has 4 - 10 inventions in it. As I got close to filing, I realized that the AI drifted and I had wasted ~40 hours of work. So now I had to trace the conversation and progress of the concept to find the exact point of mistake and then redo the work from there on. But being me, I also have to figure out why and how this mistake happened and prevent it from happening again.
The excerpts and condensed traces that I had to go through for this one family alone was 350 pages. Part of the cause was that I should not do 18 hour workdays when mental acuity is required so I can detect and correct the AI's mistakes as they happen. Another part of the problem was that my favorite feature of ChatGPT, its persistent long context memory, worked against me and the AI conflated the work in one workstream that was in a different conversation thread with this one. And then there was AI program itself that had shortcomings I had not sufficiently patched.
Once I figured out the problem and discussed with the AI, it asked me several deep probing questions. They were in the form of: "To prevent my mistake of doing X again, I need to know how to distinguish the line between A and B. How should I resolve this conflict?"
These were not easy questions to answer and I could not get any help from the AI about them (if it knew, it would have asked me to resolve them). As humans, the bulk of our knowledge is implicit. For example, when walk, you don't think about the exact mechanism of how you balance yourself, how you avoid stepping on rocks or potholes, or what you do when you do step on them. The same type of problems exist for AI inferring some meaning out of some context. And it was asking me to help it out.
The system is now substantially better than before. I am grateful for the mishap, even though it has meant that I've been up till 4 AM for several days. |
