Tero Keski-Valkama<p>How to live within the prompt size constraints of <a href="https://rukii.net/tags/LLM" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>LLM</span></a> <a href="https://rukii.net/tags/chatbots" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>chatbots</span></a> such as <a href="https://rukii.net/tags/ChatGPT" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>ChatGPT</span></a>?</p><p>When automating practically any real-world task, the amount of contextual knowledge required can be astounding. Meanwhile current ChatGPT has a limitation for tokens in the prompt+output of 4,000 tokens. The quality also goes down sharply long before this limit is reached.</p><p>The solution is simple: Decompose and structure the task into independent components which can be handled one by one in a conditional tree.</p><p>You take the input to the process, and ask ChatGPT: "Does this relate to topic X. Answer only yes/no."</p><p>You capture the answer and depending on that you go deeper into that subtopic, each time in a new, clean session. When you reach a "leaf" in the decision diagram, you ask ChatGPT to produce a conclusion for that one thing only.</p><p>You collect the conclusions across many, many sessions, in aggregate applying a huge amount of contextual knowledge to the task over multiple clean sessions, and finally you get the whole result which you can combine from the sub results either with ChatGPT or procedurally.</p><p>To do this effectively you need to <a href="https://rukii.net/tags/document" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>document</span></a> your domain in a hierarchical fashion in small bite-sized, understandable bits. This doesn't only help <a href="https://rukii.net/tags/AIs" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>AIs</span></a>, it will also help new <a href="https://rukii.net/tags/employees" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>employees</span></a>.</p><p>In the future, this encoded knowledge base will become a core asset of any <a href="https://rukii.net/tags/business" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>business</span></a>. No better time to start <a href="https://rukii.net/tags/documenting" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>documenting</span></a> your <a href="https://rukii.net/tags/processes" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>processes</span></a>, <a href="https://rukii.net/tags/procedures" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>procedures</span></a> and <a href="https://rukii.net/tags/tasks" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>tasks</span></a> than now!</p>