Currently enjoying: Prompt Tuning.

This started off at work with a requirement to give users an FAQ-type answer-bot able to answer simple questions, but our knowledge base consisted of completely unstructured data. I’ve never done this before, but it came together very quickly — by which I mean how to get some sense of control over the outcomes.

What needs to be done is to formulate question-answer pairs “by hand”, in a test catalogue. Just state what you’d ideally expect your bot to answer, imagining the user asks this particular thing or that. Come up with 20 or so of those. Then feed the bot a system prompt and the knowledge base, pass the question, and compare the answer to the formulated expectation. Do this for the whole test catalogue. Let an LLM “improve” the system prompt, and repeat the process until you have a system prompt which produces the kind of usable answer you’d like to see, given a knowledge base.

Now, this technique is coming in handy for my hobby project of ingesting books in a meaningful way (by my own definition). One task I’m interested in is getting highlighted passages automatically into my resource archive.

The same technique applies here. The underlined-text extractor looks at a page and produces the highlighted passages. Now we do this by hand for 20 or so pages and jot down our expectations, against which we run the same procedure. Run it once, compare the outcomes to the expectations, tweak the system prompt of the extractor, try again.

The one innovation, from my naive vantage point of re-discovering the wheel for myself, is to measure this (there exist pretty standard similarity measures) — which I’ll surely carry over to my work problem.

In any case, what also needs tuning are some intuitions. Driving the prompt by expectation matches the habitual driving of implementation by specification. But what costs me a little is letting the system prompt be completely generated. This stems in part from my conviction that some hand-written and curated instructions are really worth a lot (no proof!). In this case, I have learned to totally let go.

I treat the system prompt as basically some activation pattern — not different from the weights of the model itself. I believe this is the right way to see it.

Yesterday I stumbled on another article which talked about this technique as analogous to neural stimulation, where you mention certain words, then observe the neural activation in the neural net, and then directly “overstimulate” those weights (manipulating the thresholds) to nudge outcomes in certain conceptual directions. Which I think is interesting, but why not just mention those words again and again? Don’t think of a blue elephant, don’t think of a blue elephant, don’t think of a blue elephant type of thing.

Anyway, be that as it may — definitely interesting.