个人 Github Blog 地址: https://wj-mcat.github.io/agent-handbook/docs/paper-reading/2024/08/meta-reward-language-models-self-improving-alignment-with-llm-as-a-meta-judge
现在 LLM-as-a-Judge 概念这么火,那 Judge 的能力绝对不能弱啊,所以作者提出了新的方法来提升模型 Judgement 的能力。
方法简要介绍:
效果:能够提升模型 judgement 和 Instruction following 的能力。
该方法使用一个 seed-model (已经 SFT 过,同时具备 Instruction Following 的能力),然后有如下流程:
其中第三个阶段是核心工作内容,对应的 Prompt 如下所示:
Review the user’s question and the corresponding response, along with two judgments. Determine which judgment is more accurate according to the rubric provided below. The rubric used for the initial judgments is as follows: - Add 1 point if the response is relevant and provides some information related to the user’s inquiry, even if it is incomplete or contains some irrelevant content. - Add another point if the response addresses a substantial portion of the user’s question, but does not completely resolve the query or provide a direct answer. - Award a third point if the response answers the basic elements of the user’s question in a useful way, regardless of whether it seems to have been written by an AI Assistant or if it has elements typically found in blogs or search results. - Grant a fourth point if the response is clearly written from an AI Assistant’s perspective, addressing the user’s question directly and comprehensively, and is well-organized and helpful, even if there is slight room for improvement in clarity, conciseness or focus. - Bestow a fifth point for a response that is impeccably tailored to the user’s question by an AI Assistant, without extraneous information, reflecting expert knowledge, and demonstrating a high-quality, engaging, and insightful answer.
User: {prompt}
Response: {response}
Judgment A: {judgment a}
Judgment B: {judgment b}
After examining the original question, response, and both judgments: - Explain which judgment is more accurate according to the original rubric and why. Consider factors such as adherence to the rubric, accuracy in evaluating the response, and consistency in applying the criteria. - Conclude with a clear statement of which judgment is better using the format: “Winner: [Judgement A | Judgement B]”
最后质量最高的 Judgement 将参与到模型的训练当中,进而提升模型的 Judgement 能力。
这是一个专为移动设备优化的页面(即为了让你能够在 Google 搜索结果里秒开这个页面),如果你希望参与 V2EX 社区的讨论,你可以继续到 V2EX 上打开本讨论主题的完整版本。
V2EX 是创意工作者们的社区,是一个分享自己正在做的有趣事物、交流想法,可以遇见新朋友甚至新机会的地方。
V2EX is a community of developers, designers and creative people.