[Paper Reading]: Self-Improving Alignment with LLM-as-a-Meta-Judge

94 天前
 wjMcat

个人 Github Blog 地址: https://wj-mcat.github.io/agent-handbook/docs/paper-reading/2024/08/meta-reward-language-models-self-improving-alignment-with-llm-as-a-meta-judge

现在 LLM-as-a-Judge 概念这么火,那 Judge 的能力绝对不能弱啊,所以作者提出了新的方法来提升模型 Judgement 的能力。

方法简要介绍:

  1. 让模型推理得到结果
  2. 同时来评估答案的内容
  3. 用评估结果来调整训练模型

效果:能够提升模型 judgement 和 Instruction following 的能力。

方法详细介绍

该方法使用一个 seed-model (已经 SFT 过,同时具备 Instruction Following 的能力),然后有如下流程:

其中第三个阶段是核心工作内容,对应的 Prompt 如下所示:

Review the user’s question and the corresponding response, along with two judgments. Determine which judgment is more accurate according to the rubric provided below. The rubric used for the initial judgments is as follows: - Add 1 point if the response is relevant and provides some information related to the user’s inquiry, even if it is incomplete or contains some irrelevant content. - Add another point if the response addresses a substantial portion of the user’s question, but does not completely resolve the query or provide a direct answer. - Award a third point if the response answers the basic elements of the user’s question in a useful way, regardless of whether it seems to have been written by an AI Assistant or if it has elements typically found in blogs or search results. - Grant a fourth point if the response is clearly written from an AI Assistant’s perspective, addressing the user’s question directly and comprehensively, and is well-organized and helpful, even if there is slight room for improvement in clarity, conciseness or focus. - Bestow a fifth point for a response that is impeccably tailored to the user’s question by an AI Assistant, without extraneous information, reflecting expert knowledge, and demonstrating a high-quality, engaging, and insightful answer. 

User: {prompt} 

Response: {response} 

Judgment A: {judgment a} 

Judgment B: {judgment b} 

After examining the original question, response, and both judgments: - Explain which judgment is more accurate according to the original rubric and why. Consider factors such as adherence to the rubric, accuracy in evaluating the response, and consistency in applying the criteria. - Conclude with a clear statement of which judgment is better using the format: “Winner: [Judgement A | Judgement B]”

最后质量最高的 Judgement 将参与到模型的训练当中,进而提升模型的 Judgement 能力。

732 次点击
所在节点    Python
0 条回复

这是一个专为移动设备优化的页面(即为了让你能够在 Google 搜索结果里秒开这个页面),如果你希望参与 V2EX 社区的讨论,你可以继续到 V2EX 上打开本讨论主题的完整版本。

https://www.v2ex.com/t/1063131

V2EX 是创意工作者们的社区,是一个分享自己正在做的有趣事物、交流想法,可以遇见新朋友甚至新机会的地方。

V2EX is a community of developers, designers and creative people.

© 2021 V2EX