为人类水平的句子纠错铺路

导读

这是 Grammarly 工程团队关于“句子级润色/纠错”研究的技术博客：他们尝试用机器学习把不合语法或不地道的句子改写成流畅英文，并构建了人工标注数据集来做评测。对写作者的价值在于：它让我们看到机器眼中的“流畅”是什么、改写到什么程度算成功，以及为什么“地道”这件事很难有唯一正确答案。

关键词　句子润色流畅度语料与标注机器写作

可以直接用的写作要点

01
“流畅”没有唯一客观定义——同一句话在不同地区、不同母语者眼里可能一个算地道一个算别扭，因此写作时要先想清楚你的目标读者是谁。
02
评测要选对参照物：如果只赢了表现很差的基线系统，就不算真正的进步；判断自己的稿子好坏也一样，要和真正的好文本比，而不是和明显糟糕的样本比。
03
训练数据（以及任何写作素材）的质量决定上限：用带错别字、用词不当的低质句子练手，学到的也是低质模式。
04
机器常犯的错误类型提示我们：拼写相近的词（homonym/近形词）比随机打错的字更常见也更危险，写完要专门检查这类“看起来对但其实用错”的词。
05
让语言/写作专业者参与实验设计：纯技术视角容易陷入“追求唯一正确结果”，而自然语言往往不存在这样的结果。

HN 评论精选

“流畅”没有客观唯一的定义，技术团队应当与语言学家合作、谨慎设计评测。

作者要解决的问题既有趣又困难。我注意到，用 AI/机器学习处理自然语言时，计算机科学家做的工作常常能从语言学家或其他语言专家的合作中大大受益，尤其是在实验设计阶段。这篇工作就是个好例子。受过计算机科学这类精确学科训练的人，久而久之会形成“要得到正确结果”的思维定式（我自己就是这样的人）。但处理自然语言时，有时根本不存在唯一的正确结果。就拿作者研究的“流畅度”来说：有没有严格、客观的“流畅”定义？任何语言学家都会告诉你：没有。习语、语法结构、缩略形式、俚语等等，在一个国家内部从城市到城市就有差异，更别说全球了。一个母语者觉得“流畅”的句子，另一个可能觉得奇怪。所以客观地整体评估“流畅度”是不可能的。任何执业语言学家大概都能随口举出例子：某句英文被某个地区的人评为“流畅”，却被另一个地区的人评为“别扭”。此外，用 Mechanical Turk 找人给句子流畅度打分……

aisofteng查看英文原文

The problem that the authors are trying to tackle is an interesting and difficult one. I have noticed that when dealing with natural language using artificial intelligence / machine learning techniques, the work being done by computer scientists very often would have greatly benefitted from collaboration with a linguist or other sort of language expert, especially in the design phase of an experiment. This work is a good example of what I mean. People trained in CS or similar precise fields develop, over time, a tendency of thinking in terms of "getting the right result" (I say this as one of these people). When dealing with natural language, however, sometimes there simply is no single correct result. Consider the topic of fluency that the authors work on: is there a rigorous, objective definition of "fluent"? The answer, as any linguist would tell you, is "no". There are idiomatic expressions, grammatical structures, contractions, slang, and so on that vary from city to city within a country, let alone globally. What may sound "fluent" to one native speaker of a language may sounds strange to another. It is impossible to objectively generally evaluate "fluency". In particular, any practicing linguist will be able to give examples, likely off the top of their head, of English sentences that would be rated as "fluent" by someone from one geographical area and "awkward" by someone from another. Furthermore, using Mechanical Turk to find humans to rate the fluency of a particular sentence makes for an unclean dataset and evaluation benchmark. The linked post says that, in the end, 50 people found via Mechanical Turk rated sentences for fluency; since any one language is used significantly differently around the globe, there will be an unpredictable range of fluency ratings for at least some sentences across just 50 people around the world. Choosing a different 50 people to rate the same sentences would most likely result in different fluency ratings. I do not mean to detract from the authors' work; this is a difficult problem to tackle, with no clear path to a general solution. However, I am forced to wonder why the authors, who, based on their biographies linked to in the article, seem to have a range of experience, did not comment on the considerations I've mentioned here.

质疑评测：只战胜表现糟糕的基线系统意义有限，且训练数据的来源值得商榷。

我不太理解……按那个柱状图，自动系统连一半的拼写错误都改不对。比如 “advertissment”，现在 macOS 还在这文本框里拼命地想帮我改。再比如 “From this scope, social media has shorten our distance”，这里的 scope 应该是 perspective。这看起来是机器学习应该很容易发现的，我把原句（德语的）拿去 Google 翻译试了一下，也没出现这个错误。所以我不确定“赢过那些失败得相当离谱的系统”有多大价值。我也不太明白为什么这个任务需要人工创建数据，而不是直接把所有 TOEFL 作文题里写过的文本买下来，再拿《纽约时报》档案库来对照。当然，很可能是有充分理由的，他们大概比我刚想的这些考虑得更周全。

matt4077查看英文原文

I don't quite get this... According to the bar graph, the automated systems fail to correct around half of even orthographic (spelling) mistakes. One example is "advertissment", which macOS is now trying really hard to correct against my will in this text area. Another example is "From this scope, social media has shorten our distance", where "scope" is supposed to be "perspective". That seems to be something that machine learning should easily pick up on, and indeed, when I just tried it on google translate, I couldn't get it to make this mistake without my original (german) sentence also encroaching awkwardness. So I'm unsure how much value it is to win against systems that fail rather spectacularly. I also don't quite understand why you would need manually-created data for this task, instead of just buying everything ever written in TOEFL essay questions and pitting it against the New York Time's archive. It's obviously quite likely that there are good reasons for all this. They may have thought a bit longer about it than I just did.

训练数据质量是瓶颈：错误类型不真实、标注者水平有限，低质数据只会带来低质结果。

我感觉这里有点“垃圾进、垃圾出”。首先，他们的原句里放的是一些不太可能出现的拼写错误，而不是更常见的同音/近形词混淆（这肯定也让学习变复杂，因为有些改动是为了纠正拼写相近的词，而这类修改一旦应用可能改变句子意思）。其次，人工修改的质量也不怎么样。我们真该停止用匿名、被压榨、只拿几分钱的劳动力来造数据集（他们确实筛选了 Amazon Turk 用户，但如果你住在美国或在一所大学工作，身边真的缺流利的英语母语者吗？）。总的来说，这种流畅度编辑的思路有前景，对英语学习者会是好事，但训练数据需要改进。

cooper12查看英文原文

Hmm I'm sensing a bit of garbage-in garbage-out here. For starters their original sentences contain unlikely typos instead of homonyms which would be much more commons. (it complicates the learning as well I'm sure since some changes were made to correct similarly spelled terms which could really change a sentence's meaning once applied) Second, the human corrections aren't that good. We really need to stop creating data sets using anonymous exploited labor that is paid pennies. (they did screen the Amazon Turk users, but if you live in America or work at a university, is there really a shortage of fluent English speakers around you?) Overall I'd say the fluency-editing approach shows promise and would be a boon to ESL-learners, but the training data needs to be improved.

工程师们二十年里聊过的写作。

朋友分享的书单

我的书单

写作锦囊

必读 30 篇

讨论库

这些讨论是怎么挑出来的

从全量里捞

只留真正讲写作的

提炼可用的要点

为人类水平的句子纠错铺路

导读

可以直接用的写作要点

HN 评论精选

同类还值得读

我是肯尼亚人：不是我把 ChatGPT 写成了我，而是它写起来像我

使用 AI 助手完成论文写作任务会累积认知债务

学生使用 ChatGPT 写作指南

Grammarly，你也背叛了我？

用 GPT-3 进行创意小说写作

打赏支持

关注向阳乔木