Source: arXiv

Paper

KoAGE: Do Large Language Models Know, Learn, and Coin Korean Neologisms? — SungHo Kim, Eojin Jeon, SangKeun Lee, Korea University (arXiv 2610.19399).

What it claims

Korean new words are a hard test because they are not just slang dumps. They abbreviate, cross native / Sino-Korean / Western strata, and sometimes respell Hangul jamo for a visual joke. The authors’ examples in the introduction: 쉬었음 청년 (economically inactive youth), 갑분싸 (sudden awkward silence, one syllable from each of three words), 네넴띤 (a visual reading of stylized 짜파게티 lettering).

NeoTrace is their dated collector: 3.9 million candidate forms from a 12.8 million-document mix plus search snippets, filtered to a lexicon of 4,271 dated neologisms across genres — not newspapers only. KoAGE sits on that lexicon: three acquisition and two generation tasks, 5,930 test instances, with pre- and post-cutoff splits per model. They evaluate 21 models with cutoffs from 2021 to 2026.

The split that matters: lexical recall ≠ contextual inference. In a five-model temporal slice, models define only 13–30 percent of cutoff-year words, yet can often infer meaning from one usage. Even a correct definition barely helps formation analysis of pre-cutoff items. Generation: a single model given building materials often follows human formation mechanisms; multi-agent coinage favors abbreviation and depends on how the interaction is framed.

The breakdown

KoAGE is trying to stop a lazy eval habit: “the model knows Korean because it can riff on 갑분싸.” Knowing, learning-from-one-example, and coining are three jobs.

Know is the dictionary probe — and it fails on words from the model’s own cutoff year 13–30 percent of the time in that five-model cut. Learn is the in-context probe — stronger, which is why a chatbot can sound native after you paste a tweet. Structure is the morphology probe — still weak even when the gloss is right. Coin is the generation probe, including a community-reconstructed multi-agent setting the authors say is new relative to prior Korean neologism benches.

They position this against KoNeoBench, which they say is news-heavy, not cutoff-split, and does not test coinage. English benches (NEO-BENCH, SLANG, NewTerm) are the other foil: wrong language, often no generation.

Stay inside these counts. Open the PDF for the six distilled findings and per-model tables.

Why readers outside the lab should care

Korea social, brand, and community teams using LLMs to “keep up with slang” will get fluent wrong etymologies. Localization of a global model into Korean will look fine on old news and fail on messenger and comment-section coinages — the genres NeoTrace added on purpose. Agent products that invent nicknames in a group chat will over-abbreviate if you let several agents riff.

If you learn Korean from a chatbot, treat its slang lectures as guesses from one example, not as a dictionary.

What builders and Korea-touching teams should watch

• Do date your slang eval. A cutoff-blind quiz congratulates memorization.
• Don’t equate “it defined the word after I pasted a sentence” with “it knows the formation.” The paper says those come apart.
• Expect multi-agent naming sessions to collapse into abbreviations. Change the interaction frame if you need a different mechanism.
• Re-check brand-name coinages against attested community forms; KoAGE’s generation tasks exist because model coinages diverge.
• If you scrape Korean web for training, remember NeoTrace’s point: news-only neologism sets miss where formation is most productive.

Context

Read this as a cutoff-aware Korean slang-and-coinage exam, not as proof that models “can’t do Korean” and not as a meme glossary. Korelay’s frame: a model can ride one example and still be blind to how the word was built — which is exactly when it will teach your intern a fake etymology.

Source

Primary: arXiv:2610.19399 (abstract, intro counts, and task split as read from the HTML). Open the OA PDF for full results. Do not republish the PDF.