Japanese wakati / tokenizer — kuromoji morphological analysis with POS tags

What ends up in a morphological analyser

Wakati tokenisation and morphological analysis are preprocessing steps — which means the text being processed is usually real production content. Verifying the tokenisation of a corpus before building an Elasticsearch index, checking how kuromoji splits sentences in a product search dataset, preprocessing a legal document collection for BERT fine-tuning, or confirming that a news archive tokenises correctly before word2vec training — these are the actual workloads. The text is not synthetic; it is business-sensitive or otherwise confidential.

The ‘content words only’ and ‘nouns only’ filter modes are particularly revealing: they distil a document into its semantically significant terms, effectively summarising it. Running that extraction on an unpublished article or an internal legal filing and sending the result to an external service is an indirect disclosure of the document’s content even before the full text is shared.

The risk of cloud NLP APIs for document preprocessing

Cloud NLP APIs — Yahoo! Text Analysis API, AWS Comprehend, Google Natural Language API — receive the full text of every document as a POST body. Their access logs contain that text. Many APIs’ terms of service permit using submitted data to improve the model or retrain the underlying system, which means your documents can become training data without your explicit consent.

Information security policies at many organisations explicitly prohibit submitting internal documents to external NLP services. Yet the ‘quick check’ use case — confirming how a tokeniser handles a particular sentence — often slips under that policy because it feels like a reference lookup rather than a data submission.

kuromoji IPADIC (~390k entries) running entirely in the browser

This tool runs kuromoji.js (Apache 2.0) with IPADIC (~390k dictionary entries, ~12 MB) inside the browser. Tokenisation, POS tagging, conjugation form extraction, base-form normalisation, and reading assignment all execute in the browser’s JavaScript engine. The filter operations (nouns only, content words, deduplicate, base form) are applied to the local result array. CSV export is generated via Blob + URL.createObjectURL — a purely local browser operation.

No text is sent to any server. Open DevTools → Network while running an analysis: after the one-time dictionary download, no outbound request contains your text. The same kuromoji IPADIC that powers Elasticsearch’s official kuromoji plugin is the underlying engine here — which also means the tokenisation output matches what you will see in a production Elasticsearch index.

Using it to verify preprocessing before running production pipelines

Before committing a tokenisation strategy for a production search index or an NLP training corpus, run representative sample documents through this tool. Check how the analyser handles domain-specific terms, proper nouns, and compound words. Verify that the ‘content words only’ filter leaves the expected vocabulary. Confirm that ‘base form normalisation’ handles the inflected verbs in your corpus correctly. All of that verification happens locally, on real documents, without those documents leaving your browser.

Once you are satisfied with the behaviour, implement the same kuromoji IPADIC configuration in your application stack. The browser-local preview and the production Elasticsearch kuromoji analyzer produce consistent output because they share the same dictionary.

IPADIC, UniDic, NEologd, Sudachi — comparing the major dictionaries

Several major Japanese morphological dictionaries exist, each with distinct design choices. IPADIC (used by this tool, ~390k entries) follows the IPA POS hierarchy (~70 fine-grained tags) and underlies Elasticsearch’s official kuromoji analyzer, making it the default for search-indexing workflows. UniDic (developed by the National Institute for Japanese Language and Linguistics, ~870k entries) uses short units that split 東京都 into 東京 + 都, suited to academic corpus analysis. mecab-ipadic-NEologd (~3.2M entries) extends IPADIC with continually-updated SNS, news, and proper-noun vocabulary — strong on neologisms like スマホ, 鬼滅の刃, ChatGPT.

Sudachi (developed by Works Applications, Apache 2.0) is a modern dictionary that offers three switchable splitting granularities A / B / C. For 東京都: A gives 東京 + 都, B gives 東京都, and C gives 東京都 as a named entity. With SudachiDict-full (~2.8M entries), multi-granularity output, and bindings in Java, Python, and Rust, it has gained adoption as a BERT tokeniser preprocessor. This tool stays with IPADIC because (1) output parity with Elasticsearch is genuinely useful for production pipelines, (2) the 12 MB dictionary fits a browser-local context, and (3) search-indexing is the most widely-deployed preprocessing use case. NEologd improves neologism coverage but pushes dictionary size beyond 100 MB — an unworkable trade-off in the browser. To extract token readings as hiragana or generate ruby-tagged HTML directly, kanji-to-hiragana and furigana-html reuse the same kuromoji pipeline.

Viterbi decoding and the inherent limits of morphological analysis

MeCab, kuromoji, and Sudachi all use the Viterbi algorithm internally. They enumerate all valid segmentations of the input — for 東京都内, candidates like 東京 + 都 + 内, 東京都 + 内, and 東京 + 都内 — and choose the path minimising the sum of dictionary cost (emission) and POS-to-POS connection cost (transition) via dynamic programming. IPADIC’s transition cost matrix has roughly 70 × 70 = 4900 entries, encoding statistical grammatical constraints like particles precede verbs and auxiliary verbs follow inflected words.

Viterbi has known limits. Unknown words (out-of-vocabulary tokens) are extracted heuristically by character class (katakana run, kanji run, symbol run), so proper nouns and technical terms are frequently mis-split. Homographs like 東風 (こち vs. とうふう) and 生家 (せいか vs. しょうか) require context-sensitive resolution that Viterbi cannot perform alone — BERT or RoBERTa style contextual models are needed. Long-distance dependencies — subject and verb separated by intervening clauses — also exceed Viterbi’s locally-optimal search; sentences like 京都に住んでいる友人が東京に来た resist a clean morphological-only parse. Production NLP pipelines often combine kuromoji output with Universal Dependencies parsing (GiNZA, spaCy Japanese models) or Transformer models (Tohoku BERT, Waseda RoBERTa) to handle these cases. The wakati output here is a general-purpose preprocessing input that feeds into any of those upstream layers.