Module agents.tokenize.hyphenate

English word hyphenation via Frank Liang's algorithm.

Functions

def hyphenate_english(word: str) ‑> list[str]

Expand source code

def hyphenate_english(word: str) -> list[str]:
    """Split ``word`` at every legal English hyphenation point.

    Args:
        word: An English word. Short words (≤4 chars) and non-alphabetic
            tokens are returned as a single-element list.

    Returns:
        A list of substrings whose concatenation equals ``word``. Boundaries
        between consecutive pieces are valid hyphenation points.

    Examples:
        >>> hyphenate_english("hyphenation")
        ['hy', 'phen', 'ation']
        >>> hyphenate_english("the")
        ['the']
        >>> hyphenate_english("associate")
        ['as', 'so', 'ciate']
    """
    return _get_hyphenator().hyphenate(word)

Split word at every legal English hyphenation point.

Args

word

An English word. Short words (≤4 chars) and non-alphabetic tokens are returned as a single-element list.

Returns

A list of substrings whose concatenation equals word. Boundaries between consecutive pieces are valid hyphenation points.

Examples

>>> hyphenate_english("hyphenation")
['hy', 'phen', 'ation']
>>> hyphenate_english("the")
['the']
>>> hyphenate_english("associate")
['as', 'so', 'ciate']

Classes

class EnglishHyphenator (patterns: str, exceptions: str = '')

Expand source code

class EnglishHyphenator:
    """Trie-based hyphenator using Liang's algorithm.

    Instantiate via :func:`_get_hyphenator` (cached), or call
    :func:`hyphenate_english` for one-shot use.
    """

    def __init__(self, patterns: str, exceptions: str = "") -> None:
        self.tree: dict[Any, Any] = {}
        for pattern in patterns.split():
            self._insert_pattern(pattern)

        self.exceptions: dict[str, list[int]] = {}
        for ex in exceptions.split():
            points = [0] + [int(h == "-") for h in re.split(r"[a-z]", ex)]
            self.exceptions[ex.replace("-", "")] = points

    def _insert_pattern(self, pattern: str) -> None:
        chars = re.sub(r"[0-9]", "", pattern)
        points = [int(d or 0) for d in re.split(r"[.a-z]", pattern)]

        t = self.tree
        for c in chars:
            if c not in t:
                t[c] = {}
            t = t[c]
        t[None] = points

    def hyphenate(self, word: str) -> list[str]:
        """Return the word split at every legal hyphenation point.

        Short words (≤ 4 chars) pass through unchanged. Words in the
        exception list use the stored split; everything else runs the
        trie walk.
        """
        if len(word) <= 4:
            return [word]

        if word.lower() in self.exceptions:
            points = self.exceptions[word.lower()]
        else:
            work = "." + word.lower() + "."
            points = [0] * (len(work) + 1)
            for i in range(len(work)):
                t = self.tree
                for c in work[i:]:
                    if c in t:
                        t = t[c]
                        if None in t:
                            p = t[None]
                            for j, p_j in enumerate(p):
                                points[i + j] = max(points[i + j], p_j)
                    else:
                        break
            # No hyphens in the first or last two chars.
            points[1] = points[2] = points[-2] = points[-3] = 0

        pieces = [""]
        for c, p in zip(word, points[2:]):
            pieces[-1] += c
            if p % 2:
                pieces.append("")
        return pieces

Trie-based hyphenator using Liang's algorithm.

Instantiate via :func:_get_hyphenator (cached), or call :func:hyphenate_english() for one-shot use.

Methods

def hyphenate(self, word: str) ‑> list[str]

Expand source code

def hyphenate(self, word: str) -> list[str]:
    """Return the word split at every legal hyphenation point.

    Short words (≤ 4 chars) pass through unchanged. Words in the
    exception list use the stored split; everything else runs the
    trie walk.
    """
    if len(word) <= 4:
        return [word]

    if word.lower() in self.exceptions:
        points = self.exceptions[word.lower()]
    else:
        work = "." + word.lower() + "."
        points = [0] * (len(work) + 1)
        for i in range(len(work)):
            t = self.tree
            for c in work[i:]:
                if c in t:
                    t = t[c]
                    if None in t:
                        p = t[None]
                        for j, p_j in enumerate(p):
                            points[i + j] = max(points[i + j], p_j)
                else:
                    break
        # No hyphens in the first or last two chars.
        points[1] = points[2] = points[-2] = points[-3] = 0

    pieces = [""]
    for c, p in zip(word, points[2:]):
        pieces[-1] += c
        if p % 2:
            pieces.append("")
    return pieces

Return the word split at every legal hyphenation point.

Short words (≤ 4 chars) pass through unchanged. Words in the exception list use the stored split; everything else runs the trie walk.