Module agents.tokenize.patterns

Functions

def detect_script(sample: str, *, default: str = 'en') ‑> str

Expand source code

def detect_script(sample: str, *, default: str = "en") -> str:
    """Return an ISO 639-1 hint based on the first non-Latin script found.

    Looks at the sample string for characters from each supported script in
    priority order. Falls back to ``default`` when only Latin characters
    (or whitespace / digits / punctuation) are present.

    Args:
        sample: Text to inspect. The first ~200 characters are sufficient.
        default: Language code returned when no non-Latin script is detected.

    Returns:
        An ISO 639-1 code. The hint is used for abbreviation selection and
        locale-specific disambiguation (e.g. Greek ``;`` handling); it does
        not gate which strong terminators are recognised.
    """
    if not sample:
        return default
    scores: dict[str, int] = {}
    for code, pattern in _SCRIPT_RANGES:
        matches = pattern.findall(sample)
        if matches:
            scores[code] = len(matches)
    if not scores:
        return default

    if "ja" in scores:
        return "ja"
    if "ko" in scores:
        return "ko"
    return max(scores.items(), key=lambda kv: kv[1])[0]

Return an ISO 639-1 hint based on the first non-Latin script found.

Looks at the sample string for characters from each supported script in priority order. Falls back to default when only Latin characters (or whitespace / digits / punctuation) are present.

Args

sample

Text to inspect. The first ~200 characters are sufficient.

default

Language code returned when no non-Latin script is detected.

Returns

An ISO 639-1 code. The hint is used for abbreviation selection and locale-specific disambiguation (e.g. Greek ; handling); it does not gate which strong terminators are recognised.

def normalize_lang_code(code: str | None) ‑> str | None

Expand source code

def normalize_lang_code(code: str | None) -> str | None:
    """Reduce an STT/TTS language tag to a bare ISO 639-1 code, or ``None``.
    """
    if not code:
        return None
    base = code.split("-", 1)[0].split("_", 1)[0].strip().lower()
    if not base or base == "auto":
        return None
    return base

Reduce an STT/TTS language tag to a bare ISO 639-1 code, or None.