[MS] Building a Japanese LLM Evaluation Pipeline: Lessons from a Two-Day Hackathon

Introduction

A: かしこまりました。ご注文番号をお教えいただければ、配送状況をお調べいたします。 B: ご不安ですよね。ご注文番号をお教えいただければ、すぐにお調べいたします。 Which sentence do you think is more appropriate as a response from call center AI? As voice AI has evolved, expectations in call centers and customer service have shifted from traditional IVR (Interactive Voice Response) systems toward more natural and flexible conversational experiences. Services like wevnal's "BOTCHAN AI Call" require more than simple FAQ responses—they demand human operator-level dialogue quality. When selecting an LLM for customer-facing conversational features in Japanese, leaderboards often fail to serve as reliable references for specific scenarios. MT-Bench, MMLU, and similar benchmarks are English-centric, single-turn, and accuracy-focused. Conversational nuance in Japanese — appropriate keigo, emotional reading of a frustrated customer, persona consistency across a multi-turn exchange — rarely shows up in the numbers. As a result, teams end up trial-running models against in-house prompts, by hand, every time. In April 2026, we joined wevnal for a two-day hackathon to close that gap: build a benchmark and evaluation harness that is multi-provider, multi-turn, and oriented around facets of conversational quality, with the vision to contribute back to the Japanese AI community. This post walks through how it is built, why it is built that way, and what we learned along the way.

The Problem We Needed to Solve

Subjectivity without standards: No shared rubrics exist for "Naturalness" in Japanese conversation, for what makes a response feel contextually appropriate or emotionally attuned to the user's tone. Fragmented evaluation: Existing efforts covered isolated pieces—RAG components here, single-turn accuracy there—but no end-to-end pipeline tied them together. Data scarcity: High-quality Japanese conversational datasets are rare. Without ground truth or established baselines, it was hard to tell whether model changes were real improvements. Manual, slow iteration: Evaluation was largely manual and disconnected from CI/CD—too slow to systematically improve quality at the pace the product needed. Closing this gap required an automated, standardized, production-ready pipeline that could scale across hundreds of different models.

A note on what this benchmark is for

We treat the benchmark as an initial filter, not a final verdict. Its job is to narrow 50 candidate models down to the 3–5 worth deeper, per-product evaluation — which engineering teams should still own. That framing drives the design choices below: optimize for breadth and reproducibility over depth, and keep metrics swappable as the field's notion of "good Japanese conversation" evolves.

On inspiration and licensing

The first architectural decision was to keep two things strictly separate, both in code and in mental model:

The evaluation framework — pipeline code that generates model answers, orchestrates judges, aggregates scores, and renders visualizations. Knows nothing about specific questions.
The benchmark dataset — authored Japanese conversational Q&A, judge prompts, model answers, judgment outputs. Treated as queryable data by the framework.

This separation pays off in three ways: judge prompts and new facets can change without touching pipeline code; the same framework can run other benchmarks; and the licensing story stays clean — framework code inherits Apache 2.0 from FastChat, while the authored dataset, inspired by ELYZA-tasks-100, is MIT-clean and safe to publish.

The benchmark: 2-turn conversations across 3 personas, scored on 4 facets

The dataset has 39 Japanese question pairs, each with two turns (an initial query plus a follow-up), distributed evenly across three personas: customer_service, casual_friend, and senior_professional.

{
  "question_id": 101,
  "persona": "customer_service",
  "turns": [
    "あなたはアパレルECサイトのカスタマーサポート担当です。お客様から「先週注文したワンピースがまだ届きません。配送状況を教えてください」というお問い合わせを受けました。丁寧な敬語で対応してください。",
    "お客様が「追跡番号を確認したら\"配達済み\"になっているのに届いていません。どういうことですか？」と不安を示しています。引き続き対応してください。"
  ]
}

The two-turn structure is non-negotiable for what we're trying to measure. A single-turn benchmark cannot tell you whether a model restates itself unnecessarily, whether it tracks an emotional escalation (mild concern → frustration), or whether it drifts out of its persona on the follow-up. Two turns is the minimum unit of "conversation." Each conversation is scored on four semantic facets:

Brevity & Conciseness — Is the response appropriate length?
Emotional Intelligence — Does the model recognize emotional shifts and respond appropriately?
Roleplaying — Does the assigned persona hold across both turns in voice, tone, register, and vocabulary?
Fluency — Is the Japanese natural, correct, and situationally appropriate?

These four were chosen with consideration for performance and diverse use-cases in production.

Why we score conversations, not turns

An early version of the harness scored each turn independently. We swapped to whole-conversation scoring for two reasons. First, several facets — emotional intelligence and role-playing in particular — only meaningfully unfold across turns. Second, the change halves the judge API budget: a 12-model run dropped from 3,744 calls to 1,872. This enabled the benchmark to stay reproducible and re-runnable on a hackathon-scale budget.

Auditing the benchmark itself

We created a rubric-for-the-rubric to ensure the benchmark questions themselves are high-quality, uncontaminated, and linguistically authentic before they're used to evaluate any models. Mechanical checks validate schema, coverage of personas, topic domain and edge cases, and diversity in category distribution. LLM-judged checks rate a small sample for persona adherence and linguistic naturalness. Manual spot-check: Leakage hygiene is human-in-the-loop. The audit generates a worksheet extracting distinctive 10-word spans from 10% of items; the reviewer searches each span using a search engine and notes whether it appears in known public corpora. The output is a JSON + Markdown report:

{
  "timestamp": "2026-05-15T10:30:00Z",
  "checks": {
    "schema": { "status": "pass", "count": 39 },
    "coverage": {
      "status": "pass",
      "personas": {"customer_service": 13, "casual_friend": 13,
                   "senior_professional": 13}
    },
    "diversity": {
      "status": "warning",
      "category_distribution": {"customer_service": 13, "general": 26},
      "message": "High concentration in 'general'; recommend more domain specificity"
    },
    "linguistic_naturalness": {
      "status": "pass", "sample_size": 10, "avg_score": 4.3
    }
  }
}

Failures are advisory, not blocking — the audit informs human review without gating the pipeline, because some "diversity warnings" are intentional editorial choices.

Choosing what to benchmark: the model catalog

Before calling any model, we decide which models are worth calling. That decision lives in a separate model catalog. We fetch two streams:

Azure API: deployment status, lifecycle, endpoint metadata
Artificial Analysis API: capability indices, pricing

These streams are joined on model slug to create enriched records.

class CatalogModel(BaseModel):
    azure_id: str
    slug: str
    name: str
    creator: Optional[str] = None
    lifecycle_status: Optional[str] = None
    intelligence_index: Optional[float] = None
    coding_index: Optional[float] = None
    math_index: Optional[float] = None
    pricing: ModelPricing = ModelPricing()
    tokens_per_second: Optional[float] = None

    def attractiveness(self, metric: str = "intelligence") -> float | None:
        """ROI prior: AA quality index divided by AA blended price."""
        q = self.quality(metric)
        p = self.pricing.blended_price_1m
        if q is None or p is None or p == 0:
            return None
        return q / p

The attractiveness metric captures what Japanese might call コスパ (kosupa), meaning quality per yen spent — intelligence index divided by blended price per million tokens. But we don't just pick the top-N by kosupa — that would skew cheap. Instead, we use bucket-biased allocation: divide models into price quantiles, sort within each bucket by attractiveness, then allocate slots across buckets with a bias toward higher tiers. The result spans flagship, mid-tier, and budget models rather than being dominated by cheap-but-good outliers.

Content-Addressed Caching

We cache requests by SHA256 over canonical JSON — same input returns same cached output. This makes experiments reproducible, eliminates duplicate API costs, and speeds up iteration.

Cache Key Example

{
  "target_model": "openai:gpt-4o",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant..."},
    {"role": "user", "content": "Turn 1: お問い合わせです..."},
    {"role": "assistant", "content": "かしこまりました..."},
    {"role": "user", "content": "Turn 2: 追跡番号を確認..."}
  ],
  "temperature": 0.7,
  "max_tokens": 4096,
  "reasoning_effort": "low"
}
↓
SHA256: a3f4b2c1d5e6... (maps to ~/.cache/fastchat_llm_judge/a3/a3f4b2c1d5e6.json)

Aggregation: facet scores and viz

Once judgments are in, aggregation is straightforward: per-model, per-facet, per-turn, per-persona means. What the aggregated per-model JSON looks like:

{
  "model": "openai:gpt-4o",
  "overall": 8.24,
  "by_facet": {
    "brevity": 8.1, "fluency": 8.4,
    "emotional_intelligence": 8.2, "roleplaying": 8.3
  },
  "by_turn": { "1": 8.0, "2": 8.5 },
  "by_persona": {
    "customer_service": 8.5, "casual_friend": 8.0,
    "senior_professional": 8.2
  }
}

And what the leaderboard view looks like:

Rank │ Model                  │ Overall │ Brief │ Fluent │ EI    │ Role  │ $/1M  │ AA ROI
─────┼────────────────────────┼─────────┼───────┼────────┼───────┼───────┼───────┼─────
  1  │ openai:gpt-5.4-mini    │ 8.37    │ 8.2   │ 8.5    │ 8.4   │ 8.4   │ 0.30  │ 27.9
  2  │ anthropic:claude-sonnet│ 8.21    │ 8.3   │ 8.1    │ 8.3   │ 8.1   │ 0.75  │ 11.0
  3  │ azure:gpt-4o           │ 8.15    │ 8.0   │ 8.3    │ 8.2   │ 8.1   │ 0.06  │136.0
  4  │ openai:gpt-4o-mini     │ 7.92    │ 7.8   │ 8.0    │ 8.0   │ 7.9   │ 0.03  │264.0
  5  │ azure:Kimi-K2.6        │ 7.68    │ 7.5   │ 7.8    │ 7.7   │ 7.8   │ 0.05  │153.6

The right two columns — $/1M (Artificial Analysis blended price) and AA ROI (attractiveness) — come from the catalog integrated with the benchmark scores.

Headline numbers

Metric	Value
Models evaluated	24 (OpenAI, Azure, Anthropic, open-source via Hugging Face)
Questions	39 Japanese 2-turn conversational pairs
Personas	3 (customer_service, casual_friend, senior_professional)
Facets	4 (Brevity, Fluency, EI, Role-Playing)
Judge calls per run	~1,872 (conversation-level)
Visualization types	8 (radar, bar, heatmap, lollipop, diverging, parallel, Minard, scatter)
Pipeline runtime	~45 min end-to-end
Data-quality checks	6 (schema, coverage, diversity, persona, linguistic, leakage)

A four-facet, 24-model scoreboard is rendered to charts:

def render_facet_chart(pivot, bench_name, chart, style="publish",
                       output_path=None, **kwargs):
    _apply_theme(style)
    renderers = {
        "radar": _render_radar,
        "bar": _render_bar,
        "heatmap": _render_heatmap,
        "lollipop": _render_lollipop,
        "diverging": _render_diverging,    
        "parallel": _render_parallel,      
        "minard": _render_minard,          
        "scatter": plot_multi_scatter,     
    }
    fig = renderers[chart](pivot, **kwargs)
    _save(fig, output_path or _default_output_path(bench_name, chart, style),
          bench_name, chart, style)

The four core charts — radar, bar, heatmap, lollipop — are the workhorses for facet-by-facet comparison. They take the pivot table and nothing else. The four advanced overlays each answer a different question:

Diverging chart — pick one model as a reference and show all other models as deviations from it.
Parallel coordinates — render each model's scores across facets as a polyline, making trajectories and crossovers visible at a glance.
Minard "capability march" — score becomes ribbon width across persona panels, with a price strip overlaid below.
Cost-vs-performance scatter — quality on the y-axis, price on the x-axis.

Example Charts

Outcomes from the Hackathon

After two days, the team delivered:

A curated Japanese conversational benchmark dataset
A reusable evaluation pipeline with multi-turn, facet-based scoring
A 3-tier audit system
Model catalog with bucket-biased selection across price tiers
Reproducible runs via deterministic caching
Publication-ready visualization suite and CI-friendly leaderboard exports

The project established a baseline for expansion, including voice pipeline evaluation and broader multilingual scenarios.

On Collaboration

This hackathon demonstrated the power of working with technically strong engineering partners. Collaborating with wevnal's highly skilled engineers enabled us to dive into challenging technical domains and validate solutions against real-world constraints. They shaped technical direction and concrete use cases by prioritizing value delivery in production systems. Their clear goal-setting and technical depth were the main driver behind the outcome that exceeds typical prototypes—we built a foundation conscious of production, not just a technical validation. The partnership wasn't one-way delivery—it was true co-creation.

Thanks

Thanks to Mike Lazentta, Keiji Hokamura, Cathy Yeh and Parag Alurkar for their contributions, Mai Matsumoto for building customer relationships. We would also like to extend special thanks to our incredible customers, Wataru Takahashi, Harui Hatakeyama, and Loic Cunningham for their exceptional engineering excellence.
Post Updated on September 14, 2026 at 08:00AM
Thanks for reading
from devamazonaws.blogspot.com

Search This Blog

News For Dev-ops