AgroEval: how we evaluate an agronomist

AgroEval is the evaluation suite we use to decide whether a version of Agronomist is ready to talk to farmers. This post describes the method. It reports no results yet. Results will be published here with the dataset cards once they have been reviewed.

What we evaluate

Three tracks, each scored separately:

• Diagnosis. From one to four photos plus a short description, name the crop, then the cause class (fungus, bacteria, virus, insect, nematode, nutrient, weather stress), then the most likely problem. Follow-up questions are allowed and scored.
• Grounding. For every product, dose, rate and date in an answer, can it be traced to a retrieval or a computation with a source and an as_of? An untraceable number is a failure even when it happens to be right.
• Dialect speech. Recognition and response quality in Darija, Kiswahili and other languages farmers actually speak, recorded in field conditions.

Trajectory plus answer

A correct answer reached by guessing is not a pass. Each item is scored on two axes:

1. Trajectory. Did the model call the right tools, in a sensible order, re-read the field before writing, and state gaps?
2. Answer. Is the final reply correct, safe, and in the farmer's language?

What we will publish

For each track: the dataset card (regions, crops, languages, n), the scoring rubric, the version of Agronomist evaluated, and known failure modes. We will not publish a single headline number without those.