You are coding phrases pulled automatically out of English-language news reporting. Each row gives a passive subject–verb pair and the by-phrase that follows it. There are 167 rows and four judgments per row.
Already read this? Skip to the coding tool →
Judge each row on its own terms. Do not try to be consistent with any pattern you think you are seeing across the sheet, and do not try to work out what the study is about — a coder who is guessing at the answer is no longer an independent check, which is the whole reason you are doing this.
Four judgments per row. Three of them are about the by-phrase; one is about the verb. The tree shows how they fit together.
Every row has three parts. Three of your four judgments are about the by-phrase; one is about the verb. Knowing which is which makes the rest of this manual straightforward.
A second one, where the by-phrase names people:
The parser joins words with underscores and sometimes sweeps in material that does not belong to the by-phrase at all. One row's agent reads by_Putin_small_war_hawks_inner_circle_national_security_adviser_Nikolai_Patrushev… — the agent is Putin, and everything after is the sentence carrying on.
Around a fifth of the rows have this problem, and a few are much worse. Find the head noun and code that. Ignore trailing material that reads like a description, a second clause, or a list. If you genuinely cannot tell what is being named, that is what unclear is for.
The first decision is whether the by-phrase names something that brought the action about, or merely the conditions under which it happened. Only the first belongs in this study.
| Code | The by-phrase names… | Test |
|---|---|---|
| yes | an agent or actor — a person, body, object or force that brought the action about | Could you say it did something? |
| no | a condition or circumstance — a setting, measure, reason or background state | Does it only describe when, where, why or how much? |
Code no and the remaining three columns are left blank. Deciding this is part of the task — do not assume a row belongs simply because it is on the sheet.
Is the thing named in the by-phrase a person or a body of people, or is it not?
| Code | Criterion | Typical cases |
|---|---|---|
| human | An individual, a group of people, an institution, or a state — anything that acts through people | forces, police, parliament, a named person, a country |
| nonhuman | An object, substance, vehicle, or an event treated as the cause | missile, drone, shells, a chemical, an explosion |
| unclear | Genuinely cannot be decided | — |
A country counts as human. A state acts through people, so by_Russia and by_Ukraine are human.
A nationality attached to an object does not make it human. Go by the head noun.
What kind of thing is named in the by-phrase? One code per row, from this list.
| Code | Covers |
|---|---|
| weapon | Arms, munitions, ordnance, and the acts of using themby Russian shells, by a Baykar Bayraktar TB2 drone |
| military_authority | Armed forces, police, intelligence, official security bodiesby Russian forces, by police, by the Southern Military District |
| nonmilitary_party | Irregular combatants, separatists, activists, hackersby pro-Russian separatists, by a hacker cooperative |
| country_city | States, cities, regions named as suchby Russia, by Ukraine, by Washington |
| law_governance | Governments, parliaments, courts, committees, treatiesby the German parliamentary budget committee, by Congress |
| death_danger | Attacks, fighting, explosions, fire, lossby fighting, by weeks of Russian bombardment, by a strike |
| important_figure | Named individuals and office-holdersby Putin, by Shoigu, by the leaders of Ukraine |
| civilian | Ordinary people, the public, communitiesby most Ukrainians, by friends, by the public |
| location | Places and premises not covered by country_cityby the centre, by the district |
| media_tech | News outlets, broadcasters, platforms, technologyby CNN, by Ukrainska Pravda, by the Wall Street Journal |
| money_business | Firms, funds, commercial bodies, financial instrumentsby Gazprom, by private sector firms |
| commute | Transport and trafficby civilian traffic |
| nominalisation | Abstract nouns built from verbs or adjectivesby resistance, by corruption, by an inconsistency |
| neutral_entity | Anything that fits nowhere aboveby a factor, by a sound, by an object |
| unclear | Genuinely cannot be decidedgenuinely undecidable |
Some rows sit between two categories — by_Russian_forces is both a state and an armed body. Prefer the more specific code: here, military_authority. Reserve country_city for phrases where the state itself is what is named, as in by_Russia.
This is the one judgment that is not about the by-phrase. Look at the verb, and ask what it does to the referent of the subject — not whether the event seems good or bad in general.
| Code | Effect on the subject | Verbs |
|---|---|---|
| violent | Physical harm or destruction | killed, destroyed, struck, shelled, demolished, wounded |
| restrictive | Limitation or control, without direct physical harm | occupied, seized, detained, held, thwarted, controlled |
| negative | Adverse, but neither violent nor restrictive | frustrated, blamed, denounced, countered |
| neutral | No evaluative loading in this context | reported, met, taken, provided, described |
| positive | Favourable to the subject | welcomed, reclaimed, allowed, recognised, released |
| unclear | Genuinely cannot be decided | — |
Violent and restrictive are separated by physical harm. Seize, detain and occupy constrain something; shell and kill injure or destroy it. If a verb could go either way, code the reading it has in this row, not the sense the word carries most often elsewhere.
“More than 70 percent of the country’s gas reserves are held by Gazprom.” Held reads like detention, and elsewhere on this sheet it is detention — servicemen held by Russia is restrictive. Here it means owns. Nothing is being constrained; the reserves are not worse off. Code it neutral.
This is what “code the reading in this row” means in practice. The same verb takes different codes in different rows, and that is correct, not an inconsistency.
Use unclear freely. A row you could not decide is information. A row you guessed at looks identical to a row you were certain about, and quietly corrupts the comparison this exercise exists to produce.
Put the reason in the notes column. If the same kind of difficulty comes up more than three or four times, stop and raise it rather than working through it — that usually means this manual is missing a rule, and it is better fixed once than decided differently seventeen times.
Do not look up the original news articles. Code from what is on the screen. Some rows will feel thin because the parser cut them short or ran them long; that is expected, and unclear is the right answer when it leaves you unable to judge.