Classifying Resident Requests to Remove Duplicates and Reveal Systemic Problems

Why weak classification hides systemic problems

Every municipality that runs a public feedback channel eventually drowns in near-identical messages: broken lights on the same street, the same pothole reported every rainy week, the same noise complaint from a new resident. Without a shared way to name and count these requests, the inbox inflates with duplicates while the underlying problem stays invisible. Large 311 systems illustrate the scale. A study of New York City 311 records found 210 distinct complaint types, yet the top 20 accounted for roughly 70 percent of all requests, and eight different noise categories alone made up about 22 percent. In Toronto, open 311 data contained more than 850 unique request types, many describing the same issue under different names.

The practical effect is that staff spend effort filing and re-filing while managers cannot tell whether many messages mean one angry person, dozens of households, or one failing asset. That is why classification and duplicate handling are not bookkeeping chores. They are the precondition for seeing which complaints repeat, where they cluster, and which single failures generate cascades of requests.

Build two layers: taxonomy and request status

Treat each record as two independent layers. The first is a stable taxonomy describing what the request is about: a category and subcategory, a canonical object or address, a symptom or requested action, the channel, and the territory or district. The second layer is a status describing its relationship to other records: first request, repeat by the same author, duplicate of an open request, or new facts attached to an old case. Do not mix the two. Many systems fail because they store duplicate as if it were a subject, which destroys every later count.

A good taxonomy stays stable over years even as structures change. Toronto analysts had to harmonize request types after division renames and after the city moved from a 47-ward to a 25-ward model, mapping hundreds of legacy types that described the same work. When you build the taxonomy, choose a single canonical location reference and a single vocabulary, and add an explicit unclassified bucket so you can always see how much still escapes the scheme.

Removing duplicates without losing signal

Remove only true duplicates, meaning the same reporter, the same object or address, the same category, and a case still open within a defined window. Match on object plus category plus time, never on exact wording: residents describe the same hole as a pothole, a hazard, or the thing on Maple. Near-duplicate requests from different households about the same shared problem are not duplicates; they form a cluster, and collapsing them hides how many people are affected.

Agree on merge rules before you merge. Give every merge a human override and log how many records you combined, so the duplicate rate itself becomes a metric of interface quality and responsiveness.

• Merge exact duplicates into the original record and keep one clear audit trail.
• Cluster near-duplicates on the same object or block into an issue counter, not a merged ticket.
• Reopen a closed case only when a new report adds facts or signals that the fix failed.
• Route ambiguous pairs to a reviewer instead of auto-merging them.
• Track the merge rate and watch it drop as you improve self-service and status visibility.

Turn clean records into systemic signals

Once records are clean, the signal appears at the aggregate level. Count requests by category, territory, and time window, then compare with the previous period. A single request to fix one lamp is a task; fifty requests across three blocks about lamps that never stay lit is a systemic failure of maintenance scheduling. The same logic applies to snow removal, graffiti, or housing-code cases that recur on the same assets.

Forecasting work in Toronto shows why this matters. After harmonizing hundreds of request types, analysts could model ward-level demand, and the model explained over 90 percent of variance for the dominant forestry workstream, which represented the large majority of environment-related requests. Extreme weather produced genuine outlier spikes, for example more than fifteen hundred environment requests in a single day after a windstorm, which is a reminder to treat occasional canary events as a separate analytical category from steady chronic complaints.

Bias and data-quality traps to watch

Read the data with two cautions. First, reporting itself is uneven. A city ombudsman in Portland found that complaint-driven property-maintenance enforcement fell disproportionately on homeowners of color in gentrifying neighborhoods and that a substantial share of complaints led to no violation at all. If some groups report more or less than others, volumes are a proxy, not the truth.

Second, the underlying fields are messy. Curation studies of the New York City dataset found duplicate fields, blank values, and inconsistent street names, and removing redundant fields alone trimmed the file by a large share. Cities that publish open data often omit data dictionaries, and service-type names differ across jurisdictions, as seen when one city calls a case abandoned vehicle while another writes vehicle abandoned. Plan cleaning, validation, and reconciliation as permanent work rather than a one-time cleanup, and cross-check request volumes against objective conditions or proactive inspections.