A missing traffic record and a valid interval containing no vehicles are different facts. A useful data pipeline preserves that distinction from the roadside sensor to the dashboard and archive. Quality flags, observation time and event identity are part of the measurement, not optional metadata.
Table of Contents
What belongs in the data contract?
Specify the measured value, units, lane, interval start and end, valid observation duration, quality state and configuration version. Preserve event time separately from arrival time.
FHWA’s freeway management handbook explains that data resolution and accuracy requirements depend on the operational application. A real-time queue warning and a monthly planning report therefore need different lateness and completeness rules.
Do not assume that a successfully received packet means the sensor observed the road throughout the interval. The device may be connected while its measurement is invalid or frozen.
The traffic-sensor API example translates these semantics into event identity, observation and receipt times, versioned lanes and replay handling.
Make zero, missing and invalid distinguishable
Use an explicit state model in both the API and the display. The following example is a proposed contract, not an industry-wide code list.
| State | Value handling | Meaning for a consumer |
|---|---|---|
| Valid | Retain measured value, including zero | Observation met the agreed quality checks |
| Partial | Retain value with valid duration | Only part of the interval was observed |
| Missing | No measured value | Expected record did not arrive |
| Invalid | Preserve raw value separately | Observation failed a defined check |
| Stale | Mark age and last valid time | Current state is unknown |
| Estimated | Keep method and version | Value was inferred rather than directly observed |
Choose machine-readable codes and verify that exports preserve them. A dashboard that converts every null to zero destroys information even if the sensor’s original record was correct.
How should completeness be calculated?
Report both record completeness and valid-time completeness when they answer different questions. Receiving every interval can coexist with poor measurement coverage.
Suppose 900 one-minute records are expected and 855 arrive. Record completeness is 95%. If only 810 contain a full valid minute, full-valid-record completeness is 90%. If partial intervals contribute usable seconds, calculate valid-time completeness from those seconds and state the method.
Do not scale a partial count to a full interval without labeling the estimate and explaining the traffic assumption. Multiplying a five-minute observation during a quiet period does not recreate a missing queue discharge.
Analyze gaps by lane, time of day, weather and failure type. A high monthly average can conceal a recurring outage during the period that matters most.
Protect aggregates from late and duplicated records
Store source event time, receipt time and a stable deduplication key or documented identity rule. Network retries and device reboots should not silently add vehicles twice.
Define when an interval is provisional and when it is considered complete for each consumer. A late backfill may improve an archive while being useless for a real-time warning. Preserve both the original operational view and the revised historical aggregate where needed.
Clock changes deserve special handling. Prefer an unambiguous time representation and retain local display rules separately. If a device clock jumps, mark affected records rather than forcing them into the nearest convenient interval.
Test the pipeline with known defects
Replay a small dataset containing a valid zero, a missing interval, duplicate records, delayed arrivals, an invalid lane ID and stale measurements. Verify the dashboard, API and exported file, not only the database.
These checks complement sensor calibration and classification validation. They test whether valid measurements survive the information chain and whether defects remain visible.
Save representative inputs, quality states and expected outputs with the integration specification. Replaying them after a software change helps reveal altered gap handling or aggregation behavior.
FAQs
Should missing traffic counts be stored as zero?
No. A zero count means a valid observation found no crossings. Missing or invalid observations require a distinct quality state.
Is a received data packet necessarily a valid measurement?
No. A packet may contain stale, incomplete, invalid or placeholder values. Measurement validity and communications availability should be assessed separately.
How should late traffic records be handled?
Retain event time and receipt time, apply a documented lateness policy, and version affected aggregates. Avoid duplicate counting during replay or backfill.
