Data Manager Dashboard

Data Manager Dashboard

Purpose

The data-manager dashboard provides scope-based operational visibility into ODK submissions before coder allocation.

Route:

  • /data-management

Access Model

Dashboard visibility is grant-scoped.

Supported scope behavior:

  • project grants see all submissions/forms in the project
  • project-site grants see only that project/site pair

The dashboard and sync actions both enforce scope derived from current user grants.

What The Dashboard Shows

Current dashboard sections include:

  • summary KPI cards
  • project-site submission chart
  • coder daily statistics table
  • sync action modal and preview
  • recent sync results
  • browse/search filters
  • submission table with ODK, workflow, SmartVA, and attachment visibility
  • export dropdown for filtered CSV exports

The export dropdown now also includes a coded COD snapshot CSV:

  • /api/v1/data-management/submissions/export-coded-cod-snapshot.csv

This export is one row per submission and is backed by va_submission_cod_snapshot_mv.

The coded COD snapshot export currently includes:

  • coder, reviewer, authoritative, and SmartVA COD fields, each with its WHO_2022_VA_2026 bucket section, bucket and *_who_bucket_provenance (icd10, icd11_native, unmapped); ICD-11-coded deaths are bucketed through the scheme's ICD-11 rows
  • active payload unique_id and survey_block fields when present
  • individual NQA question columns alongside score/rating
  • Social Autopsy analysis summary fields
  • raw Social Autopsy questionnaire payload fields (sa*) from the active submission payload

Daily Operations Grid

C-01 is computed live for every day (owner decision 2026-10-27, digitva-0hul, step 2): va_daily_kpi_aggregates is not read, written or refreshed, and the flask kpi commands and the duplicate-change recount are gone. compute_c01 in app/routes/api/dm_kpi/dm_kpi_grid.py runs three queries whatever the window: created submissions (Total, New), workflow events (Coded, Not analysable, Not Codeable) and bucket_flow_by_day (app/routes/api/dm_kpi/dm_kpi_scope.py) for Pending. Every day of the window has a row. Days follow the viewer's timezone (DmScope.tz, default Asia/Kolkata); coding gates stay on UTC.

  • Total: cumulative, the scope's count now minus the submissions created after the day.
  • New: submissions created that day (ODK and web).
  • Updated in ODK: 0 for now; its per-scope source is not defined.
  • Coded: non-demo coder_finalized / recode_finalized events (NOT_DEMO_OUTPUT_SQL).
  • Pending: the 6 pending states at the end of the day, the current count minus the net crossings of every later event (previous_state / current_state).
  • Not analysable / Not Codeable: per-day event counts, shared as EVENT_CONSENT_REFUSED_SQL / EVENT_NOT_CODEABLE_SQL in app/services/workflow/definition.py.

bucket_flow_by_day also serves C-04 (days_to_clear = pending ÷ the 28-day mean outflow, and the end-of-day series behind pipeline/backlog-trend) and C-19 (inflow and outflow across the pending boundary, net_28d, does_not_clear). See Data Manager KPI Framework.

Core Panels And Registered KPIs

Target state (owner decision 2026-10-27, digitva-r5s0; implemented in steps 2 to 6 of that bead, so the dashboard still shows the earlier card set until then). The dashboard keeps 18 core panels in fixed positions and makes the other 35 KPIs (registered) available by id, with no fixed panel. The definitions and the classification of each are in Data Manager KPI Framework.

  • Grouping: results-chain stage (input, process, output, outcome), then category.
  • Core panels: coder availability (C-21), language gap (C-15), uncoded and its forecast (C-04), inflow against outflow (C-19), blockers (C-23), aging (C-07), time to code (C-08), the critical path (D-WF-02), ODK sync (C-03), the daily grid (C-01), final-cause coverage (AD-FCOD), reported-death coverage (AD-COV, death-register projects only), not-codeable (C-05), not-analysable (C-06), unknown-cause share (O-UNK), coder-reviewer agreement (D-QG-09) and the two equity panels (EQ-01, EQ-02).
  • Analytic views: blockers (count, age and owner per reason), the critical path (dominant stage and bottleneck), a forecast (days to clear, range, "does not clear" flag, 12-week backtest) and a trend (period change, same period last year, rising, falling or flat).
  • Endpoints stay. Trimming the panels does not delete any endpoint; every KPI endpoint keeps serving the registry and MCP.
  • Registry and MCP: the machine-readable list is app/data/kpi_registry.json; the help page at /help/kpis renders from it; the kpi search kind finds a KPI by name; and DMs and admins can query aggregates over POST /api/v1/mcp with a personal token (see Access Control Model).

Summary Cards

The former Consent Refused card is Not analysable (2026-10-01): refused, respondent unavailable and partially completed interviews in the consent_refused workflow state (ODK consent = no counts as refused), with a per-reason count under the number, read from the active payload's interview_outcome in get_dm_kpi_from_mv. See Data Manager KPI Framework, C-06.

Current cards include:

  • Submissions
  • Pending Coding
  • SmartVA Queue
  • Coded
  • Flagged Not Codeable
  • ODK - Has Issues
  • SmartVA Missing
  • Missing in ODK

SmartVA Missing semantics:

  • counts submissions with no active SmartVA output
  • includes consent_refused workflow rows when the dashboard is filtered to smartva=missing, so KPI cards stay aligned with the table filter results

Missing in ODK semantics:

  • counts submissions retired from ODK, i.e. va_sync_issue_code = 'missing_in_odk' (ODK Retired Submissions Policy)
  • it is the only card that counts them: every other card, the table, the charts and the exports exclude them by default
  • clicking it filters the table to odk_sync=missing_in_odk

Some card counts apply filters directly to the table when clicked. Coded counts DM_CODED_STATES (not finalized_upstream_changed) and Not Codeable (DM) counts not_codeable_by_data_manager only, exactly what workflow=coded and workflow=not_codeable_by_data_manager list (digitva-l945).

Current Pending Coding semantics (owner decision 2026-10-27, Data Manager KPI Framework):

  • Pending Coding is the 6 PENDING_STATES: screening_pending, attachment_sync_pending, smartva_pending, ready_for_coding, coding_in_progress, coder_step1_saved
  • the card and its list filter workflow=pending_coding (page, exports and build_dm_mv_filter_conditions) use the same constant, so the card equals the list total
  • only ready_for_coding is handed to a coder now (CODER_READY_POOL_STATES); there is no separate list filter for it

Submission Table

The dashboard shell does not preload the submission table server-side.

Current load path:

  • the HTML route renders KPI cards, chart containers, filters, and grid shell
  • the top-level dashboard template is now a thin shell that composes partials for the page header, KPI cards, charts, table shell, offcanvas panels, sync modal, and workflow guide
  • dashboard-specific styles are served from a cacheable static CSS file instead of a large inline <style> block
  • the main dashboard bootstrap code is served from static/js/data_manager_dashboard.js instead of a large inline <script> block
  • the Mermaid workflow guide is served from static/js/data_manager_workflow_guide.js and only loads Mermaid when the guide is opened
  • the table rows load lazily through the paginated API
  • AG Grid uses the infinite row model and fetches rows from /api/v1/data-management/submissions
  • the submissions API also takes visit_note=1 (only cases whose active payload has a visit note) and org_unit=<uuid> (that unit's subtree, within the viewer's scope), both sent by the area dashboard's Visit notes link; each row carries visit_note (date, address, remarks; date only for a viewer should_redact_pii redacts), read in one query per page. the KPI cards, the coder daily statistics and every CSV export take both through the same builders as the list (_dm_submission_query_parts for the list and exports, build_dm_mv_filter_conditions for the cards, one org_unit_subtree_condition), and the filter drawer has an Org unit picker fed by filter-options org_units (only units the viewer's grants reach that hold or contain a listable case). The date_from/date_to filters are the submission date and are labelled so. death_from/death_to (Date of death from/to, inclusive ISO days) filter the list, the KPI cards and every CSV export through the same builders (death_date_conditions). The date of death is the case's va_death_register.date_of_death (confirmed duplicates and cancelled cases ignored). An ODK submission without a case, or whose case has no date, is not matched: while either bound is set only submissions with a known date of death are shown, and the filter tooltip says so. It uses ix_va_death_register_va_sid; no new index.
  • the submissions API avoids a full COUNT(*) on each page request by over-fetching one row to detect the end of the result set
  • the page opts out of the shared DataTables, jQuery, Select2, Cropper, and HTMX assets from the VA base template
  • Chart.js is loaded only when deferred chart rendering begins
  • KPI counts load immediately after the shell is ready
  • filter option hydration runs separately and does not block the initial table shell
  • non-critical charts are deferred until after the first paint / idle window

The table currently shows scoped submissions with:

  • masked ID
  • project/site
  • submitted date (the calendar day in the viewer's own timezone, not UTC)
  • workflow state
  • coded-on / coded-by for coded submissions
  • ODK review state
  • ODK sync issue state
  • SmartVA status (Available, Failed, Missing, Ineligible)
  • attachment count
  • optional columns such as project, collector, and flagged-at timestamp

The table also includes an Edit action that resolves a server-side ODK Central submission redirect and sends the user to the returned ODK submission page URL.

For finalized_upstream_changed rows, the current actions also include:

  • View Changes, which opens a modal backed by the normalized upstream-change details API

The modal now presents:

  • Data Changes
  • Metadata Changes
  • Formatting-Only Changes

Current action placement:

  • AG Grid shows View and View Changes
  • Accept And Recode and Keep Current ICD Decision are modal-only actions
  • the data-manager detail page also uses a modal-only View Changes flow

Current action semantics:

  • Accept And Recode uses the new ODK data, clears old assigned ICD codes, and returns the form for recoding
  • Keep Current ICD Decision adopts the latest ODK payload locally while preserving the current finalized ICD Code decision and finalized workflow state

Both actions now open a second confirmation modal before execution.

Filters

Current filter behavior includes:

  • project
  • site
  • submitted from/to
  • ODK status
  • SmartVA
  • ODK sync
  • workflow state
  • free-text search
  • optional column toggles

SmartVA status semantics in the dashboard table:

  • Available: an active SmartVA projection row exists and its outcome is not failed
  • Failed: an active SmartVA projection row exists with va_smartva_outcome = 'failed'
  • Ineligible: workflow state is consent_refused (SmartVA not required)
  • Missing: no active SmartVA projection row exists for the submission

ODK sync filter semantics:

  • In sync (default) — the default. Submissions retired from ODK are hidden from the table, the summary cards, the charts and the exports
  • Missing in ODK — only the retired submissions
  • All — both, with no condition on sync state

The default is part of the saved filter state and of the dashboard URL (odk_sync=in_sync), and clearing the filters restores it rather than blanking it. The read-only submission view stays reachable for retired submissions.

Filter state persists across page reloads in the browser.

Coder Daily Statistics

The main /data-management dashboard now includes a coder-by-day output table.

Current semantics:

  • rows are coder names derived from workflow events
  • columns cover the last 7 local calendar days, inclusive of today
  • Total is the sum of those 7 daily columns
  • Cumulative is the all-time first-pass coding total for the currently filtered submission scope
  • only coder_finalized events are counted
  • recode_finalized events are explicitly excluded

Current filter behavior:

  • the same dashboard filters apply to this table as to the main grid and KPI cards
  • this includes project, site, submitted from/to, ODK status, SmartVA status, age group, gender, ODK sync, workflow state, and text search
  • date grouping uses the current user's timezone
  • the table's submitted date, its date sort and the submitted from/to filters all use the viewer's local calendar day (user.timezone, default Asia/Kolkata); from/to are converted to UTC instants (>= local start, < next local day) so the plain column is compared. The KPI cards and charts read the analytics materialized views and still bucket by UTC day, so a late-evening case can count on a different day there than in the table
  • the full-submissions CSV writes va_submission_date as local wall time (YYYY-MM-DD HH:MM:SS); the timezone is part of the data-management cache key and of the export cache identity

Exports

The dashboard header includes an Export dropdown.

Current export actions:

  • Export Data
  • SmartVA Input Data
  • Results
  • Likelihoods

Current export behavior:

  • all exports reuse the current dashboard filters and sort order
  • CSVs are written UTF-8 with BOM for Excel compatibility — the BOM is part of the stored object, so both stores deliver the same bytes
  • Export Data includes current workflow/coding state and the filtered ODK payload data
  • SmartVA Input Data exports the cleaned SmartVA preparation input shape for the active payload
  • Results exports active SmartVA summary result fields only
  • Likelihoods exports raw active SmartVA likelihood rows only

Current PII handling:

  • exports omit payload fields marked is_pii=true in field mapping
  • exports also omit selected local identifier columns such as va_uniqueid_real, va_instance_name, and va_data_collector
  • the main data export retains non-PII narrative text such as Id10476
  • SmartVA exports do not include the full raw va_data payload

Where an export is stored

An export is derived data: a data manager can always ask for it again. So it is not kept on the app server any longer than it has to be. app/services/export_store_service.py owns the whole lifecycle over the existing attachment_store — there is no second S3 client:

ATTACHMENT_STORE=s3 ATTACHMENT_STORE=local
Written to exports/<kind>/<user_id>/<UTC ts>_<filter hash>.csv in the DigitVA bucket, SSE-S3, Cache-Control: private, no-store the same key layout under APP_DATA/exports/
Delivered as 302 to a presigned attachment GET, lifetime ATTACHMENT_PRESIGN_EXPIRY_SECONDS the stored file, served by the app
On the VM nothing — the CSV goes out through one bounded temp file that is removed in a finally the export file

The response to the export endpoint carries Cache-Control: private, no-store and X-Export-Cache: HIT / MISS / BYPASS on both stores. The dashboard button is a plain window.location.assign, so the browser follows the redirect and the download starts from the bucket; no JavaScript change was needed.

BYPASS means the store could not be written: the CSV is already in hand, so it is sent inline and only the caching is lost.

Export memory

No export is ever one in-memory string. The route writes the CSV into a scratch file under DB_BACKUP_TMP_DIR (else the system temp dir) and hands that file to write_export(csv_file=...), which copies it to the store in 64 KB chunks; the BYPASS response streams the same file. The two payload-bearing exports (the main data export and the SmartVA input export) are write_dm_submissions_export_csv / write_dm_smartva_input_export_csv: one query through a server-side cursor (yield_per=500, planned with SET LOCAL cursor_tuple_fraction = 1.0 so the unordered result keeps the order the buffered query gave), each row's finished cells and filtered payload pickled into a second scratch file, then replayed once the payload column union and the org-unit labels are known. Memory is flat in the row count; the dm_*_export_csv string functions are thin wrappers for tests.

The export cache

The stored object is the cache. Its name carries the sha256 of the export kind, the user and every filter value, so a repeat request for the same export is answered by one bounded listing of exports/<kind>/<user_id>/ rather than by recomputing the query — exactly the DM_EXPORT_CACHE_TTL_SECONDS (900 s) semantics the old APP_DATA/exports/cache/ directory had, with no second store to keep in step. The user_id is part of the key, so a lookup can only ever reach the requesting user's own exports.

The legacy APP_DATA/exports/cache/ directory is no longer written or read. It holds nothing but expired derived data and can be deleted on any deployment.

Retention

EXPORT_RETENTION_HOURS (default 24) — an export older than that is deleted from whichever store holds it. The prune runs nightly inside run_db_backup (see runtime-and-operations.md), which is the app's one daily housekeeping slot; it runs whether or not the dump succeeded, because exports must not pile up on a VM whose disk is the scarce resource. Deletion is the point here: unlike attachments, backups and SmartVA runs, nothing under exports/ is an archive.

Sync UX

The dashboard supports scoped sync initiation through a modal.

Current sync behavior:

  • project/site scope selection inside the modal
  • explicit matched-form confirmation
  • live preview of local versus ODK counts
  • warning that missing ODK records are flagged locally, not deleted
  • browser-side queueing of one per-form sync task per matched form through Celery

Current sync modal wording reflects queueing rather than completion:

  • the primary action queues the matched form sync tasks
  • success feedback reports queued form sync tasks, not finished sync completion

Supported actions:

  • per-form sync

The submission-refresh endpoint still exists for API/programmatic use, but the AG Grid row-level Refresh action is no longer shown in /data-management.

ODK Visibility

The dashboard mirrors key ODK metadata locally so data managers do not need to use ODK Central for routine triage.

Current locally exposed items include:

  • ODK review state
  • ODK review comments
  • local sync-issue markers such as missing_in_odk

Current triage rendering also heals one stale workflow edge case:

  • if a submission is still recorded as smartva_pending but an active SmartVA result already exists, the route reconciles the canonical workflow row before rendering and treats the submission as ready for coding
  • locally tracked attachment counts
  • SmartVA summary details inside the Data Triage panel on /data-management/view/<va_sid>
    • includes outcome, result-for, age/gender, causes 1-3 with ICD/likelihood, key symptoms, all-symptoms text, generated-at timestamp, run-id, and failure stage/detail when outcome is failed
  • normalized changed-field details for pending protected upstream updates via:
    • /api/v1/data-management/submissions/<va_sid>/upstream-change-details
    • data-manager detail-page modal View Changes
    • dashboard modal View Changes

Charts

The dashboard includes a compact project-site submissions chart rendered with a local vendored copy of Chart.js.

Current chart API:

  • /api/v1/data-management/project-site-submissions

This chart currently uses a route-level scoped query. It has not yet been migrated to the submission analytics materialized view.

The workflow distribution donut and workflow guide both show actual local workflow states, including pre-coding pipeline states such as:

  • screening_pending
  • attachment_sync_pending
  • smartva_pending
  • ready_for_coding

Clicking a segment filters the dashboard table to that exact workflow state.

Recent Sync Results

The dashboard includes a collapsed-by-default recent sync results panel.

It shows recent data-manager initiated sync runs with:

  • a resolved form target label even when the underlying run was a single-submission refresh
  • start time in the user timezone
  • run status
  • records added/updated
  • latest progress/error entries

User and Grant Management

The data-manager dashboard includes a "Manage Users" button in the page header that links to /data-management/users.

This page provides a separate interface (outside the admin panel) where data-managers and admins can:

  • Users tab: search users, create new users (email + confirm email, name, phone, VA language selection, and immediate initial grant selection with project-first scope), and open per-user details with:
    • status and email-verification visibility
    • user language visibility and update
    • in-scope active grant breakdown (project and project-site)
    • verification resend
    • creator-scoped email update
  • Grants tab: view, create, and toggle coder, coding-tester, and data-manager grants scoped to the data-manager's own grant scope. When a user is selected in the new-grant section, the grants table is filtered to that selected user's current grants; clearing the selected user restores the broader filtered table. The user-details modal also includes a Manage Grants shortcut that switches to the grants tab with that user's email prefilled in the grants search filter and the same user preselected in the new-grant form.

Scope rules:

  • project-scoped data-managers can assign grants at project or project-site level within their project
  • site-scoped data-managers can only assign grants at project-site level for their specific sites
  • admins bypass all scope restrictions through this interface

Assignable roles are limited to coder, coding_tester, and data_manager. The interface does not expose admin, project-PI, site-PI, reviewer, or collaborator grants.

See dm-user-grant-management.md for the full policy baseline.

Unrouted Submissions Queue

For projects with an organization tree, submissions are attributed to a unit at sync time from the payload's org_<level_code>_code fields. Ones that did not resolve to their own unit are listed here so a data manager can assign them.

  • GET /api/v1/data-management/submissions/unrouted — submissions in the manager's granted scope that are unrouted or sitting on their ODK mapping's fallback unit. include=unrouted narrows it to the ones with no unit at all; project=<id> filters by project. Capped at 200 rows, with truncated set when more exist. Projects without a tree never appear.
  • POST /api/v1/data-management/submissions/<va_sid>/org-unit with {"org_unit_id": "<uuid>"} pins the submission to that unit; the pin sets org_unit_resolution = 'manual' and later syncs will not move it. Posting {"org_unit_id": null} clears the pin and hands the submission back to routing.
  • Both require a data_manager grant covering the submission's project-site, and the unit must belong to that submission's project and be active. Pins are written to the submission audit log.

Policy: docs/policy/organization-model.md.

Outcome and critical-path KPIs

The /api/v1/analytics/dm-kpi/outcomes/* endpoints (O-UNK, D-QG-09, EQ-01, EQ-02, AD-FCOD, AD-COV) and /critical-path/ (D-WF-02) are API-only so far; the dashboard does not render them. coders/disagreement now returns the D-QG-09 bucket comparison. Routes are listed in api-v1.md.

Main route and query logic:

Template and client-side behavior:

Background sync tasks: