Field Mapping System

The field mapping system controls how ODK submission data is displayed in DigitVA: labels, category groupings, choice translations, flip colours, info flags, and PII handling. All configuration is stored in the database under the mas_* table family and managed through the admin UI at /admin/?panel=/admin/panels/field-mapping.

Data Model

Six tables form the core of the system. All are scoped to a form_type_id so multiple form types can coexist independently.

mas_form_types
  └─► mas_category_order          (legacy compatibility copy)
  └─► mas_category_display_config (category labels, icons, render mode, role visibility)
  └─► mas_subcategory_order       (sub-groupings within categories)
  └─► mas_field_display_config    (per-field labels and flags)
  └─► mas_choice_mappings         (choice value → label translations)

mas_form_types

Registry of supported VA form types.

mas_category_order

Legacy compatibility table for category identity and order within a form type. Runtime and admin rendering now treat mas_category_display_config as the authoritative category source.

Unique constraint: (form_type_id, category_code).

mas_category_display_config

Category-level display metadata per form type. This is now the authoritative category table for runtime rendering and admin category management.

Unique constraint: (form_type_id, category_code).

mas_subcategory_order

Controls the display order of sub-categories within a category.

Unique constraint: (form_type_id, category_code, subcategory_code).

Important design note:

• subcategory render behavior is now configurable
• render_mode = media_gallery is currently used by the attachments renderer to show image carousels/thumbnails for sections such as medical_documents and death_documents
• subcategories left as default render as normal query/response sections, even if some individual fields happen to contain image or audio attachments

mas_field_display_config

One row per field per form type. Controls every display aspect of a field.

Unique constraint: (form_type_id, field_id).

mas_choice_mappings

One row per field-choice combination. Translates ODK coded values to human-readable labels.

Important design note:

• XLSForm choice lists are reusable by list_name, so multiple ODK fields may point to the same source choice set
• DigitVA intentionally stores and renders choice mappings per field, not as a shared list-name master table
• the effective application lookup key is (form_type_id, field_id, choice_value)
• this means identical ODK choice sets may appear duplicated across fields in DigitVA by design
• the Choices admin panel follows that application model to avoid confusion about what the renderer actually uses at runtime

Unique constraint: (form_type_id, field_id, choice_value).

Admin UI

The Field Mapping panel lives at /admin/?panel=/admin/panels/field-mapping and is restricted to admin users.

Form Type Management

The panel header shows a count of registered form types and two action buttons: Import and New Form Type.

Each registered form type is shown as a card with:

• Code, name, and description
• Stats: category count, field count, choice count
• Action buttons: Fields · Categories · Choices · Sync ODK · Duplicate · Export

Create a blank form type

New Form Type button → modal:

Route: POST /admin/api/form-types Body: { "form_type_code": "...", "form_type_name": "...", "description": "..." } Response 201: { "form_type_code": "...", "form_type_name": "..." } Response 409: { "error": "Form type already exists: ..." }

Duplicate an existing form type

Duplicate button on any card → same modal (labelled for duplicate, shows source info banner).

Route: POST /admin/api/form-types/<source_code>/duplicate Body: { "new_code": "...", "new_name": "...", "description": "..." }

The service copies all child records under a new form_type_id with fresh UUIDs:

• All MasCategoryDisplayConfig rows
• All MasSubcategoryOrder rows
• All MasFieldDisplayConfig rows (labels, flags, PII settings, ODK labels)
• All MasChoiceMappings rows

The duplicate is fully independent. Changes to it do not affect the source.

Category display metadata is duplicated as well:

• MasCategoryDisplayConfig rows are copied with fresh UUIDs
• current display_order, render_mode, labels, icons, role visibility, and always-include flags are preserved

Export a form type

Export button on any card → browser downloads form_type_<code>.json.

Route: GET /admin/api/form-types/<code>/export

Field Editing

The field edit modal is the authoritative admin UI for field-scoped choice labels.

• Field Type is shown read-only from synced ODK metadata
• choice rows are shown per field, not per shared XLSForm list_name
• each row shows the ODK choice_value and an editable DigitVA choice_label
• the saved choice_label is what coding templates render for that specific field
• editing a choice label here does not affect other fields, even if the source XLSForm reused the same choice list
• choice lookups at render time now normalize numeric JSON values like 3.0 to match string-coded choice values like 3, which is important for some SOCIAL form select fields synced from ODK

Categories Panel

The Categories subpanel is a three-panel browser:

• Categories
• Subcategories
• Fields

Current special behaviors in that panel:

• the header includes a Special Categories tooltip note
• vahealthhistorydetails / medical_history shows a contextual note because fields assigned there feed the DISEASE / CO-MORBIDITY summary grouping
• subcategory add/edit modals now expose Render Mode
• subcategories with render_mode = media_gallery are marked with a Gallery badge Response: JSON file attachment.

Export bundle format:

{
  "schema_version": 1,
  "exported_at": "2026-03-10T12:00:00+00:00",
  "form_type": {
    "form_type_code": "WHO_2022_VA",
    "form_type_name": "WHO 2022 VA",
    "form_type_description": "...",
    "base_template_path": null,
    "mapping_version": 1
  },
  "categories": [
    { "category_code": "A", "category_name": "Identification", "display_order": 1, "is_active": true }
  ],
  "category_display_configs": [
    {
      "category_code": "vademographicdetails",
      "display_label": "Demographic / Risk Factors",
      "nav_label": "Demographic / Risk Factors",
      "icon_name": "fa-user",
      "display_order": 2,
      "render_mode": "table_sections",
      "show_to_coder": true,
      "show_to_reviewer": true,
      "show_to_site_pi": true,
      "always_include": false,
      "is_default_start": true,
      "is_active": true
    }
  ],
  "subcategories": [
    { "category_code": "A", "subcategory_code": "A1", "subcategory_name": "...", "display_order": 1, "is_active": true }
  ],
  "fields": [
    {
      "field_id": "Id10007", "category_code": "A", "subcategory_code": "A1",
      "odk_label": "Date of Interview", "short_label": "Interview Date",
      "full_label": null, "summary_label": null,
      "field_type": "date", "age_group": null,
      "flip_color": false, "is_info": false, "summary_include": true,
      "is_pii": false, "pii_type": null,
      "display_order": 1, "is_active": true, "is_custom": false
    }
  ],
  "choices": [
    { "field_id": "Id10019", "choice_value": "1", "choice_label": "Male", "display_order": 1, "is_active": true }
  ]
}

The file is safe to commit to version control and share between environments.

Import a form type

Import button in the panel header → modal with file picker.

On file selection a preview strip shows the embedded code, name, and record counts.

Route: POST /admin/api/form-types/import (multipart, enctype="multipart/form-data")

Form fields:

If the code in the file already exists in the database the import fails with a 409 conflict. Use the "Import with a different code / name" toggle to resolve this.

Response 201:

{
  "form_type_code": "WHO_2022_VA_COPY",
  "form_type_name": "WHO 2022 VA Copy",
  "categories_created": 14,
  "subcategories_created": 42,
  "fields_created": 410,
  "choices_created": 1203
}

Import always creates a new form type. It does not update or merge into an existing one.

Deterministic Category Display Seed

Migration c7f1d2e3a4b5 adds mas_category_display_config and seeds the current display metadata deterministically for:

• WHO_2022_VA
• WHO_2022_VA_SOCIAL

The seeded snapshot is intentionally checked into the migration rather than derived from whatever happens to be in the target DB at runtime. As of March 12, 2026:

• WHO_2022_VA seeds 14 category display rows
• WHO_2022_VA_SOCIAL seeds 15 category display rows
• social_autopsy is seeded at display order 15 for WHO_2022_VA_SOCIAL
• vahealthhistorydetails seeds render_mode = health_history_summary
• vanarrationanddocuments seeds render_mode = attachments and always_include = true

The WHO Excel migrator now also upserts MasCategoryDisplayConfig rows from the same deterministic defaults when Who2022VaMigrator().run() is used in seed/test paths.

Current runtime use:

• CategoryRenderingService now reads mas_category_display_config for:
  • left-nav category order
  • role visibility
  • always-include behavior
  • default initial category selection
  • previous/next traversal
• coder/reviewer render data still starts from the legacy static coder mapping, but now bridges in DB-backed category mappings for role-visible categories that the static dict does not define, such as social_autopsy
• table_sections categories now render through the generic template stack:
  • category_table_sections.html
  • _table_section.html
  • _field_value.html
• health_history_summary categories now render through category_health_history_summary.html
• attachments categories now render through category_attachments.html plus attachment-specific includes:
  • _attachments_section.html
  • _attachments_workflow_panels.html
• the attachments render mode is intentionally hybrid: submission-data sections are generic, while SmartVA / QA / assessment workflow panels remain explicit inside the workflow include

Field Management

Fields button on any card → opens the field list sub-panel below the cards.

Route: GET /admin/panels/field-mapping/fields?form_type=<code>

The table shows all fields for the selected form type:

Editing a field

Pencil icon on any row → edit modal (GET /admin/panels/field-mapping/field/<code>/<field_id>).

Editable fields in the modal:

ODK Label and Field ID are shown read-only for reference. Hover tooltips on each label explain the field's purpose in the app.

On submit → POST /admin/panels/field-mapping/field/<code>/<field_id>

On success the table row updates in-place via HTMX (outerHTML swap) and the modal closes. No full-page reload is needed.

ODK Sync

Sync ODK button on any card → opens the sync sub-panel.

Route: GET /admin/panels/field-mapping/sync?form_type=<code>

Selecting the ODK source

Three cascading dropdowns load progressively:

1. ODK Connection — active connections from MasOdkConnections; auto-selects if only one
2. ODK Project — loaded from GET /admin/api/odk-connections/<id>/odk-projects
3. ODK Form — loaded from GET /admin/api/odk-connections/<id>/odk-projects/<pid>/forms

Preview Changes and Run Sync buttons are disabled until a form is selected.

Preview changes (dry run)

Route: POST /admin/panels/field-mapping/sync/preview Body: { "form_type_code", "connection_id", "odk_project_id", "odk_form_id" }

Downloads the XLSForm XLSX and computes a diff without writing to the database.

Response shows three change lists:

If no changes are pending, a "Database is already up to date" message is shown.

Running a sync

Route: POST /admin/panels/field-mapping/sync Body: same as preview.

The sync service:

1. Downloads the XLSForm XLSX from ODK Central via pyODK
2. Parses the survey sheet: reads name, type, and the first label* column
3. Parses the choices sheet: reads list_name, name, and the first label* column
4. Joins choices to fields on list_name extracted from type strings like select_one <list_name>
5. For each field in MasFieldDisplayConfig: updates odk_label if changed
6. For each choice from the XLSForm: upserts into MasChoiceMappings
   • New → insert
   • Changed label → update
   • Unchanged → skip (not counted as updated)
   • Absent from XLSForm → left untouched (sync is additive only)

Result table:

Why sync is additive-only: The WHO VA XLSForm is a universal template covering all deployment sites. A site-specific ODK project only contains choices for that site. Deactivating choices absent from one site's form would silently remove choices needed by other sites. Choices seeded from resource/mapping/mapping_choices.xlsx are the canonical source; ODK sync augments them.

Label column discovery

XLSForm label columns vary by form: label, label::English (en), label::English, etc. The sync service picks the first column whose name starts with label.

Field Mapping Service (Runtime)

At runtime, the FieldMappingService (singleton) serves field and choice data to rendering code. It caches queries per form type.

from app.services.field_mapping_service import get_mapping_service

svc = get_mapping_service()
config = svc.get_field_config(form_type_id, field_id)   # MasFieldDisplayConfig | None
choices = svc.get_choices(form_type_id, field_id)       # list[MasChoiceMappings]
label = svc.get_choice_label(form_type_id, field_id, value)  # str | None

The cache is invalidated after any field edit via svc.clear_cache().

Ordering behavior in get_fieldsitepi(form_type_code):

• categories are ordered by MasCategoryDisplayConfig.display_order
• subcategories within a category are ordered by MasSubcategoryOrder.display_order
• fields within a subcategory are ordered by MasFieldDisplayConfig.display_order

If a field-bearing subcategory does not yet have a MasSubcategoryOrder row, it still renders after the explicitly ordered subcategories for that category.

FormTypeService (Admin)

The FormTypeService singleton handles all admin operations.

from app.services.form_type_service import get_form_type_service

svc = get_form_type_service()

# Register a blank form type
ft = svc.register_form_type("MY_FORM", "My Form", description="...")

# List all active form types
types = svc.list_form_types()

# Stats for one form type
stats = svc.get_form_type_stats("WHO_2022_VA")
# {"form_type_code", "form_type_name", "form_count", "category_count", "field_count", "choice_count"}

# Duplicate (copies all categories, subcategories, fields, choices)
new_ft = svc.duplicate_form_type("WHO_2022_VA", "WHO_2022_VA_V2", "WHO 2022 VA v2")

# Export to dict (JSON-serializable)
data = svc.export_form_type("WHO_2022_VA")

# Import from dict
new_ft, stats = svc.import_form_type(data, override_code="NEW_CODE", override_name="New Name")

# Soft-delete (raises ValueError if forms are still linked)
svc.deactivate_form_type("OLD_FORM")

Bootstrap Note

If the admin field-mapping panel shows no registered form types, the default WHO 2022 form type and mapping set have not been seeded into mas_form_types and related mas_* mapping tables yet.

The standard bootstrap command is:

docker compose exec minerva_app_service uv run flask seed run

That command:

• registers WHO_2022_VA if it is missing
• runs Who2022VaMigrator
• loads field definitions from resource/mapping/mapping_labels.xlsx
• loads choice mappings from resource/mapping/mapping_choices.xlsx

Languages being present in the admin UI does not imply form types are present. The language seed and the field-mapping seed are separate steps within flask seed run.

Field Display Flags Reference

What is_pii actually does

One consumer: FieldMappingService.get_pii_field_ids() feeds data_management_service._filter_export_payload(), which drops those payload keys from the data-manager CSV export, on top of the hardcoded CSV_EXPORT_OMIT_PAYLOAD_FIELDS list. It does not mask anything on the coding screen — the coding screen renders only fields that have a non-NULL subcategory_code, so identifier questions are excluded by never being mapped rather than by being flagged.

That distinction is what makes app/services/pii_field_registry.py work. It records which payload fields carry personal data and flags them, creating a row with category_code and subcategory_code left NULL for fields that have no mapping (most identifiers, plus web intake's abha_number / abha_address). Such a row is visible to export redaction and invisible to the coding screen.

The registry is applied by migration b8e3d1f7a2c4 and re-applied by flask seed run, because the Excel mapping source has no is_pii column: flags set by hand in the admin panel do not survive a mapping reseed and are not reproducible on a fresh install. Applying it never replaces a pii_type an operator already chose — the registry decides whether a field is PII, not how it is labelled.

The PII set must be confirmed before anything is exported

Because the registry applies a WHO-keyed field list to every form type, a form type always ends up with some is_pii rows — including a questionnaire that has none of those field ids. "Has is_pii rows" therefore says nothing. The set counts as confirmed only when at least one field the form type owns is flagged: a row with a subcategory_code (mapped), an odk_label (synced from ODK), or is_custom = false. FieldMappingService.get_pii_set_status() derives this on every read; nothing is stored.

While a form type is unconfirmed the system fails closed — a viewer subject to PII redaction sees no payload on the submission page, and the submissions CSV export withholds every payload value for that form (column shape unchanged). The SmartVA input export is exempt — _filter_export_payload is called there with withhold_unconfirmed=False, so it strips the flagged fields and nothing more, because it is an admin/data_manager processing feed and withholding would make SmartVA unusable on a new questionnaire. It logs pii set unconfirmed | <form_type_code> | smartva input export not withheld once per form instead. The field-mapping panel shows a warning on that form type's card, GET /admin/api/form-types returns pii_set_confirmed: false, and flask form-types stats --code=<code> prints it. An admin clears it by flagging one owned field Is PII. Form type creation and activation are not blocked, since a new form type has no fields until ODK sync runs.

The PII set is also the one mapping cache that is not clear_cache()-dependent: it re-checks (count(*), max(updated_at)) over the form type's mas_field_display_config rows on every call, so a flag change made in another worker (admin edit, Celery-run sync) takes effect without a restart.

Labels hierarchy (first non-null wins for display):

short_label  →  full_label  →  odk_label  →  field_id

API Reference

All state-changing routes require X-CSRFToken header and admin role.

Related Files

Related Documentation

• Data Model
• ODK Sync

  - Admin and Setup

title: Field Mapping System doc_type: current-state status: active owner: engineering

last_updated: 2026-09-18