AI Audit — Scope & Progress Tracker

Owner: Srinidhi Scope: All APIs for fetching, updating, and auditing TCs and installations in the AI Audit section.

---

API Quick Reference

All routes are prefixed with /<serviceName>/v1 (service name from root package.json). Single route per endpoint — user type (Admin vs AE) is resolved from request.user.userType. Auth: Authorization: Bearer <sessionToken> header required on all endpoints.

Method	Route	LLD	Status
GET	/audit/summary	LLD	Done
GET	/audit/transformers	LLD	Done
GET	/audit/transformers/:id	LLD	Done
GET	/audit/transformers/:tcId/installations	LLD	Done
POST	/audit/transformers/:id/audit	LLD	Done
GET	/audit/transformers/:id/consumption-trend	LLD	Done
GET	/audit/transformers/:id/revenue-trend	LLD	Done
GET	/user/hierarchy	LLD	Done
POST	/audit/transformers/:id/ai-audit	LLD	Done
GET	/audit/transformers/:id/ai-audit/installations	LLD	Done
GET	/audit/transformers/:id/ai-audit/remarks	LLD	Done
POST	/audit/transformers/:id/freeze	LLD · Alignment LLD	Done
POST	/seed/prepare-demo-data	LLD	Not started
POST	/seed/restore-demo-data	LLD	Not started

---

Completed

1. Audit Summary — GET /audit/summary

• LLD: LLD-fetch-audit-summary.md
• Files: api/src/entities/audit/controller/fetchAuditSummary.js
• UI: Summary cards (Total Transformers, Audited, Audit Failed), Loss bucket counts (7 ranges), Failed reason counts (4 categories)
• Status: Done

2. Fetch Transformers (TC List) — GET /audit/transformers

• LLD: LLD-fetch-transformers.md
• Files: api/src/entities/audit/controller/fetchTransformers.js
• UI: TC table with columns — S.No, TC Name, Installations, TC Capacity, TC Consumption, Installation Consumption, Flags/TC Issues, Action
• Filters: Audit status (Audited/Failed), loss bucket range, failed reason, search by TC Name, pagination
• Extended columns (when filtered):
  • Audited view: Loss % (prev month, prev-to-prev month, current month with trend arrows), Audit Status, Division name, Sub-division name, Section name
  • Failed view: Remark, Division name, Sub-division name, Section name
• Status: Done

3. Fetch Transformer Detail — GET /audit/transformers/:id

• LLD: LLD-fetch-transformer-detail.md
• Files: api/src/entities/audit/controller/fetchTransformerDetail.js
• UI:
  • TC Score gauge (score/100 + rating)
  • TC Consumption, TC Capacity (KP, HP), Installation Consumption, Loss % (with previous value)
  • Revenue parameter cards (ARR, AT&C, Demand, Collection, Billing efficiency, Collection efficiency)
  • Loss Analysis cards (Unbilled, Vacant, MNR, Zero consumption, Doorlock, Abnormal, Subnormal, Bill Cancellation)
  • Loss Analysis pie chart (current month billing count breakdown) — data already in billingCount
  • "View TC Details" modal (TC Number, Name, Serial Number, Make, Capacity, TIMS Code, DTLMS, DTR, Reading Day, GPS Location, Feeder, Execution Type, Meter Make, Meter Serial, CT Ratio, Meter Constant)
  • Overloaded badge
• Status: Done

4. Fetch Installations — GET /audit/transformers/:tcId/installations

• LLD: LLD-fetch-installations.md
• Files: api/src/entities/audit/controller/fetchInstallations.js
• UI: Installation table — RR Number, Tariff, Consumer name, Sanctioned load, Consumption, Bill Amount
• Filters: Billing status tabs (Unbilled, Vacant, MNR, etc.), search by RR Number, pagination
• Status: Done

5. Audit TC (Phase 1) — POST /audit/transformers/:id/audit

• LLD: LLD-audit-tc.md
• Files: api/src/entities/audit/controller/auditTC.js, helpers/computeAuditFields.js, helpers/recomputeAuditSummaries.js
• UI: "Start Audit with AI" button on TC detail page
• Scope: Recompute audit fields for a single TC + cascade audit summaries at all 6 levels (Section → Sub-Division → Division → Circle → Zone → MESCOM)
• Status: Done

6. Supporting Infrastructure

• Auth: Login, Logout, Password Reset — Done
• Alerts: GET /alert/:month — Done
• Health: Info, Service Ping, Mongo Ping, Route Info — Done
• Models: transformer, installation, auditSummary, user, session, alert — Done
• Helpers: resolveLocationFilter, lossBucketRanges — Done
• Cerbos policies: audit resource configured — Done

6.1 Data Scripts

• Backfill Audit Status (transformer_backfillAuditStatus) — Sets auditStatus on all transformer docs based on lossPercentage presence — Done
• Fix Invalid Audited TCs (transformer_fixInvalidAuditedTCs) — LLD — Fixes TCs incorrectly marked AUDITED — Done
• Compute Audit Summary (auditSummary_computeAuditSummary) — LLD — Replaces dummy audit-summaries with real data computed from transformer collection across all 6 hierarchy levels — Done
• Script workspace LOCAL mode — Added .env loading support for local development (envVars.js, mongo.js with DATABASE_NAME) — Done
• Backfill Avg Consumption (installation_backfillAvgConsumption) — LLD — Pre-computes 6-month rolling avg consumption for remarked installations. Uses server-side aggregation ($out + $lookup + $merge) for performance. Supports excludeSections to skip seed data. — Done, ran successfully (7 months, ~15 min)
• Seed AI Audit Test Data (seed_aiAuditTestData) — LLD — Generates test data for one section (15 TCs, ~135 installations × 7 months) with inline avgConsumption6Months dummy data to exercise entire AI Audit flow — Done, ran successfully

---

7. Location Hierarchy — GET /user/hierarchy

• LLD: LLD-fetch-hierarchy.md
• Files: api/src/entities/user/controller/fetchHierarchy.js
• UI: Cascading dropdown modal — Zone (CE) → Circle (SE) → Division (EE) → Sub-Division (AEE) → Section (AE)
• Status: Done

8. Consumption Trend — GET /audit/transformers/:id/consumption-trend

• LLD: LLD-fetch-consumption-trend.md
• Files: api/src/entities/audit/controller/fetchConsumptionTrend.js
• UI: TC-Installation Consumption line chart — Yellow line (tcConsumption) vs Blue line (installationConsumption) across configurable month window (default 6 months)
• Status: Done

9. Revenue Trend — GET /audit/transformers/:id/revenue-trend

• LLD: LLD-fetch-revenue-trend.md
• Files: api/src/entities/audit/controller/fetchRevenueTrend.js
• UI: Revenue Parameters bar chart — Switchable via dropdown (ARR, AT&C, Demand, Collection, Billing efficiency, Collection efficiency) across configurable month window (default 6 months)
• Status: Done

10. AI Audit — Run (GPS + Remarks Analysis) — POST /audit/transformers/:id/ai-audit

• LLD: LLD-ai-audit-run.md
• Schemas: LLD-ai-audit-schemas.md
• Files: api/src/entities/audit/controller/aiAuditTC.js, helpers/gpsConstants.js, helpers/deriveRemarkType.js, common/src/schemas/aiAuditResult.model.js, common/src/schemas/aiAuditInstallation.model.js
• UI: "Start Audit with AI" button → GPS-based installation classification + remark analysis, stores results in staging collections
• Note: Originally used $geoNear but switched to $match + Haversine due to 9.3M doc scan timeout. Observed latency ~3.3s on Atlas.
• Status: Done

11. AI Audit — Fetch Installations + Remarks — GET /audit/transformers/:id/ai-audit/installations + GET /audit/transformers/:id/ai-audit/remarks

• LLD: LLD-ai-audit-fetch.md
• Files: api/src/entities/audit/controller/fetchAiAuditInstallations.js, api/src/entities/audit/controller/fetchAiAuditRemarks.js
• UI:
  • Table 1 (Tagging): All installations with GPS distance, within-range status, AI suggestion (TAG/UNTAG/NO_CHANGE), source TC info. Filters: ALL / CHANGES_SUGGESTED / NO_CHANGES_REQUIRED. Sort by distance ascending.
  • Table 2 (Remarks): Installations with billing remarks (MNR, unbilled, vacant, etc.) + avg consumption suggestions. Filters: ALL + 7 remark types. Sort by remarkType then rrNumber.
• Shared patterns: Reads from ai-audit-installation staging collection, $facet for filter counts, AE section validation, search by RR number
• Status: Done

12. AI Audit — Freeze — POST /audit/transformers/:id/freeze

• LLD: LLD-ai-audit-freeze.md
• Files: api/src/entities/audit/controller/freezeAiAudit.js
• UI: "Freeze" button on AI Audit review page — commits AI audit changes to real data
• Scope: Validates staging data, applies user overrides (TAG/UNTAG/useAvgConsumption), writes target TC with computed audit fields, updates real installation tagging, reaudits source TCs (skips target TC), updates adjacent month loss %, cascades audit summaries at all 6 levels, marks staging result as frozen
• Status: Done

---

Remaining

13. Phase 2 — Adjacent Month Loss % Update

• LLD: Noted in LLD-audit-tc.md Section 12
• Purpose: When a TC's lossPercentage changes for month M, update previousMonthLossPercentage on month M+1 TC doc and previousToPreviousMonthLossPercentage on month M+2 TC doc
• Trigger: After any audit/freeze that changes a TC's loss %
• Complexity: Low
• Status: Not started

---

UI Screen → API Mapping (Quick Reference)

Screen	API(s) Used	Status
AI Audit list (default)	GET /audit/summary + GET /audit/transformers	Done
AI Audit list (Audited filter)	GET /audit/summary (loss buckets) + GET /audit/transformers?auditStatus=AUDITED&lossBucket=X	Done
AI Audit list (Failed filter)	GET /audit/summary (failed reasons) + GET /audit/transformers?auditStatus=AUDIT_FAILED&failedReason=X	Done
Month picker	month query param on all endpoints	Done
Search by TC Name	search query param on GET /audit/transformers	Done
TC Detail page — cards & scores	GET /audit/transformers/:id	Done
TC Detail — Loss Analysis pie chart	GET /audit/transformers/:id → billingCount	Done
TC Detail — TC-Installation Consumption line chart	GET /audit/transformers/:id/consumption-trend	Done
TC Detail — Revenue Parameters bar chart	GET /audit/transformers/:id/revenue-trend	Done
TC Detail — View TC Details modal	GET /audit/transformers/:id	Done
TC Detail — Installation table	GET /audit/transformers/:tcId/installations	Done
TC Detail — Billing status filter tabs	GET /audit/transformers/:tcId/installations?billingStatus=X	Done
AI Audit page — Location hierarchy dropdown	GET /user/hierarchy	Done
AI Audit page — Run AI Audit	POST /audit/transformers/:id/ai-audit	Done
AI Audit page — Installation tagging	GET /audit/transformers/:id/ai-audit/installations	Done
AI Audit page — Installations with Remarks	GET /audit/transformers/:id/ai-audit/remarks	Done
AI Audit page — Freeze the result	POST /audit/transformers/:id/freeze	Done

---

Bugs & Fixes in Existing Code

B1. billingStatus field mismatch — FIXED

• Fix: Query param now maps to individual boolean flags (485b942). Stale index on billingStatus remains in installation.model.js:154 (harmless, just wasted space).
• Status: Fixed

B2. demand (Bill Amount) excluded from installations API — LOW

• File: api/src/entities/audit/controller/fetchInstallations.js
• Problem: demand is the bill amount field in the installation schema, but it's excluded from the projection (line 41: demand: 0) and missing from response serialization. UI shows "Bill Amount" column.
• Fix: Remove demand from projection exclusion, add to response serialization (as billAmount or demand).
• Status: Not started

B3. abnormal, subnormal, billCancellation never counted — HIGH

• File: api/src/entities/audit/helpers/computeAuditFields.js
• Problem: billingCount initializes all 8 categories (lines 36-45) but the loop (lines 46-52) only increments 5: unbilled, mnr, vacant, zeroConsumption, doorlock. The abnormal, subnormal, billCancellation counts are always 0. Installation schema also lacks these boolean fields.
• Fix: Add abnormal, subnormal, billCancellation booleans to installation schema (or derive from billing data in migration), then count them in computeAuditFields.
• Status: Not started

B4. untaggedInstallationCount not in schema, not computed — HIGH

• Files: common/src/schemas/transformer.model.js, api/src/entities/audit/helpers/computeAuditFields.js
• Problem: untaggedInstallationCount is expected in TC detail and list API responses, but the field doesn't exist in the transformer schema and computeAuditFields never computes it. Always returns undefined.
• Fix: Add field to transformer schema, compute in computeAuditFields by querying installations with tcId: null in the same section.
• Status: Not started

B5. Migrated data: lossPercentage: 100 when tcConsumption: 0 — HIGH

• Discovered: 2026-04-14 during AI audit of TC 3783042 for month 2025-11
• Problem: 2,022 transformer docs (across 2025-09 to 2025-12) migrated from production have lossPercentage: 100 and auditStatus: "AUDITED" despite tcConsumption: 0. Loss % is mathematically undefined when TC consumption is zero (division by zero). This causes the AI audit original snapshot to show inconsistent data (lossPercentage: 100 with tcConsumption: 0), while computed values correctly return null.
• Impact: 1,500 distinct TCs affected. Cascading: 1,160 docs have bad previousMonthLossPercentage: 100 and 824 docs have bad previousToPreviousMonthLossPercentage: 100 referencing these months.
• Breakdown by month: 2025-09: 323, 2025-10: 777, 2025-11: 496, 2025-12: 426
• Fix (4 steps):
  1. Fix 2,022 transformer docs: set lossPercentage: null, auditStatus: "AUDIT_FAILED" where tcConsumption: 0 and lossPercentage: { $ne: null }
  2. Fix cascading: for each bad (TC number, month) pair, null out previousMonthLossPercentage on next month's doc and previousToPreviousMonthLossPercentage on month-after-next's doc
  3. Fix computeAuditSummary.js aggregation: noTcConsumptionWithRemarks and noTcConsumptionNoRemarks checks (lines 340-368) only match tcConsumption: null, not 0 — update to match both
  4. Recompute all audit summaries for months 2025-09 through 2025-12
• Status: Fixed — ran on Local, DEV, STG (2026-04-14)

B7. AI Audit freeze — frontend/backend loss% divergence — HIGH

• Files: api/src/entities/audit/controller/fetchAiAuditInstallations.js, api/src/entities/audit/controller/fetchAiAuditRemarks.js, api/src/entities/audit/controller/freezeAiAudit.js, api/src/entities/audit/controller/aiAuditTC.js, common/src/schemas/aiAuditInstallation.model.js

• Discovered: 2026-04-21 during demo run (TC SRB87 KAIDOTLU BHAVI KATTE I.P, Dec 2025)

• Problem: After any tagging change, frontend-displayed loss% didn't match the loss% the freeze persisted.

  • Root cause: frontend and backend used different consumption values for the same installation. Backend used avgConsumption6Months for remark installations when summing installationConsumption; frontend used actual consumption (because fetchAiAuditInstallations excludes avgConsumption6Months, hasRemark, aiSuggestsConsumption from its projection, leaving the frontend with no way to know which value to use).
  • Concrete example: TC with computed.installationConsumption = 17,041.67; user untags JGAEH30941 (actual 153, avg 34.67, ABNORMAL). Frontend shows 16,888.67 (subtracted 153); backend saved 17,007 (subtracted 34.67). ~0.6% loss% mismatch.

• Short-term demo fix (Option X): 3076b1a — freeze handler now applies override deltas using actual consumption, matching frontend math. Ships the tagging-change scenario. Does NOT cover the "Add consumption" toggle case where user keeps an installation tagged but opts out of avg (Scenario B in LLD-ai-audit-freeze-alignment.md).

• Full fix (planned): Aligned with frontend team post-demo. See LLD. Three backend PRs:

  1. Add effectiveConsumption field to aiAuditInstallation schema, populate during AI audit run
  2. Expose effectiveConsumption in fetchAiAuditInstallations + fetchAiAuditRemarks responses
  3. Revert Option X; freeze handler sums effectiveConsumption over finalTaggedSet, with useAvgConsumption: false override switching to actual

  Frontend then replaces local inst.consumption math with inst.effectiveConsumption, handles "Add consumption" toggle by swapping avg↔actual per installation, sends useAvgConsumption in overrides when user deviated from default.

• Status: Fixed. Option X shipped 2026-04-21 (commit 3076b1a). Full fix shipped 2026-04-24 — three commits: 450e258 (schema + populate), b6fc301 (expose in fetch endpoints), 8564c2e (freeze handler revert + useAvgConsumption honoring). Verified on dev with all 5 test scenarios passing. Frontend coordination required: switch to inst.effectiveConsumption for local loss% math; send useAvgConsumption: false in freeze overrides when user unchecks "Add consumption".

B6. Migrated installations missing billing boolean flags — CRITICAL (STG)

• Files: installation collection (migrated data), api/src/entities/audit/controller/fetchInstallations.js
• Problem: Migrated production data has all billing boolean flags (mnr, vacant, doorLock, zeroConsumption, unbilled) set to false (Mongoose defaults) on installation documents. The transformer billingCount aggregates were migrated correctly from production (e.g. MNR: 5), but the individual installation boolean fields were never populated. The fetchInstallations API filters by these boolean flags (e.g. { mnr: true }) and returns 0 results — UI shows "No results found" despite billingCount chips showing non-zero counts.
• Root cause: Production stores billing status as a single string billing.status: "MNR" in a nested array. Demo schema uses individual boolean flags. Migration carried over the transformer aggregates but skipped the installation-level billing status conversion.
• Verified: db.installation.countDocuments({ tcId: 'R37IHSkTm7NN6_pqL6ARY', month: '2025-12' }) → 26 total, 0 with any boolean flag true. Confirmed widespread across all migrated TCs (sampled 5 TCs — all show the same pattern).
• Fix: Write a backfill script (installation_backfillBillingFlags) that reads from the production DB, maps each installation's billing.status string to the corresponding boolean flag, and updates the demo DB. Then recompute billingCount on all transformers via computeAuditFields. Also populate consumption, demand, collection which are all null on migrated docs.
• Status: Not started

---

Minor Issues

C1. auditedOn excluded from API response — MEDIUM

• Files: api/src/entities/audit/controller/fetchTransformerDetail.js (line 32), api/src/entities/audit/controller/auditTC.js
• Problem: computeAuditFields sets auditedOn: new Date() but both detail and audit response projections exclude it (auditedOn: 0). Field is saved to DB but never returned.
• Fix: Remove auditedOn from projection exclusion.
• Status: Not started

C2. meterConstantChanged never set by audit — MEDIUM

• File: api/src/entities/audit/helpers/computeAuditFields.js
• Problem: meterConstantChanged exists in transformer schema (line 69) and UI shows a "Changed" badge, but computeAuditFields never sets it. Always stays false (default).
• Fix: Determine when meter constant changes (likely during AI tagging/freeze flow) and set the flag accordingly.
• Status: Not started

---

Environments

Env	MongoDB Connection	DB Name	Purpose
Local	mongodb+srv://demo:...@energyaudit.0l6duna.mongodb.net/	dtcea-mumbai-demo	Local development
Dev	mongodb+srv://dgi-ai-widgets-demo-dev:...@dev.e45p6au.mongodb.net/	dtcea-mumbai-demo	Development environment
Stg	mongodb+srv://dgi-ai-widgets-demo-stg:...@stg.ivmjjjb.mongodb.net/	dtcea-mumbai-demo	Staging environment

Connection strings stored in both api/.env and script/.env — toggle by commenting/uncommenting.

Scripts Run Order (all envs)

1. seed_aiAuditTestData → 2. transformer_backfillAuditStatus → 3. transformer_fixInvalidAuditedTCs → 4. installation_backfillAvgConsumption → 5. transformer_backfillHealthScore → 6. auditSummary_computeAuditSummary

Execution plan: 2026-03-31-run-scripts-all-envs.md

---

Progress Summary

Category	Count	Status
Completed endpoints	20	Done
Data scripts (done)	5 (backfillAuditStatus, fixInvalidAuditedTCs, computeAuditSummary, seedAiAuditTestData, backfillAvgConsumption)	Done
New endpoints remaining	2 (prepare-demo-data, restore-demo-data)	Not started
Enhancement remaining	1 (adjacent month loss %)	Not started
Bugs to fix	5 (B2-B6), 1 fixed (B1)	Mixed
Minor issues	2 (C1-C2)	Not started
API LLDs written	13 (8 + schemas + run + fetch + freeze + prepare-demo-data)	Done
Script LLDs written	5 (fixInvalidAuditedTCs, computeAuditSummary, backfillAvgConsumption, seedAiAuditData)	Done
LLDs remaining	0	All done