> ## Documentation Index
> Fetch the complete documentation index at: https://docs.younegotiate.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Import Consumer Processing

> Manual CSV uploads and SFTP imports share the same consumer-import queue pipeline.

## Real-World Example

ABC Collections can import consumers in two ways: an employee manually uploads a CSV in the portal, or the company drops a CSV into its SFTP folder. Once the file is accepted, both paths use the same background processing pipeline.

That shared pipeline stores the file, creates import history, splits the work into chunks, processes rows safely, records failed rows, finalizes counts, and triggers the correct consumer communications.

## Visual Flow

```mermaid placement="top-right" actions={true} theme={"system"}
flowchart TD
    A["Manual CSV upload"] --> C["File stored in import_consumers"]
    B["Scheduled SFTP pickup"] --> C
    C --> D["File upload history created"]
    D --> E["Coordinator reads rows"]
    E --> F["Create 500-row chunk records"]
    F --> G["Dispatch chunk jobs on imports queue"]
    G --> H["Create, update, or deactivate consumers"]
    H --> I{"Any row errors?"}
    I -->|Yes| J["Append failed CSV with Errors column"]
    I -->|No| K["No failed file needed"]
    J --> L["Finalize import"]
    K --> L
    L --> M["Send deferred communications and completion notices"]
    classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1.5px,color:#0F172A;
    classDef actor fill:#E0F2FE,stroke:#0284C7,stroke-width:2px,color:#0C4A6E;
    classDef system fill:#F8FAFC,stroke:#64748B,stroke-width:1.5px,color:#0F172A;
    classDef decision fill:#FEF3C7,stroke:#D97706,stroke-width:2px,color:#78350F;
    classDef risk fill:#FEE2E2,stroke:#DC2626,stroke-width:2px,color:#7F1D1D;
    classDef outcome fill:#DCFCE7,stroke:#16A34A,stroke-width:2px,color:#14532D;
    class A,B actor;
    class C,D,E,F,G,H,K,L,M system;
    class I decision;
    class J risk;
    class M outcome;
    linkStyle default stroke:#94A3B8,stroke-width:2px;
```

## Shared Processing Rules

| Stage            | What Happens                                                                                                                                                             |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| File storage     | The CSV is copied into `import_consumers` so queue workers can read it outside the web request or SFTP connection.                                                       |
| Import history   | A `file_upload_histories` record tracks company, sub-account, import type, filename, status, totals, failed count, and whether the source was SFTP.                      |
| Chunk creation   | The coordinator skips blank rows and creates one `consumer_import_chunks` record per 500 data rows.                                                                      |
| Chunk processing | Each chunk validates rows, performs create/update/delete work, batches database writes, and records failed rows.                                                         |
| Finalization     | The final job aggregates chunk counts, marks the import completed or failed, records duration, sends deferred notifications, and runs billing or subscription sync work. |

## Manual Import Path

Manual import starts from [Import File](/portals/creditor/import-export/import-file).

The creditor selects a mapped Header Profile, chooses Add, Add + CFPB, Update, or Delete, uploads a CSV, and the app validates the file headers before storing the file. Manual imports calculate total records before the coordinator creates chunks.

## SFTP Import Path

SFTP import starts from [SFTP Connections](/portals/creditor/manage-account/sftp-connections).

The scheduled command scans active mapped SFTP header profiles twice daily. When it finds CSV files in the expected folders, it copies each file into `import_consumers`, creates SFTP import history, and dispatches the same coordinator job used by manual uploads.

After dispatching the import, the SFTP follow-up job moves the remote source file into `proceed`. If a failed-records CSV is available, it is copied back into the remote `failed` folder.

## Four Import Types

| Type       | Processing Result                                                                                                                        |
| ---------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| Add        | Creates active consumer accounts, reusing an existing consumer profile when the same PII identity already exists.                        |
| Add + CFPB | Creates active consumer accounts and makes the completed file available in CFPB Validation Letter workflows.                             |
| Update     | Finds existing accounts by account number and updates only the selected fields. Pay-term changes can queue updated-offer communications. |
| Delete     | Deactivates matching accounts, cancels scheduled or failed transactions, and queues removed-account communications.                      |

## Communication Behavior

The import pipeline does not blindly send messages while rows are still processing. It stores communication work and lets follow-up jobs send or schedule messages after the import work is safe.

* New accounts can trigger new-account or existing-PII-account communication.
* Pay-term updates can trigger updated-offer communication.
* Deleted accounts can trigger creditor-removed-account communication.
* Secure CFPB EcoLetters trigger CFPB delivery communication when the creditor sends those letters.
* Email/SMS still depends on feature flags, opt-out state, contact permissions, valid email/phone, and available communication templates.

For provider delivery, open tracking, SMS opt-out replies, and webhook callbacks, see [Communication Delivery & Webhooks](/portals/creditor/background-processing/communication-delivery-and-webhooks).

## How It Should Work

* It should process manual and SFTP files through the same safe queue pipeline.
* It should make import progress visible through import history.
* It should keep failed row details attached to the import.
* It should make SFTP imports auditable by marking the history as SFTP.
* It should finalize only after all chunks finish.

## How It Should Not Work

* It should not process all rows in a single browser request.
* It should not mix files or failed rows between creditors.
* It should not mark an import completed while chunks are still pending or processing.
* It should not send communications for rows that failed validation.

## Developer Notes

* `ImportConsumersCoordinatorJob` owns chunk creation and dispatch.
* `ProcessConsumerChunkJob` owns row-level validation and create/update/delete work.
* `FinalizeConsumerImportJob` owns aggregate status, duration, large-upload mail, over-limit billing, and subscription sync after deletion.
* `ProcessConsumerImportNotificationsJob` dispatches deferred consumer communication work in 500-notification batches.
* SFTP imports use `ImportConsumersViaSFTPCommand` and then reuse the same coordinator/chunk/finalize jobs.

## Related App Areas

* `app/Livewire/Creditor/ImportConsumers/IndexPage.php`
* `app/Console/Commands/ImportConsumersViaSFTPCommand.php`
* `app/Console/Kernel.php`
* `app/Enums/FileUploadHistoryType.php`
* `app/Enums/ConsumerUpdateFields.php`
* `app/Jobs/ImportConsumersCoordinatorJob.php`
* `app/Jobs/ProcessConsumerChunkJob.php`
* `app/Jobs/FinalizeConsumerImportJob.php`
* `app/Jobs/ProcessConsumerImportNotificationsJob.php`
* `app/Jobs/GenerateErrorFileOfImportedConsumersViaSFTPJob.php`