Import Consumer Processing - YouNegotiate Docs

Real-World Example

ABC Collections can import consumers in two ways: an employee manually uploads a CSV in the portal, or the company drops a CSV into its SFTP folder. Once the file is accepted, both paths use the same background processing pipeline. That shared pipeline stores the file, creates import history, splits the work into chunks, processes rows safely, records failed rows, finalizes counts, and triggers the correct consumer communications.

Visual Flow

Shared Processing Rules

Stage	What Happens
File storage	The CSV is copied into `import_consumers` so queue workers can read it outside the web request or SFTP connection.
Import history	A `file_upload_histories` record tracks company, sub-account, import type, filename, status, totals, failed count, and whether the source was SFTP.
Chunk creation	The coordinator skips blank rows and creates one `consumer_import_chunks` record per 500 data rows.
Chunk processing	Each chunk validates rows, performs create/update/delete work, batches database writes, and records failed rows.
Finalization	The final job aggregates chunk counts, marks the import completed or failed, records duration, sends deferred notifications, and runs billing or subscription sync work.

Manual Import Path

Manual import starts from Import File. The creditor selects a mapped Header Profile, chooses Add, Add + CFPB, Update, or Delete, uploads a CSV, and the app validates the file headers before storing the file. Manual imports calculate total records before the coordinator creates chunks.

SFTP Import Path

SFTP import starts from SFTP Connections. The scheduled command scans active mapped SFTP header profiles twice daily. When it finds CSV files in the expected folders, it copies each file into import_consumers, creates SFTP import history, and dispatches the same coordinator job used by manual uploads. After dispatching the import, the SFTP follow-up job moves the remote source file into proceed. If a failed-records CSV is available, it is copied back into the remote failed folder.

Four Import Types

Type	Processing Result
Add	Creates active consumer accounts, reusing an existing consumer profile when the same PII identity already exists.
Add + CFPB	Creates active consumer accounts and makes the completed file available in CFPB Validation Letter workflows.
Update	Finds existing accounts by account number and updates only the selected fields. Pay-term changes can queue updated-offer communications.
Delete	Deactivates matching accounts, cancels scheduled or failed transactions, and queues removed-account communications.

Communication Behavior

The import pipeline does not blindly send messages while rows are still processing. It stores communication work and lets follow-up jobs send or schedule messages after the import work is safe.

New accounts can trigger new-account or existing-PII-account communication.
Pay-term updates can trigger updated-offer communication.
Deleted accounts can trigger creditor-removed-account communication.
Secure CFPB EcoLetters trigger CFPB delivery communication when the creditor sends those letters.
Email/SMS still depends on feature flags, opt-out state, contact permissions, valid email/phone, and available communication templates.

For provider delivery, open tracking, SMS opt-out replies, and webhook callbacks, see Communication Delivery & Webhooks.

How It Should Work

It should process manual and SFTP files through the same safe queue pipeline.
It should make import progress visible through import history.
It should keep failed row details attached to the import.
It should make SFTP imports auditable by marking the history as SFTP.
It should finalize only after all chunks finish.

How It Should Not Work

It should not process all rows in a single browser request.
It should not mix files or failed rows between creditors.
It should not mark an import completed while chunks are still pending or processing.
It should not send communications for rows that failed validation.

Developer Notes

ImportConsumersCoordinatorJob owns chunk creation and dispatch.
ProcessConsumerChunkJob owns row-level validation and create/update/delete work.
FinalizeConsumerImportJob owns aggregate status, duration, large-upload mail, over-limit billing, and subscription sync after deletion.
ProcessConsumerImportNotificationsJob dispatches deferred consumer communication work in 500-notification batches.
SFTP imports use ImportConsumersViaSFTPCommand and then reuse the same coordinator/chunk/finalize jobs.

app/Livewire/Creditor/ImportConsumers/IndexPage.php
app/Console/Commands/ImportConsumersViaSFTPCommand.php
app/Console/Kernel.php
app/Enums/FileUploadHistoryType.php
app/Enums/ConsumerUpdateFields.php
app/Jobs/ImportConsumersCoordinatorJob.php
app/Jobs/ProcessConsumerChunkJob.php
app/Jobs/FinalizeConsumerImportJob.php
app/Jobs/ProcessConsumerImportNotificationsJob.php
app/Jobs/GenerateErrorFileOfImportedConsumersViaSFTPJob.php

​Real-World Example

​Visual Flow

​Shared Processing Rules

​Manual Import Path

​SFTP Import Path

​Four Import Types

​Communication Behavior

​How It Should Work

​How It Should Not Work

​Developer Notes

​Related App Areas