Skip to main content

CSV Validation and Queuing

This document covers the first stage of the import pipeline: CSV validation and cloud task queuing. This stage runs in Import Manager

Trigger​

Cloud Scheduler sends an HTTP POST request to Import Manager's /policy-import endpoint with the following parameters:

  • leaseId — the lease company identifier
  • persist — true to run the full import, false for validation only
  • schema — the schema type for validation
  • importFileName — the CSV file to process.
  • bucket, importFileDropPath, appDirectory — GCS location details
  • email, fatalErrorEmailToAddress — notification addresses
  • reimportAll — whether to force reimport of all records downstream
  • shouldTerminate — whether to run termination after import

Flow​

CSV Validation​

Download and Parse​

The CSV file is downloaded from GCS and each row is validated against a schema. Rows with the wrong number of columns are caught and marked as invalid.

Row Validation​

Each row is validated against a schema that defines the business rules for the import — which fields are required, what formats are expected, and what constraints apply. The schema is determined by the schema parameter and varies by provider and locale.

Before validation, a pre-validation transformer normalises each row. Rows that pass are then passed through an output transformer that enriches them with additional fields.

Row Handling​

There are three outcomes for each row:

OutcomeConditionAction
ValidPasses validation, no skip reasonWritten to *_valid.csv
InvalidFails validationLogged with error details, counted as invalid
SkippedPasses validation but matches a skip conditionLogged with skip reason, excluded from *_valid.csv

Common column mismatch errors are logged separately from validation errors.

Error Handling​

If the CSV file is not found in GCS, the import stops immediately.

Row-level validation failures and column mismatches are logged and counted but do not stop the import — processing continues with the remaining rows.

If a fatal error occurs such as a stream failure, the entire import fails. A fatal error email is sent to the configured address.

Row-Level Logging​

Every row produces a log entry recording the VRM, VIN (where applicable), row number, and the validation result — SUCCESS, FAILED with error messages, or a skip reason. This log is uploaded to GCS at the end of the stage.

Processed Fleet File​

For fleet file imports, a processed version of the original CSV is created after validation completes. Empty lines are stripped from the file, but the data is otherwise unchanged. This file preserves the raw fleet scope and is used during termination to identify vehicles no longer present in the import.

The *_valid.csv file, by contrast, already contains the enriched fields (leaseId, subCustomerId, formatted dates) added during validation. The valid file flows through the BigQuery pipeline.

Cloud Task Queuing​

If persist is true, a cloud task document is written to Firestore at the path cloudTasks/queue/{year}/{month}/{day}/. The document contains:

  • payload — provider, bucket details, lease ID, email, and import flags
  • type — the cloud task type that determines which endpoint Queue Services will call
  • ETA — delay before the task is dispatched (60 seconds)

A Firebase Function watches this Firestore collection and dispatches the task as an HTTP request to Queue Services, which begins the BigQuery pipeline.

If persist is false, no cloud task is written and the pipeline stops here. This is used for validation-only runs to check file quality before committing to an import.

Post-Validation​

After validation and queuing, Import Manager:

  1. Archives the original CSV file in GCS (moves it from the drop path to an archive path)
  2. Uploads the validation log file to GCS
  3. Truncates temporary files from local disk

Outputs​

OutputDescription
*_valid.csvRows that passed validation, uploaded to GCS
*_processed.csvFleet files only — original CSV with empty lines removed, uploaded to GCS
Validation logPer-row results (success, failure, skip), uploaded to GCS
Cloud task documentWritten to Firestore cloudTasks/queue/{year}/{month}/{day}/
Status: Approved
Category: Protected
Authored By: Hadley on Jul 08, 2026