Batches

Batches let you import many records into Flowize from a JSON, CSV, or XML payload. You create a draft, select the interface and flow that should process the records, validate the payload, and then run the batch. The batch feature is available from Batches inside Triggers. The overview displays existing batches in a paginated table. Use Search to filter by status, environment, batch ID, name, flow, interface, or date range.

Before you start

You will need:

• A published flow, consult flow status for more information.
• An interface that contains the fields that are present in the payload, you can find here more info.

Batch status

• Draft: The initial state of a batch, once it is configured.
• Ready: The batch is ready to be run.
• Processing: The batch is currently being processed.
• Finished: The batch has been successfully processed.
• Finished with errors: The batch has been processed but some records failed.

Create a batch

1. Open Batches and select New Batch.
2. In Batch Details, enter a name. Activate extra options if you also want to add a description or customize the input and output filenames.
3. Click Continue.
4. Choose the interface, running flow, and processing priority. The priority must be numeric and within the configured limits. Interface must contain the scope batch.
5. Click Continue.
6. In the payload step, paste the payload into the editor or select Import from file.
7. Wait for payload validation to complete, then click Save.

note

The payload is validated against the interface. The interface must contain the fields that are present in the payload.

info

When extra options are enabled:

• The input filename is sanitized and receives the payload format extension.
• The output filename is sanitized and receives a .txt extension.
• The description is saved with the batch.
• Priority: Defines the priority in which the data records will be executed. By default, is 50, and you can put a number from 100 (lowest) to 0 (highest).

Run a batch

After the draft has been saved and contains a valid record count, select Run and confirm the launch. Running a batch changes its status to READY and hands it to the processing queue. Processing may begin later.

Configuration can be edited only while the batch is a draft.

Monitor processing

The overview can include:

• Current status and progress
• Process timestamps
• Inserted and skipped record totals
• Batch configuration details
• Links to generated files

The available status and file details depend on whether the batch uses the v1 or v2 processing path. New JSON and CSV batches use v2 chunk processing. Other formats use v1 processing.

Location of batches

It is possible to access anytime all batches that have been created in the account. These are available within the “batch” folder of the file manager. For further information about the file manager, please refer to this post.

Batch version 2

This version improves the way batches are inserted in Flowize, you can send bigger files with more records, and also, they will be inserted faster. Version 2 also allows you to send the batch file as a file in the request, sending it as form-data.

v1 and v2 Comparison

File type	File size	Records	v1 processing time	v2 processing time
JSON	20 MB	96000	05h 10m 51s	12m 21s
JSON	40 MB	192000	09h 10m 51s	17m 28s

note

• In designer, only the first 20 records of the batch will be inserted and executed.
• Currently, v2 works only for JSON and CSV interfaces

How to use it?

Go to your interface details, and in the API endpoints, you will see a Batch v2 link. Use this one instead of the v1 link. The other process is all the same.

Headers (optional)

X-Batch-Status: When you are making a POST request for the batch process to start, you can send the header 'X-Batch-Status: READY' in your request to instantly start the batch process.

Batch report

In the batch report, you can find details about the batch process. You can see when the batch started, when it finished, and how long it took.

Name	Description
Created	When Flowize received the batch
Ready	Set to begin the start process
Started	The time when the process started
Finished	The time when the process finished

Once the process is finished, Flowize will automatically create 4 files that you will be able to download:

File Name	Description
input	Input file will store the data that you send via POST request.
output	Output file will show the list of UUIDs of data records.
process	Process log will show you a detailed report for each of the records.
error	The error log will appear only if an error occurs, and it will show the details of the error.

Recover a failed v2 batch

When a v2 batch enters an error state and has records to process, the process overview can offer two recovery actions:

• Resume requeues the errored chunks from the last inserted record.
• Skip error and resume requeues the errored chunks and adds those chunks to the skipped count.

Send a batch request with a file via postman.

For the batch v2, you also are able to attach a JSON or CSV file in your request as form-data.

Click on body > form-data > file > select the file