Batch CSV Processing (WIP)

Description

Whereas standard CSV processing in Helium processes an entire blob, representing a CSV file, at once to produce a collection of object instances, the batch CSV processing features takes a multi-step approach:

• First the Mez:createBatch built-in function is used to create a batch from a blob representing a valid CSV file. 
• The built-in MezBatch and MezBatchItem objects are used to store the created batch.
• The created batch can then be processed item by item using the fromCsvLine built-in function.

MezBatch and MezBatchItem Built-in Objects

The following two objects are used to store batches that are created by the use of the Mez:createBatch built-in function. These objects are built-in objects meaning they are included, by Helium, as part of every Helium app:

@NotTracked
persistent object MezBatch {
 
	// User specified description of this batch
    string name;
 
	// String representing the headers for the CSV file associated with this batch
    string header;
}

@NotTracked
persistent object MezBatchItem {
    
	// Flag indicating whether this batch item has been processed with fromCsvLine
	bool processed;
 
	// A string representing an entire line of the CSV file associated with the batch
    string value;
    
	// Relationship to the built-in MezBatch object
    @ManyToOne
    MezBatch batch via batchItems;
}

These objects can be queried like any other object in Helium using for example the native SQL statements provided by the DSL or using selectors.

Creating a Batch for Processing

To create a batch for processing the Mez:createBatch built-in function can be used:

MezBatch stockUpdateBatch = Mez:createBatch(fileUpload._id, fileUpload.data);

In this case fileUpload represents an object instance containing a blob attribute named data.

The first parameter of Mez:createBatch represents the name of the batch. This can be any value that can be implicitly converted to a string. Note that there is no validation on uniqueness for this value meaning more than one batch can have the same name. If, however, batch creation and batch processing are done separately it makes sense that the batch should be uniquely identifiable. For this purpose we use the id of the fileUpload object instance to ensure uniqueness. 

The second parameter represents the blob value that represents the CSV file to process. blob values can populated using, amongst other methods, the file upload widget, or the inbound API.

The above command will create entries in the database that looks as follows:

Processing an Existing Batch

Once a batch has been created, it can be processed line for line using the fromCsvLine built-in function:

// Get the batch items related to the batch being processed
MezBatchItem stockUpdateBatchItems = MezBatchItem:relationshipIn(batch, stockUpdateBatch);
 
// Results from CSV processing will be stored in this collection
StockUpdate[] results;
 
// Iterate over the batch items and process one by one using fromCsvLine
for(MezBatchItem stockUpdateBatchItem: stockUpdateBatchItems) {
	StockUpdate stockUpdate = StockUpdate:fromCsvLine(stockUpdateBatch.header, stockUpdateBatchItem.value);
	results.append(stockUpdate);
	stockUpdateBatchItem.processed = true;
}

To retrieve the batch items associated with the batch being processed, we make use of the relationship between the two objects.

The first parameter of fromCsvLine represents the header string of the CSV file associated with our batch. This can be retrieved from the MezBatchItem object instance being processed.

Exception Handling and Debugging

Additional Mentions and References

Description

Whereas standard CSV processing in Helium processes an entire blob, representing a CSV file, at once to produce a collection of object instances, the batch CSV processing features takes a multi-step approach:

• First the Mez:createBatch built-in function is used to create a batch from a blob representing a valid CSV file. 
• The built-in MezBatch and MezBatchItem objects are used to store the created batch.
• The created batch can then be processed item by item using the fromCsvLine built-in function.

MezBatch and MezBatchItem Built-in Objects

The following two objects are used to store batches that are created by the use of the Mez:createBatch built-in function. These objects are built-in objects meaning they are included, by Helium, as part of every Helium app:

@NotTracked
persistent object MezBatch {
 
	// User specified description of this batch
    string name;
 
	// String representing the headers for the CSV file associated with this batch
    string header;
}

@NotTracked
persistent object MezBatchItem {
    
	// Flag indicating whether this batch item has been processed with fromCsvLine
	bool processed;
 
	// A string representing an entire line of the CSV file associated with the batch
    string value;
    
	// Relationship to the built-in MezBatch object
    @ManyToOne
    MezBatch batch via batchItems;
}

These objects can be queried like any other object in Helium using for example the native SQL statements provided by the DSL or using selectors.

Creating a Batch for Processing

To create a batch for processing the Mez:createBatch built-in function can be used:

MezBatch stockUpdateBatch = Mez:createBatch(fileUpload._id, fileUpload.data);

In this case fileUpload represents an object instance containing a blob attribute named data.

The first parameter of Mez:createBatch represents the name of the batch. This can be any value that can be implicitly converted to a string. Note that there is no validation on uniqueness for this value meaning more than one batch can have the same name. If, however, batch creation and batch processing are done separately it makes sense that the batch should be uniquely identifiable. For this purpose we use the id of the fileUpload object instance to ensure uniqueness. 

The second parameter represents the blob value that represents the CSV file to process. blob values can populated using, amongst other methods, the file upload widget, or the inbound API.

The above command will create entries in the database that looks as follows:

Processing an Existing Batch

Once a batch has been created, it can be processed line for line using the fromCsvLine built-in function:

// Get the batch items related to the batch being processed
MezBatchItem stockUpdateBatchItems = MezBatchItem:relationshipIn(batch, stockUpdateBatch);
 
// Results from CSV processing will be stored in this collection
StockUpdate[] results;
 
// Iterate over the batch items and process one by one using fromCsvLine
for(MezBatchItem stockUpdateBatchItem: stockUpdateBatchItems) {
	StockUpdate stockUpdate = StockUpdate:fromCsvLine(stockUpdateBatch.header, stockUpdateBatchItem.value);
	results.append(stockUpdate);
}

To retrieve the batch items associated with the batch being processed, we make use of the relationship between the two objects.

The first parameter of fromCsvLine represents a string representing the header of the CSV file being processed. This can be retrieved from the MezBatch object instance as shown.

The second parameter of fromCsvLine represents a string representing the current line of the CSV file being processed. This can be retrieved from the MezBatchItem object instance as shown.

Exception Handling and Debugging

Additional Mentions and References