Custom Processors

Custom processors are Javascript scripts that you can run which allow you to transform data either before or after it's processed by Chain.io.

With a little bit of coding experience, you can use custom processors to make powerful enhancements to your flows to help with specific use cases like:

• adding or removing fields
• augmenting data with custom calculations
• making adjustments to data formats
• running extra validations

Types of Custom Processors

The are two type of custom processors, and they work the same way.  

A pre-processor runs on the data in between when Chain.io receives it from the source and when your Source File Type adapter runs.  Use this to alter or augment data before it runs through the processing engine.

A post-processor runs on the data in between when the Destination File Type adapter runs and when the data is delivered to its final location.  Use this to alter or augment data after it runs through the processing engine.

How To Add A Custom Processor

You add Custom Processors by adding code in the Custom Processors section of the Edit Flow Screen.  There is a separate data entry box for the pre-processor and post-processor.  When you deploy the flow, the custom processor will be automatically invoked at the appropriate time during the flow execution.

How to Write A Custom Processor

Custom processors are scripts written in Javascript.  They run in a sandboxed Node.js environment running Node 14 or higher. You write your code using normal Node.js javascript conventions with the following special exceptions:

• You cannot write asynchronous code, async/await and Promises are not allowed.
• You cannot require or import libraries.
• Your code must execute within 30 seconds or it will be terminated resulting in a failure state.
• You cannot use scheduling functions such as setTimeout or setInterval.
• You cannot access the Symbol object or it's related functions.
• Your code must be less than or equal to 10,000 characters in total.  This limit is per custom processor so you can include both a 10,000 character pre-processor and a 10,000 character post-processor in the same flow setup.

Within your script, you'll have access to the following global variables:

• sourceFiles (pre-processors only): An array of File objects including the data that was submitted to the flow.  You should expect zero or more File objects.  You may receive multiple file objects in certain scenarios like an email source connection where the body and each attachment are sent as separate file objects.  You may need to iterate through the sourceFiles to identify the one you need to work with.
• destinationFiles (post-processors only): An array of File objects including the data that was returned by the flow.  You should expect zero or more File objects. You would receive an empty array if the destination file adapter did not return any data and you may receive multiple files if the output adapter returns multiple objects.  For example, you may submit a CSV with multiple shipments which are output as separate shipment registration requests to a CO2 provider.  In that case, each registration request would be represented by a different File object.
• userLog: An object that you can use to write messages that the user will see in the flow execution screen.  The object has 3 functions which all take a single String variable: info(), warning(), and error().  
• returnSuccess, returnSkipped, returnError: Call one of these functions as the last line that executes in your script and pass in an array of 0 or more file objects.
• lodash: An instance of the lodash library at version >= 4.17.21.  This library is helpful for accomplishing many common javascript tasks for handling objects and arrays without writing extra code.
• xmldom: An instance of @xmldom/xmldom at version >= 0.8.8 that you can use to parse XML files. 
• xpath: An instance of xpath at version >= 0.0.32 that you can use with xmldom to quickly access values in xml files via xpath notation.
• DateTime: An instance of the Luxon DateTime object at version >= 3.3.0 that you can use to manipulate times and dates.
• uuid: A v4 compliant uuid generator.  Usage:
  uuid() // 51639e8d-a12b-4bba-be4b-e2337b166a9c

A Hello World Example

This simple example replaces the body of every file in the flow with "Hello World"

const myFiles = sourceFiles.map((item) => { 
  f.body = 'Hello World!'
  return f
})
returnSuccess(myFiles) // the output of your last command in the script will be returned

An Example Using XML Processing

Let's say you have a system that submits XML data to you that you want to quickly convert into Chain.io Standard Event JSON for further processing. You also only want to capture arrival dates and want your flow to skip on any other dates.

Assume the XML looks like this:

<events>
  <master_bill>MBOL123</master_bill>
  <event>
    <!-- time in epoch seconds -->
    <actual>1542674993</actual>
    <code>Arrival</code>
  </event>
  <event>
    <actual>1542674000</actual>
    <code>Departure</code>
  </event>
</events>

You could write a pre-processor script like this

function makeEventJS (event, masterBill) {

  // read the date as a string value
  const actual = xpath.select('./actual', event)[0].firstChild.data

  // convert the date to ISO format
  const actualDate = DateTime.fromSeconds(parseInt(actual, 10)).toUTC().toISO()

  const code = xpath.select('./code', event)[0].firstChild.data

  return {
    actual_date: actualDate,
    event_code: code,
    master_bill: masterBill
  }
}

function handleFile (file) {
  // get the body
  const source = file.body
  // parse the xml
  const xml = new xmldom.DOMParser().parseFromString(source)

  // use xpath to only get the events we care about
  const events = xpath.select('/events/event/code[text()="Arrival"]/..', xml)

  if (events.length === 0) {
    userLog.warning("No Arrival events found, so we're not going to pass anything into the rest of the flow")
  }

  const masterBill = xpath.select('/events/master_bill', xml)[0].firstChild.data

  // create the JSON representation of the events
  const eventJS = events.map(event => makeEventJS(event, masterBill))

  // create Chain.io standard event json structure
  const standardJSON = {
    doc_type: 'event_json',
    events: eventJS
  }

  // replace the body with the standard json and return the file
  return {
    ...file,
    body: JSON.stringify(standardJSON)
  }
}

userLog.info('Beginning custom script')

// filter to only the files that have the xml we're looking for
const myFiles = sourceFiles.filter(sf => sf.body.match('<events>'))

// process each matching file and use the result as the return value of the script
const output = myFiles.map(handleFile)
returnSuccess(output)

Given this script, the output would be

[
  {
    type: 'file',
    body: '{"doc_type":"event_json","events":[{"actual_date":"2018-11-20T00:49:53.000Z","event_code":"Arrival","master_bill":"MBOL123"}]}'
  }
]