Performing Batch Processing

PayRun.io provides the ability to enqueue batches of tasks in a single request. This is accomplished using the Batch Job Instruction. and listing the requred actions.

Batch Job

Batch processing is handled using a background job. You enqueue a batch job then monitor the job progress, waiting until it has completed.

The batched actions are described as a list of batch items. The job can contain any number of batch items, which are executed in the listed order.

Validate Only

Setting the Validate Only value to true, performs a validation only check against the job batch commands. Errors will be reported in the job results, but changes are not commited to the database.

Example Batch Job

curl -X POST \
  'https://api.test.payrun.io/jobs/batch' \
  -H 'Accept: application/xml' \
  -H 'Api-Version: default' \
  -H 'Authorization: {OAuthHeader}' \
  -H 'Cache-Control: no-cache' \
  -H 'Content-type: application/xml' \
  -d '<?xml version="1.0"?>
<BatchJobInstruction xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <HoldingDate xsi:nil="true" />
  <ValidateOnly>false</ValidateOnly>
  <Instructions>
    <PUT Href="/Employer/ER001">
      <Body xsi:type="Employer">
        <EffectiveDate>2018-04-06</EffectiveDate>
        <Name>Batch Employer</Name>
        <Region>NotSet</Region>
        <Territory>UnitedKingdom</Territory>
        <RuleExclusions>None</RuleExclusions>
        <ClaimEmploymentAllowance>false</ClaimEmploymentAllowance>
        <ClaimSmallEmployerRelief>false</ClaimSmallEmployerRelief>
        <ApprenticeshipLevyAllowance>0</ApprenticeshipLevyAllowance>
      </Body>
    </PUT>
    <POST Href="/Employer/ER001/PaySchedules">
      <Body xsi:type="PaySchedule">
        <Name>Test1</Name>
        <PayFrequency>Monthly</PayFrequency>
      </Body>
    </POST>
    <PUT Href="/Employer/ER001/PaySchedule/TEST001">
      <Body xsi:type="PaySchedule">
        <Name>Test2</Name>
        <PayFrequency>Monthly</PayFrequency>
      </Body>
    </PUT>
    <PUT Href="/Employer/ER001/Employee/EE001">
      <Body xsi:type="Employee">
        <EffectiveDate>2018-04-06</EffectiveDate>
        <Code>EMP001</Code>
        <FirstName>John</FirstName>
        <MiddleName xsi:nil="true" />
        <LastName>Smith</LastName>
        <DateOfBirth>0001-01-01</DateOfBirth>
        <Region>England</Region>
        <Territory>UnitedKingdom</Territory>
        <NicLiability>IsFullyLiable</NicLiability>
        <PaySchedule href="/Employer/ER001/PaySchedule/TEST001" />
        <AEAssessmentOverride>None</AEAssessmentOverride>
      </Body>
    </PUT>
    <PATCH Href="/Employer/ER001/Employee/EE001">
      <Body><![CDATA[<Employee><EffectiveDate>2018-04-06</EffectiveDate><Deactivated>true</Deactivated></Employee>]]></Body>
    </PATCH>
    <DELETE Href="/Employer/ER001/PaySchedule/SCH001" />
  </Instructions>
</BatchJobInstruction>'
curl -X POST \
  'https://api.test.payrun.io/jobs/batch' \
  -H 'Accept: application/json' \
  -H 'Api-Version: default' \
  -H 'Authorization: {OAuthHeader}' \
  -H 'Cache-Control: no-cache' \
  -H 'Content-type: application/json' \
  -d '{
  "BatchJobInstruction": {
    "HoldingDate": null,
    "ValidateOnly": "false",
    "Instructions": {
      "PUT": [
        {
          "@Href": "/Employer/ER001",
          "Body": {
            "@xsi:type": "Employer",
            "EffectiveDate": "2018-04-06",
            "Name": "Batch Employer",
            "Region": "NotSet",
            "Territory": "UnitedKingdom"
          }
        },
        {
          "@Href": "/Employer/ER001/PaySchedule/TEST001",
          "Body": {
            "@xsi:type": "PaySchedule",
            "Name": "Test2",
            "PayFrequency": "Monthly"
          }
        },
        {
          "@Href": "/Employer/ER001/Employee/EE001",
          "Body": {
            "@xsi:type": "Employee",
            "EffectiveDate": "2018-04-06",
            "Code": "EMP001",
            "FirstName": "John",
            "LastName": "Smith",
            "DateOfBirth": "1980-01-01",
            "Region": "England",
            "Territory": "UnitedKingdom",
            "NicLiability": "IsFullyLiable",
            "PaySchedule": {
              "@href": "/Employer/ER001/PaySchedule/TEST001"
            },
            "StartDate": "2018-01-01",
            "AEAssessmentOverride": "None"
          }
        }
      ],
      "POST": {
        "@Href": "/Employer/ER001/PaySchedules",
        "Body": {
          "@xsi:type": "PaySchedule",
          "Name": "Test1",
          "PayFrequency": "Monthly"
        }
      },
      "PATCH": {
        "@Href": "/Employer/ER001/Employee/EE001",
        "Body": {
          "#cdata-section": "<Employee><EffectiveDate>2018-04-06</EffectiveDate><Deactivated>true</Deactivated></Employee>"
        }
      },
      "DELETE": [
        {
          "@Href": "/Employer/ER001/PaySchedule/SCH001"
        }
      ]
    }
  }
}'

Batch Items

Actions are included using one of the batch item types. Each item type name describes the batch action to perform and includes the resource location to target.

Delete

The delete item only requires the target resource location.

<DELETE Href="/Employer/ER001/PaySchedule/SCH001" />
"DELETE": {
    "@Href": "/Employer/ER001/PaySchedule/SCH001"
}

Patch

Patch items allow you to specify a partial object graph along with the target url. The patch body must describe the patch in XML wrapped inside a CDATA section.

<PATCH Href="/Employer/ER001/Employee/EE001">
    <Body><![CDATA[<Employee><EffectiveDate>2018-04-06</EffectiveDate><Deactivated>true</Deactivated></Employee>]]></Body>
</PATCH>
"PATCH": {
    "@Href": "/Employer/ER001/Employee/EE001",
    "Body": {
        "#cdata-section": "<Employee><EffectiveDate>2018-04-06</EffectiveDate><Deactivated>true</Deactivated></Employee>"
    }
}

NOTE:
When putting or posting you must specify the body data type using the xsi:type attribute.

Post

The post item allows you to insert a new object into a parent collection. It requires both the target collection url and complete object to be posted. Posted objects receive an auto generated unique key.

<POST Href="/Employer/ER001/PaySchedules">
    <Body xsi:type="PaySchedule">
        <Name>Test1</Name>
        <PayFrequency>Monthly</PayFrequency>
    </Body>
</POST>
"POST": {
    "@Href": "/Employer/ER001/PaySchedules",
    "Body": {
        "@xsi:type": "PaySchedule",
        "Name": "Test1",
        "PayFrequency": "Monthly"
    }
}

Put

Performing a put allows you specify the exact resource location to be updated or inserted. Use this option if you do not want to rely on auto-generated unique key identifiers.

<PUT Href="/Employer/ER001">
    <Body xsi:type="Employer">
        <EffectiveDate>2018-04-06</EffectiveDate>
        <Name>Batch Employer</Name>
        <Region>NotSet</Region>
        <Territory>UnitedKingdom</Territory>
    </Body>
</PUT>
{
    "@Href": "/Employer/ER001",
    "Body": {
        "@xsi:type": "Employer",
        "EffectiveDate": "2018-04-06",
        "Name": "Batch Employer",
        "Region": "NotSet",
        "Territory": "UnitedKingdom"
    }
}

Nondeterministic Execution Order

The batch service uses multi-threading to process the batch instructions in the fast time possible. The downside to this approach is that order in which the instructions are processed cannot always be determined.
This means that you cannot reliably use a single batch to generate multiple levels of dependency.

For example, you cannot reliably create a new employee and assign that employee a pay instruction in a single batch because the processing order cannot be guaranteed. Its possible that the pay instruction could be processed before the employee is generated resulting in a 404 Not Found result.

To work around this limitation, it is advised to only specify either unrelated or sibling level entities in a single batch job.

Transactional Batch Processing

During the batch processing, it is possible that the some instructions will be invalid and fail to complete. In order to avoid partially complete actions and an unknown data state, all items in a batch are processed within a single transaction. If any item within the batch fails, the entire transaction is rolled back.

Job Errors

In the event that a batch instruction fails to successfully complete, details of that error are added to the Job Info Errors collection.

Error messages follow a fixed pattern to help with identification of the failed batch instruction.

Error message pattern

Batch job error messages follow a pre-determined pattern as follows:

Batch item index - [verb] data type "url" failed. Error: error detail

Error detail item Detail
index the index position of the failed batch item in the batch job instruction.
verb the http action verb of the failed batch item; PUT, POST, PATCH or DELETE.
data type the failed batch item entity type. E.g. Employee.
url the failed batch item targeted URL.
error detail the detail of the failure.

Example

Batch item 3 - [PUT] PaySchedule "/Employer/ER001/PaySchedule/B" failed. Error: Validation Exception: Data Integrity Violation Detected. Unable to perform requested 'Insert' action on entity 'PaySchedule'.

Bad Batch Error Example

There's an error in the below batch job instruction. The job process will fail on the third batch item due to a data integrity rule (DIR4200 - Pay schedule names must be unique) violation.

curl -X POST \
  'https://api.test.payrun.io/jobs/batch' \
  -H 'Accept: application/xml' \
  -H 'Api-Version: default' \
  -H 'Authorization: {OAuthHeader}' \
  -H 'Cache-Control: no-cache' \
  -H 'Content-type: application/xml' \
  -d '<BatchJobInstruction xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <HoldingDate xsi:nil="true" />
  <ValidateOnly>false</ValidateOnly>
  <Instructions>
    <PUT Href="/Employer/ER001">
      <Body xsi:type="Employer">
        <EffectiveDate>0001-01-01</EffectiveDate>
        <Revision>0</Revision>
        <Name>Batch Employer</Name>
        <Region>NotSet</Region>
        <Territory>UnitedKingdom</Territory>
        <BacsServiceUserNumber xsi:nil="true" />
        <RuleExclusions>None</RuleExclusions>
        <ClaimEmploymentAllowance>false</ClaimEmploymentAllowance>
        <ClaimSmallEmployerRelief>false</ClaimSmallEmployerRelief>
        <ApprenticeshipLevyAllowance>0</ApprenticeshipLevyAllowance>
        <CalculateApprenticeshipLevy>false</CalculateApprenticeshipLevy>
        <Address xsi:nil="true" />
        <HmrcSettings xsi:nil="true" />
        <BankAccount xsi:nil="true" />
        <AutoEnrolment xsi:nil="true" />
        <MetaData xsi:nil="true" />
      </Body>
    </PUT>
    <PUT Href="/Employer/ER001/PaySchedule/A">
      <Body xsi:type="PaySchedule">
        <Name>Schedule A</Name>
        <PayFrequency>Monthly</PayFrequency>
        <MetaData xsi:nil="true" />
      </Body>
    </PUT>
    <PUT Href="/Employer/ER001/PaySchedule/B">
      <Body xsi:type="PaySchedule">
        <Name>Schedule A</Name>
        <PayFrequency>Monthly</PayFrequency>
        <MetaData xsi:nil="true" />
      </Body>
    </PUT>
  </Instructions>
</BatchJobInstruction>'
curl -X POST \
  'https://api.test.payrun.io/jobs/batch' \
  -H 'Accept: application/json' \
  -H 'Api-Version: default' \
  -H 'Authorization: {OAuthHeader}' \
  -H 'Cache-Control: no-cache' \
  -H 'Content-type: application/json' \
  -d '"PUT": {
  "BatchJobInstruction": {
    "HoldingDate": null,
    "ValidateOnly": "false",
    "Instructions": {
      "PUT": [
        {
          "@Href": "/Employer/ER001",
          "Body": {
            "@xsi:type": "Employer",
            "EffectiveDate": "0001-01-01",
            "Revision": "0",
            "Name": "Batch Employer",
            "Region": "NotSet",
            "Territory": "UnitedKingdom",
            "BacsServiceUserNumber": null,
            "RuleExclusions": "None",
            "ClaimEmploymentAllowance": "false",
            "ClaimSmallEmployerRelief": "false",
            "ApprenticeshipLevyAllowance": "0",
            "CalculateApprenticeshipLevy": "false",
            "Address": null,
            "HmrcSettings": null,
            "BankAccount": null,
            "AutoEnrolment": null,
            "MetaData": null
          }
        },
        {
          "@Href": "/Employer/ER001/PaySchedule/A",
          "Body": {
            "@xsi:type": "PaySchedule",
            "Name": "Schedule A",
            "PayFrequency": "Monthly",
            "MetaData": null
          }
        },
        {
          "@Href": "/Employer/ER001/PaySchedule/B",
          "Body": {
            "@xsi:type": "PaySchedule",
            "Name": "Schedule A",
            "PayFrequency": "Monthly",
            "MetaData": null
          }
        }
      ]
    }
  }
}'

Job Info Error Response

The following job failure information examples demonstrate how the exceptions are reported after a batch job has failed.

NOTE:
Additional error information (on the failure) can be reported on a subsequent error line.

<?xml version="1.0"?>
<JobInfo xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <JobId>a980bcab-6e45-4bb0-b396-f29c667a1833</JobId>
  <Created>2021-03-02T08:52:38.2666667</Created>
  <LastUpdated>2021-03-02T08:52:39.2666667</LastUpdated>
  <JobType>BatchJob</JobType>
  <JobStatus>Failed</JobStatus>
  <Progress>1.000</Progress>
  <EmployerKey />
  <HoldingDate xsi:nil="true" />
  <Errors>
    <Error>Batch item 3 - [PUT] PaySchedule "/Employer/ER001/PaySchedule/B" failed. Validation Exception: Data Integrity Violation Detected. Unable to perform requested 'Insert' action on entity 'PaySchedule'.</Error>
    <Error>[DIR4200] - Cannot apply pay schedule change; name is already in use. Pay schedule names must be unique within the employer scope.</Error>
  </Errors>
</JobInfo>
{
  "JobInfo": {
    "JobId": "e79fa160-577e-4fde-b8b5-6d37ad3c06b7",
    "Created": "2021-03-02T08:52:00.8933333",
    "LastUpdated": "2021-03-02T08:52:02",
    "JobType": "BatchJob",
    "JobStatus": "Failed",
    "Progress": "1.000",
    "EmployerKey": null,
    "HoldingDate": null,
    "Errors": {
      "Error": [
        "Batch item 3 - [PUT] PaySchedule \"/Employer/ER001/PaySchedule/B\" failed. Validation Exception: Data Integrity Violation Detected. Unable to perform requested 'Insert' action on entity 'PaySchedule'.",
        "[DIR4200] - Cannot apply pay schedule change; name is already in use. Pay schedule names must be unique within the employer scope."
      ]
    }
  }
}