CoreETL Step 1-1

CoreETL Supplier Data Onboarding & Enablement (API Based)


Product Description

  • Overview

    CoreETL is an instant deployment middleware ETL application to reduce product based middleware implementations from months to days. In standard operation CoreETL normally takes simplified data from flat files, placing the data through mapping and conversion routines to transform the data in to a standardised format. From here the data loaded in to the target application using client connectors where users and/or customers can see the updated information. This all happens seemlessly with either batch processes or manually triggers to move data from source via CoreETL to the target applications without any intervention.

    The solution is dynamic and controlled via simple table entries eliminating any coding requirements to convert the data in to the technical formats required by most applications. The strength of CoreETL is the ability for instant deployment of product dataset conversion from sources such as vendors in to target applications without technical knowledge. A new source can be configured in minutes to convert the source data in to the format required for the target application such as MDM tools, eCommerce, ERPs, CRM, Accounting among many others.

    CoreETL has pre-built functionality for common mapping functions such as looking up attributes, category mappings, validating and formatting data and can be synchronised with connectors to the target applications providing seamless integration. Custom mapping functions can be integrated in to the application providing unlimited extensibility.

  • Features

    CoreETL provides an unprecedented level of features to be dynamic and adjustable to individual business requirements whilst requiring minimal changes and maintenance with an architecture that reduces ETL deployment time from month to days.

    Supported Sources

    Source data is normally supplied via flat files coming via ftp from external servers, http requests or manual uploads, CoreETL can support most types of applications using a flat file format with the architecture design being platform and application independent.


    On standard platforms source data is processed at approximately 2,000 records per minute bringing Enterprise levels of scalability and data processing to all supported applications, new rules and data sources can be implemented in a matter of minutes.

    Data Types

    CoreETL is primarily designed to handle product datasets in both single and parent/child formats with support data such as attributes, categories and images providing rapid initial data loads and ongoing conversion of product related data.


    CoreETL supports all core applications with pre-built connectors being available for common applications including eCommerce, Accounting, ERP, CRM, POS among other applications as both the source and target.


    CoreETL has been designed to handle core types of data processing and converting the complex data either via compression or explosion in to the target applications format as both one-off data loads and also as part of ongoing data maintenance activities.

    Ease of Use

    The solution uses pre-built routines to map, convert and validate datasets along with entries in an Enterprise level architecture to handle business rules providing flexibility to suit all business requirements and situations.


    CoreETL is designed to only load delta (changed) records to the target application increasing performance and scalability, this includes data such as pricing, categories, images, fixed attributes and dynamic attributes.


    Performance can easily be scaled with increased hardware making sure that your business is always one step ahead of the competition, CoreETL is designed from the ground up to handle 10s of millions of products.

  • Business Rules

    CoreETL has been designed to provide rapid deployment of the normally complex process of ETL conversion between applications. The core system has been designed to allow the addition of new data sources such as vendor product lists in minutes without any technical resource and minimal training. This provides businesses with ongoing reduced maintenance costs, business user control of the data mapping process and rapid release of new datasets to target applications.


    CoreETL comes with two types of routines, core routines built in to the solution such as category lookups, data validation, attribute lookups and fixed settings, the second are custom developed routines written in PHP allowing further extensions to the core processing of the solution.


    The core routines have been designed to allow callbacks on previously processed data; using the ProcessOrder flag you can post-process data and run through multiple routines making incremental changes to the core data.


    Based on the middleware and source data a single category can be exploded in to multiple categories with relevant attributes being set. Examples would be for Clothing/Women to be passed in the source data and automatically setting the attribute Gender->Women and Department->Clothing. There are two levels to categories being a core set based on the source dat and at the target application level, each level can be overridden to allow for exceptions.


    Attributes are the core blocks of data such as Brand and Colour, reference data can be entered in to the CoreETL and lookups performed to match the incoming data against the reference data including alternative spellings and exact matches. This eliminates errors in data entry and provides a level of validation logging where necessary.


    Images are processed in the solution providing delta processing functionality so that only changes are sent to the target applications reducing server overhead, client connectors are designed to add, update or remove images based on the mapped source data.