Skip to main content
Data collections allow working with data in an external application using a consistent API. They let you do the following:
  • Get schema of data record fields, including custom fields configured for a particular connection.
  • Perform CRUD operations on data records using a consistent API.
  • Understand which operations are possible and not possible for a given collection.
  • Subscribe to data change events even if webhooks are not supported by the underlying API.
To learn how to work with data collections, see Data Collections API Reference
Data Collections are connector-level interfaces. To change them, you need to modify the Connector that implements a given data collection.

​
Spec

The data collection specification defines the structure and capabilities of a data collection. It is located in the root of the data collection directory (data/<collection-key>/spec.yml). Here’s an example: 
contacts:
  name: Contacts
  fieldsSchema:
    type: object
    properties:
      id:
        type: string
        readOnly: true
      email:
        type: string
      firstName:
        type: string
      lastName:
        type: string
      fullName:
        type: string
        writeOnly: true
  methods:
    list:
      filterFields:
        - email
    create:
      requiredFields:
        - email
    update:
      excludedFields:
        - id
    delete: {}
    findById: {}
    match:
      fields:
        - email
    search: {}
  events:
    created:
      implementationType: webhook
    updated:
      implementationType: pull-latest-records
    deleted:
      implementationType: custom-pull

​
Properties

PropertyTypeDescription
namestringDisplay name for the collection
parametersSchemaobjectData Schema for collection-level parameters
fieldsSchemaobjectData Schema defining the structure of records in the collection
methodsobjectMethod implementations. See below for details.
eventsobjectEvent implementations. See below for details.

​
Parameters

Data collection parameters are used to configure the data collection implementation. They are passed to execution context of all data collection functions as parameters argument. You should only add parameters to the data collection when you cannot use function-level parameters. If collection-level and function-level parameters overlap, function-level parameters take precedence.

​
Fields Schema

This Data Schema defines fields of the records in this data collection. It is recommended that all the fields that could participate in any operation of this collection be present in the fields schema. If some fields are only available in read or write operations, you can mark them as readOnly or writeOnly. Furthermore, you can configure which fields are available for which operations (i.e. create or update) in the corresponding operation configuration. Custom Fields If a data collection supports custom fields that can be configured on a per-connection basis, you can implement a customFields function that returns the custom fields schema.\ The custom fields schema will be merged with fieldsSchema when returning the collection’s specification.

​
Methods

You can configure the following methods for the data collection:
MethodDescription
listRead all the records from the collection (with optional filters).
findByIdGet one record from the collection using its id.
matchFind a single matching record in a data collection using provided fields (e.g., by email).
searchFind a list of records using a string query. Supports type-ahead search whenever possible.
createCreate one record in a data collection.
updateUpdate a record in a data collection using its id.
deleteDelete a record with a given id.

​
Data Collection Functions

These are non-method functions that affect the data collection functionality.
FunctionDescription
recordFromFieldsBuild Data Records from their fields - every data collection must have one
specDynamically generate the collection specification based on API capabilities
customFieldsDynamically generate custom fields schema based on API configuration
fieldsFromApiTransform API response fields to match your fieldsSchema format
fieldsToApiTransform fieldsSchema format to API format before sending requests
When you want to transform fields that are coming from the API before returning them from the connector, or transform fields that come from the client before sending them to the API, you can use fieldsFromApi and fieldsToApi functions respectively. These functions will be automatically applied to fields for all method and event implementations. fieldsToApi will be automatically applied after method is executed and before sending the request to the API. fieldsFromApi will be automatically applied after the method is executed and before the result is returned to the client. This could be useful if:
  • Fields in the API are inconsistent between endpoints.
  • Field structures in the API are too complex and you want to simplify them.

​
Inheritance (Mixins)

When multiple data collections share the implementation of some of their functions, you can use mixins. Example: 
# spec.yml
name: Users
extends:
  - key: _webhooks
    parameters:
      objectKey: user
If a collection uses a mixin, all the functions (and methods) from the mixin will be used as the collection’s functions unless overridden. If a mixin collection has parameters, each collection that uses the mixin can provide different parameter values.