Data Collections

Data collections allow working with data in an external application using a consistent API. They let you do the following:

Get schema of data record fields, including custom fields configured for a particular connection.
Perform CRUD operations on data records using a consistent API.
Understand which operations are possible and not possible for a given collection.
Subscribe to data change events even if webhooks are not supported by the underlying API.

To learn how to work with data collections, see Data Collections API Reference

Data Collections are connector-level interfaces. To change them, you need to modify the Connector that implements a given data collection.

Spec

The data collection specification defines the structure and capabilities of a data collection. It is located in the root of the data collection directory (data/<collection-key>/spec.yml). Here’s an example:

contacts:
  name: Contacts
  fieldsSchema:
    type: object
    properties:
      id:
        type: string
        readOnly: true
      email:
        type: string
      firstName:
        type: string
      lastName:
        type: string
      fullName:
        type: string
        writeOnly: true
  methods:
    list:
      filterFields:
        - email
    create:
      requiredFields:
        - email
    update:
      excludedFields:
        - id
    delete: {}
    findById: {}
    match:
      fields:
        - email
    search: {}
  events:
    created:
      implementationType: webhook
    updated:
      implementationType: pull-latest-records
    deleted:
      implementationType: custom-pull

Properties

Property	Type	Description
`name`	string	Display name for the collection
`parametersSchema`	object	Data Schema for collection-level parameters
`fieldsSchema`	object	Data Schema defining the structure of records in the collection
`methods`	object	Method implementations. See below for details.
`events`	object	Event implementations. See below for details.

Parameters

Data collection parameters are used to configure the data collection implementation. They are passed to execution context of all data collection functions as parameters argument. You should only add parameters to the data collection when you cannot use function-level parameters. If collection-level and function-level parameters overlap, function-level parameters take precedence.

Fields Schema

This Data Schema defines fields of the records in this data collection. It is recommended that all the fields that could participate in any operation of this collection be present in the fields schema. If some fields are only available in read or write operations, you can mark them as readOnly or writeOnly. Furthermore, you can configure which fields are available for which operations (i.e. create or update) in the corresponding operation configuration. Custom Fields If a data collection supports custom fields that can be configured on a per-connection basis, you can implement a customFields function that returns the custom fields schema.\ The custom fields schema will be merged with fieldsSchema when returning the collection’s specification.

Methods

You can configure the following methods for the data collection:

Method	Description
`list`	Read all the records from the collection (with optional filters).
`findById`	Get one record from the collection using its id.
`match`	Find a single matching record in a data collection using provided fields (e.g., by email).
`search`	Find a list of records using a string query. Supports type-ahead search whenever possible.
`create`	Create one record in a data collection.
`update`	Update a record in a data collection using its id.
`delete`	Delete a record with a given id.

Data Collection Functions

These are non-method functions that affect the data collection functionality.

Function	Description
`recordFromFields`	Build Data Records from their fields - every data collection must have one
`spec`	Dynamically generate the collection specification based on API capabilities
`customFields`	Dynamically generate custom fields schema based on API configuration
`fieldsFromApi`	Transform API response fields to match your fieldsSchema format
`fieldsToApi`	Transform fieldsSchema format to API format before sending requests

When you want to transform fields that are coming from the API before returning them from the connector, or transform fields that come from the client before sending them to the API, you can use fieldsFromApi and fieldsToApi functions respectively. These functions will be automatically applied to fields for all method and event implementations. fieldsToApi will be automatically applied after method is executed and before sending the request to the API. fieldsFromApi will be automatically applied after the method is executed and before the result is returned to the client. This could be useful if:

Fields in the API are inconsistent between endpoints.
Field structures in the API are too complex and you want to simplify them.

Inheritance (Mixins)

When multiple data collections share the implementation of some of their functions, you can use mixins. Example:

# spec.yml
name: Users
extends:
  - key: _webhooks
    parameters:
      objectKey: user

If a collection uses a mixin, all the functions (and methods) from the mixin will be used as the collection’s functions unless overridden. If a mixin collection has parameters, each collection that uses the mixin can provide different parameter values.

Overview

Workspace Elements

Management API

Spec

Properties

Parameters

Fields Schema

Methods

Data Collection Functions

Inheritance (Mixins)

​Spec

​Properties

​Parameters

​Fields Schema

​Methods

​Data Collection Functions

​Inheritance (Mixins)

Spec

Properties

Parameters

Fields Schema

Methods

Data Collection Functions

Inheritance (Mixins)