To interact with a Data Source, you need to get a Data Source Instance for a specific Connection.

You can get an instance by Data Source Identifier (key or id) and Connection Identifier (integration key or connection id):

const dataSourceInstance = await membrane
  .connection('hubspot')
  .dataSource('{DATA_SOURCE_KEY}')
  .get({ autoCreate: true })

Data Source Instance has the following properties:

Property	Description
`id`	Unique identifier of the instance.
`path`	Path to the Data Collection the data source is pointing to.
`defaultPath`	Path to the Data Collection the data source is pointing to by default.
`udm`	A Universal Data Model records in this data source will be transformed to / from.
`pullUpdatesIntervalSeconds`	How often the data source will be pulled for new events.
`fullSyncIntervalSeconds`	How often the data source will be fully synced (to get events that require a full sync).
`subscriptions`	Information about active subscriptions to Data Source Events (see below for details).
`isCustomized`	Whether this data source was customized by your tenant (path was changed, etc.)

Advanced Options

Data Sources can be configured with the following advanced options:

Pull Updates Interval – number of seconds between each pull for updates such as new events (except events that require Full Sync). Setting it lower will increase the number of API requests and may result in hitting rate limits.
Full Sync Interval – number of seconds between full syncs of the data source. Full Sync is a very heavy operation since it pulls ALL the data from the data source. It generates events for subscriptions that require Full Sync.

API

Here is how to interact with data in external applications using Data Sources.

Create

await membrane
  .connection('hubspot')
  .dataSource('{DATA_SOURCE_KEY}')
  .create()

Get

const dataSourceInstance = await membrane
  .connection('hubspot')
  .dataSource('{DATA_SOURCE_KEY}')
  .get({ autoCreate: true })

Update

await membrane
  .connection('hubspot')
  .dataSource('{DATA_SOURCE_KEY}')
  .patch('{INPUT}')

Setup or Refresh

await membrane
  .connection('hubspot')
  .dataSource('{DATA_SOURCE_KEY}')
  .setup()

Reset

await membrane
  .connection('hubspot')
  .dataSource('{DATA_SOURCE_KEY}')
  .reset()

Navigating Data Source Instances

Data Sources point to Data Collections in external apps.
They let you store and customize data collection and their parameters per tenant.

This page describes how to navigate data sources and configure them for your tenants.

List Data Collections

To get the list of available data collections in a given connector:

await membrane
  .integration('hubspot')
  .getDataCollections()

The response will contain a locations property with a list of nested directories and collections.
If there are too many to return at once, the response will also contain a cursor property. You have to pass it back to get the next page of results.

To list locations inside a specific directory, pass its path to the getLocations method.

Point Data Source to a Collection

When a Data Source is created, it will point to a default collection (if one exists in the external app). You can change it by updating the collectionKey property of the Data Source:

const dataSourceInstance = await membrane
  .connection('hubspot')
  .dataSource('{DATA_SOURCE_KEY}')
  .patch({
    collectionKey: '{COLLECTION_KEY}',
    collectionParameters: '{COLLECTION_PARAMETERS}',
  })

You can only update the collectionKey to point to a Collection. After you update collectionKey, accessing the data source's collection uses the new path.

Data Collection Parameters

Some collections have parameters. For example, a collection "Tasks by Project" may have a projectId parameter.

If a collection has parameters, it will have a parametersSchema property.
It is a Data Schema for its parameters.

To apply parameters to a data collection, add them as a query string to the path property of the Data Source Instance, like this:

{
  "path": "/data/tasks?projectId=123"
}

Working with Data

You can read and write data records using a Data Collection identified by the collectionKey property of the Data Source Instance.

Multiple Data Source Instances per Connection

By default, each pair of Connection and Data Source will have only one Data Source instance.
If you change it (for example, set a collectionKey), it will apply to every place you use a given Data Source with a given connection.

In some cases you may want to create multiple Data Source Instances for the same Connection and Data Source—
for example, if you have multiple locations you want to import.

This can be achieved by providing instanceKey when accessing a Data Source. Each instanceKey will create a new Data Source Instance.

await membrane
  .connection('hubspot')
  .dataSource('{DATA_SOURCE_KEY}', { instanceKey: '{INSTANCE_KEY}' })
  .get({ autoCreate: true })