Data Link Connectors

Introduction

The Kinvey Data Link is a protocol that defines an interface between a Kinvey backend and an external datasource. A Data Link Connector (DLC) links your backend on Kinvey and your data source, providing the client developer a single, unified interface for accessing data.

A data source that is connected via a DLC can be augmented with pre and post business logic collection hooks, and can utilize all of the data features of the client libraries, including data modeling, caching and offline save, and data encryption. Additionally Data Links can receive authentication and authorization tokens (such as SAML) from Kinvey's Mobile Identity Connect, which can be used when accessing the Enterprise data source.

Core Concepts

Architecture

The DLC architecture is defined as a REST API that can be implemented by any server and allows you to request data from that server via your Kinvey backend. The DLC can be developed in any language, such as node.js, Java, and C#, as long as it adheres to the defined REST interface. The Data Link receives REST requests from the Kinvey Cloud Service (KCS), translates those requests into requests native to the underlying data source, and transforms the results into JSON to be sent in the response.

Individual data endpoints are defined as collections in the Kinvey console, and accessed using the normal CRUD operations found in the client libraries. The Kinvey Cloud Service routes these requests to the appropriate DLC for processing. The connection between KCS and the DLC uses a standard TLS 1.2 encryption for secure transport. If Mobile Identity Connect (MIC) is used to authenticate against an enterprise identity system, an enterprise token (such as a SAML assertion) can be retrieved and included in the Data Link request.

The underlying data source can be one of any number of types of data stores. These include but are not limited to:

  • Direct RDBMS systems such as Oracle and Microsoft SQL Server
  • SOAP and REST APIs
  • Data sources accessed via an SDK
  • API Managements systems
  • Enterprise Service Bus (ESB) and SOA Systems
  • Third Party Cloud Services

Best Practices

Data Link Connectors are lightweight services that integrate with enterprise and cloud systems. They are not monolithic integration services for combining all enterprise data integrations into a single service.

There are two primary drivers for the creation of a DLC:

  1. A mobile app needs to use data from an Enterprise data store.
  2. An IT organization requires that a number of mobile apps use a number of overlapping enterprise data sources.

The fundamental concept behind a DLC is that it converts some amount of data sitting on one or more enterprise or cloud systems into one or more entities that are suitable for use with a REST-based API. The basic architectural pattern for this is either a facade or an adaptor. A DLC that takes a single data source and exposes its entities to Kinvey as transformed JSON objects utilizes the adapter pattern, while a more complex DLC implements a facade that will communicate with multiple internal enterprise systems to expose them as single entity to Kinvey.

The most important point about DLCs is that they are meant to be small. They can be thought of as microservices that expose very specific access to a data domain that is meant to be consumed via mobile. A Node.js data link connector that exposes a single endpoint can be written in less than 25 lines of code. They should be kept small, lightweight, and easily deployable.

DLCs should be built so that they expose a group of related entities to the Kinvey platform. The correct amount of entities and the number of systems that each DLC communicates with will vary based on the specific enterprise architectures and applications involved, but should follow some basic guidelines:

  • DLCs should be atomic: Each separate DLC can be developed, tested and maintained without affecting other DLCs. This means that to update the backend data store for a DLC you do not need to re-test and re-deploy other DLCs.
  • One collection maps to one DLC: A Kinvey collection can only be connected to one DLC, so each collection of entities must be served by a single DLC.
  • Build DLCs as microservices: It is better to build out a larger number of simple, targeted DLCs using a microservices architecture than a small number of more complex DLCs. While having more DLCs increases initial build-out time, having fewer DLCs will have higher maintenance costs per change and introduce greater quality risks. When developing DLCs as microservices, each DLC should focus on a single business function and/or data domain.
  • Use shared libraries: DLCs can frequently share code at a module level (such as an XML library for the SOA layer) that can reduce some initial development time. This allows for good code reuse and faster time to production, but may required more deployments if issues are found in the module.

DLCs should transform and minimize data returned so as to be suitable for mobile. Because of the bandwidth/data limit constraints of mobile, the minimum amount of data necessary should be returned. The DLC should transform the data from a single or multiple data sources to a simple, lightweight JSON format to avoid bandwidth usage, data costs, and to provide a positive user experience (UX), as mobile users don't have patience to wait for data to sync or become visibile, or for screens to load.

While DLCs can be used for web and other use cases, it is recommended that DLCs be designed with mobile in mind.

Building a DLC

To build a Data Link, an HTTP REST service must be implemented to receive requests from the Kinvey Cloud Service. The REST API for Data Link should implement an endpoint for each data collection that will be accessed via the Kinvey Cloud Service.

RouteMethodsNotes
/:collectionName/GET, POST, DELETECollection CRUD
/:collectionName/:idGET, PUT, DELETEEntity CRUD
/:collectionName/?queryGET, DELETEParameter Passing
/:collectionName/_countGETCollection Count
/:collectionName/_count?queryGETCollection count with query

Implementing DELETE /:collectionName/ has the scemantic effect of deleting all entities in the specified collection. This is a very dangerous operation, and it is recommended to not implement this method unless there is a very strong use case for doing so.

The DLC API offers a rich set of CRUD operations and querying capabilities. However, not all aspects of a DLC need to be implemented. For example, you could implement a read-only collection by only implementing the GET verb for that endpoint. You could implement basic querying but not support sorting. At minimum, your DLC should implement at least one operation per endpoint, as well as the _count method if you want your data to appear in the Kinvey Console's Data Browser.

CRUD operations

CRUD (Create, Read, Update, Delete) operations are all supported via the Data Link Protocol. The protocol uses standard HTTP Verbs to map to the CRUD operators, as defined by REST. The expected behavior of each of the HTTP Methods associated with each endpoint is:

MethodOperation
GETRetrieve the specified entity, query, or collection
POSTCreate a new entity within the specified collection
PUTUpdate the specified entity
DELETEDelete the specified entity, query, or collection of entites

Kinvey Data Link forwards REST requests to your Data Link Connector using endpoints based on the collection name of the Data Link Data Integration collection. HTTP verbs may address the collection name or an ID. PUT and POST may also contain JSON bodies - however, this is not required.

_count

The _count endpoint identifies a method to return a count of all entities in a collection. While not strictly required, it is necessary in order to view data from a Data Link in the Kinvey Console's Data Browser. The _count endpoint should return a single JSON object containing the count attribute, with a 200 HTTP Response code.

{
  count: 150
}

The count can also take a query object, which should return the count for that specific query.

Queries, Sorting, and Pagination

Kinvey Data Link passes any parameters passed via the Kinvey querying API directly to your Data Link adaptor for GET and DELETE requests.

You can use any of Kinvey's operators and translate them to your own operations for accessing the data source. Details on the Kinvey Query API can be found in the REST API Data Store Guide. The Data Link Connector architecture supports the following query string parameters:

ParameterDescriptionExample
queryKinvey query filterquery={"country":"USA"}
sortSort the result set by one or more propertiessort={age: 1}
limitSet a maximum number of results to return.limit=10
skipSkips the first n that match the query. Used in conjunction with limitskip=20

Query

Queries are a JSON representation of the result query you want to return. The full query guide can be found at REST API Data Store Guide

For example, to pass a first name and last name as parameters to your Data Link Connector you can use

/appdata/:appkey/:collectionName/?query={"firstName": "Bob", "lastName": "Smith"}

Sort

A sort parameter can be passed to sort the result set. The sort parameter can be specified by either a single field:

?sort=age

which should sort by the supplied value in ascending order, or or a JSON object containing the fields to sort by in order, with a value of 1 for ascending and -1 for descending. For example:

?sort={"age":-1, "lastName":1, "firstName":1}

will sort first by age in descending order, then by lastName and firstName in ascending order.

Limit and Skip

Limit and skip can be used to limit the size of the result set, or to skip over the first n records. They can be used independently or together.

For example, to limit the result set to the first 1,000 records:

?limit=1000

You can also skip over the first n results in the result set. For example, to ignore the first returned result:

?skip=1

Limit and skip can be used together for pagination. For example:

?limit=20&skip=0
?limit=20&skip=20
?limit=20&skip=40

should retrieve 20 records at a time, skipping over the previously retrieved result set each time.

Request Headers

API Key

A shared secret can be configured in the Kinvey Console to be used as a basic means of secure handshake between Kinvey and the DLC. This can be anything, but it is recommended that for production use, it should be a random shared secret of at least 32 randomly-generated characters made up of mixed-case letters, numbers, and symbols. The DLC is responsible for validating the key included in the request header x-auth-key.

Client App Version

The client application can optionally set the client app version to be included in the request. This is included as a string in the optional HTTP Header x-kinvey-client-app-version. The client application version is recommended (but not required) to be in the format of major.minor.patch (e.g. 3.5.15).

Custom Request Properties

The client application can optionally include custom request properties that are passed to the DLC. This is passed as a stringified JSON object of property:value pairs in the optional HTTP Header `x-kinvey-custom-request-properties'.

Identity Token

If Mobile Identity Connect is used and an enterprise identity token (SAML, OAuth, custom) is used, it will be included in the DLC request as a base64 encoded string in the optional X-Kinvey-Auth HTTP header.

Response Entity Format

Response Types

The DLC should return either a JSON object or an Array of JSON objects, depending on the request method. The below table lists the response types for the various endpoints for GET, POST, and PUT methods:

EndpointMethodResponse TypeDescription
/:collectionName/GETArrayAn array of the entities retrieved
/:collectionName/?query=GETArrayAn array of the entities retrieved, even if only a single entity is returned.
/:collectionName/POSTObjectThe object that was created
/:collectionName/:idGETObjectThe requested object
/:collectionName/:idPUTObjectThe updated object

Requests that return no data should include an empty object {} or an emtpy array [].

DELETE and _count should return the number of records affected in the format:

{ 
  "count": 15
}

Response Attributes

Your Data Link Connector is required to return data in the JSON format if any data is returned, also each data item returned must have the following attributes included:

AttributeDescription
_idThe unique identifier for the entity (Primary Key). This must be typed as a String.
_kmdKinvey Metadata. This includes create time (ect) and last modified time (lmt)
_aclThis defines access control methods in Kinvey. For DLC purposes, it should be returned as an empty object {}

The ect and lmt should be in ISO-8601 formatted UTC time in the format YYYY-MM-DDThh:mm:ss.sssZ, for example:

2015-03-23T13:03:39.475Z

A skeleton Kinvey Entity without any additional attributes would look similar to:

  {
    "_id": "55100f2bb928bfef2b0181a7",
    "_acl": {},
    "_kmd": {
      "lmt": "2015-03-23T13:03:39.475Z",
      "ect": "2015-03-23T13:03:39.475Z"
    }
  }

Response Codes

Your Data Link Connector is not required to respond to all methods, and should respond with an HTTP status code of 403 (Forbidden) if you don't implement a method.

Your Data Link host may return the following HTTP status codes:

CodeMeaning
200Ok, request completed successfully
201Created, to indicate a POST created an object
301Redirection to other resource, 'Location' header must have new location
302Redirection to other resource, 'Location' header must have new location
400Bad Request received
401Unauthorized (if you implement an authorization strategy, or if the x-auth-key is incorrect)
403Forbidden, used when an endpoint/method is not implemented
404Resource not found
500Internal Data Link error

Other codes should not be used, as they will not be handled by Kinvey's client libraries.

Configuring a DLC in the Console

Configuring a DLC requires two discrete steps - setting up the Data Link Connector and configuring collections to use the Data Link Connector.

Setting up the Data Link Connector

Datalink Configuration 1

To set up a Data Link Connector in the Kinvey Console, first go to the "Data Link" section in the left-hand navbar, and then click "New Data Link". Give the Data Link a name, and select "Custom" from the list of options. Configuring the Data Link service requires two parameters, the hostname where the Data Link Connector is running and a shared secret key used for authentication.

The hostname field must contain the protocol, host and port that Kinvey will connect to make requests via the Data Link. Several examples of hostnames:

http://my.devhost.com:1337
https:/my.prodhost.com
https://host.example.com:8888/prefix

The key as an arbitrary string that will be sent via the x-auth-key header. You can use this key to verify the request is coming from Kinvey, or to dispatch several Data Link Connectors on the same physical host. The key is not meant for user authentication, but as an initial verification of traffic source.

Your key will be sent over the internet in cleartext if you don't have a TLS configured host and if you haven't specified 'https' for your protocol. If you use http for development be sure to change your key when you switch to production https.

Mapping Collections to a DLC

Datalink Configuration 1

To map a collection to a specific DLC, create the collection and select the DLC that you want to map that collection to. The collection name will map to the specific :collectionName endpoint in the DLC. For example, if your DLC base url is 'https://somecompany.com/datalink', and your collection name is Contacts, the request will be made to `https://somecompany.com/datalink/contacts'

Once you have mapped a collection to a DLC, you can access that collection through any of the Kinvey Client Libraries, or via the REST API as you would any Kinvey collection. If the _count route is implemented in your DLC, you can view your data via the data browser in the console.

Reference Implementations

Data Link for Oracle

Kinvey's reference implementation of the Data Link Connector for Oracle is a RESTful Java Web Service that adheres to the Data Link protocol built using production quality open source tools:

  • Jersey to provide REST capabilities
  • Spring Framework for object graph dependency management and database access
  • Hibernate for its JPA implementation and underlying data store abstraction
  • Jackson for JSON parsing/marshalling
  • Maven for managing project dependencies

Please contact us for a copy of the reference implementation source code.

Data Link for Microsoft Dynamics CRM

Microsoft Dynamics® CRM is a highly configurable CRM system which can be tailored to each organization. Client applications integrating with Dynamics via Kinvey have many demanding requirements and rely on many Kinvey specific features. To maintain this flexibility, Kinvey provides a highly customizable reference implementation of a Dynamics data link connector.

The reference implementation utilizes several open source tools and frameworks:

Please contact us to enquire about receiving a copy of the reference data link connector.

Data Link for Oracle PeopleSoft

PeopleSoft provides a collection of Human Resource Management Systems (HRMS), Financial Management Solutions (FMS), Supply Chain Management (SCM), Customer Relationship Management (CRM), and Enterprise Performance Management (EPM) software systems. These systems are highly configurable and can be tailored to each organization. Client applications integrating with PepleSoft via Kinvey have many demanding requirements and rely on many Kinvey specific features. To maintain this flexibility, Kinvey provides a customizable reference implementation of a PeopleSoft data link connector.

The reference implementation is built using the Node.js platform.

Please contact us to enquire about receiving a copy of the reference data link connector.

Data Link for SOAP

Client applications integrating with SOAP APIs via Kinvey have many demanding requirements and rely on many Kinvey specific features. To maintain this flexibility, Kinvey provides a customizable reference implementation of a SOAP data link connector.

The reference implementation is built using the Node.js platform.

Please contact us to enquire about receiving a copy of the reference data link connector.

Data Link for XML

Client applications integrating with XML-based APIs via Kinvey have many demanding requirements and rely on many Kinvey specific features. To maintain this flexibility, Kinvey provides a customizable reference implementation of a XML data link connector.

The reference implementation is built using the Node.js platform.

Please contact us to enquire about receiving a copy of the reference data link connector.

Got a question?