Data Store

The simplest use case of Kinvey is storing and retrieving data to and from your cloud backend.

The basic unit of data is an entity and entities of the same kind are organized in collections. An entity is a set of key-value pairs which are stored in the backend in JSON format. Kinvey's libraries automatically translate your native objects to JSON.

Kinvey's data store provides simple CRUD operations on data, as well as powerful filtering and aggregation.

Collections

To start working with a collection, you need to instantiate the library-provided Kinvey.Collection class which extends Backbone.Collection. Each instance represents a collection on your backend.

By default, the model class of the Kinvey.Collection is Kinvey.Model.

// Define a new class
var MyCollection = Kinvey.Collection.extend({
  url: 'collection-name'
});

// Instantiate a new collection.
var collection = new MyCollection([]);

Optionally, you can configure the collection type when creating it. To understand the types of collection and when to use each type, refer to the DataStore Types section.

DataStore Types

When you get an instance of a collection in your application, you can optionally select a datastore type. The type has to do with how the library handles intermittent or prolonged interruptions in connectivity to the backend.

Choose a type that most closely resembles your data requirements.

  • DataStoreType.Sync: You want a copy of some (or all) the data from your backend to be available locally on the device and you like to sync it periodically with the backend.
  • DataStoreType.Cache: You want data stored on the device to optimize your app’s performance and/or provide offline support for short periods of network loss. This is the default DataStore type if a type is not explicitly set.
  • DataStoreType.Network: You want the data in your backend to never be stored locally on the device, even if it means that the app will not offer offline use.

Head to the Specify DataStore Type section to learn how to request the DataStore type that you selected.

Entities

The Kinvey service has the concept of entities, which represent a single resource. In Backbone, such a resource is represented by a Backbone.Model.

The library provides the Kinvey.Model class to represent a single entity in your project. This class extends Backbone.Model. It is recommended to specify a collection as options, or set the urlRoot on the model. The collection points to a Backbone.Collection, while urlRoot should hold the collection name.

// Define a new Model class with a urlRoot option
var MyModel = Kinvey.Model.extend({
  urlRoot: 'collection-name'
});

/*
// Define a new Model class with a collection as an options
var MyModel = Kinvey.Model.extend({
  collection: MyCollection
});
*/

// Instantiate a new model.
var model = new MyModel({});

Fetching

You can retrieve entities by either looking them up using an id, or by querying a collection.

The Kinvey JavaScript Library uses the observable pattern to return data on find operations. APIs that fetch data require the application to register a subscriber that gets invoked when data is available. We use Observables to accomplish this.

When data is retrieved from multiple sources (i.e. cache and backend), the subscriber gets invoked multiple times - first with the results from the cache; then with the results from the backend.

Fetching by Id

To fetch an (one) entity by id, call dataStore.findById.

var stream = dataStore.findById('entity-id');
stream.subscribe(function onNext(entity) {
  // ...
}, function onError(error) {
  // ...
}, function onComplete() {
  // ...
});

Fetching by Query

To fetch all entities in a collection, call dataStore.find.

var stream = dataStore.find();
stream.subscribe(function onNext(entities) {
  // ...
}, function onError(error) {
  // ...
}, function onComplete() {
  // ...
});

To fetch multiple entities using a query, call dataStore.find and pass in a query.

var query = new Kinvey.Query();
query.equalTo('field', 'value');
var stream = dataStore.find(query);
stream.subscribe(function onNext(entities) {
  // ...
}, function onError(error) {
  // ...
}, function onComplete() {
  // ...
});

Saving

You can save an entity by calling dataStore.save.

var promise = dataStore.save({
  _id: 'optional-id',
  field: 'value'
}).then(function onSuccess(entity) {
  // ...
}).catch(function onError(error) {
  // ...
});

The library uses the _id property of the entity to distinguish between updates and inserts.

  • If the entity has an _id, the library treats it as an update to an existing entity.

  • If the entity does not have an _id, the library treats it as a new entity. The Kinvey backend assigns an automatic _id for a new entity.

Deleting

To delete an entity, call dataStore.removeById and pass in the entity _id.

var promise = dataStore.removeById('entity-id')
promise = promise.then(function onSuccess(result) {
  // ...
}).catch(function onError(error) {
  // ...
});

Deleting Multiple Entities at Once

To delete multiple entities at once, call dataStore.remove. Optionally, you can pass in a query to only delete entities matching the query.

var query = new Kinvey.Query();
query.equalTo('property', 'value');

var promise = dataStore.remove(query)
promise = promise.then(function onSuccess(result) {
  // ...
}).catch(function onError(error) {
  // ...
});

Metadata

Every model has metadata associated with it. There are a number of methods exposed which allow you to easily extract metadata from the model.

  • model.getCreatedAt() returns the Date when the model was created on Kinvey.
  • model.getLastModified() returns the Date when the model was last updated on Kinvey.
  • model.getAcl() returns the Acl object. Read the Security guide on Entity Level Permissions for more information.

Querying

The Kinvey.Query class allows you to build queries for use in collections. An empty query, by default, matches all models in a collection.

var query = new Kinvey.Query();

Operators

All operator methods as exposed by Kinvey.Query follow the same pattern: the first argument must be the field under condition, while the other arguments specify the exact condition on that field. All operators return the query itself, so it is easy to concatenate multiple conditions on one line.

For example, to select all models with a rate between 25 and 50:

var query = new Kinvey.Query();
query.greaterThanOrEqualTo('rate', 25).lessThanOrEqualTo('rate', 50);

Comparison Operators

  • equalTo matches if the field is = the supplied value.
  • greaterThan matches if the field is > the supplied value.
  • greaterThanOrEqualTo matches if the field is >= the supplied value.
  • lessThan matches if the field is < the supplied value.
  • lessThanOrEqualTo matches if the field is <= the supplied value.
  • notEqualTo matches if the field is != the supplied value.
  • exists matches if the field exists.
  • mod matches if the field modulo the supplied divisor (second argument) has the supplied remainder (third argument).
  • matches matches if the field matches the supplied regular expression.

Regular expressions need to be anchored (prefixed with ^), and case sensitive. To do case insensitive search, create a normalized (i.e. all lowercase) field in your collection and perform the match on that field.

Array Operators

  • contains matches if any of the supplied values is an element in the field.
  • containsAll matches if the supplied values are all elements in the field.
  • notContainedIn matches if the supplied value is not an element in the field.
  • size matches if the number of elements in the field equals the supplied value.

Modifiers

Query modifiers control how query results are presented. A distinction is made between limit, skip, and sort modifiers.

Limit and Skip

Limit and skip modifiers allow for paging of results. Set the limit to the number of results you want to show per page. The skip modifier indicates how many results are skipped from the beginning.

// Show results 20–40
var query = new Kinvey.Query();
query.limit = 20;
query.skip = 20;

Sort

Query results are sorted either in ascending or descending order. It is possible to add multiple fields to sort on.

// Sort on last name (ascending), then on age (descending).
var query = new Kinvey.Query();
query.ascending('last_name');
query.descending('age');

Data is sorted lexicographically, meaning B comes before a, and 10 before 2.

Field Selection

By default, all fields in an entity will be retrieved. You can, however, specify specific fields to retrieve. This can be useful to save bandwidth.

var query = new Kinvey.Query();
query.fields = [ 'last_name', 'age' ];

Saving entities after retrieving them using Field Selection will result in the loss of all fields not selected. Further, these partial entities will not be available for use with Caching & Offline Saving.

Compound Queries

You can combine filters with modifiers within a single query.

// Returns the first five users with last_name “Doe”, sorted by first_name.
var query = new Kinvey.Query();
query.limit = 5;
query.equalTo('last_name', 'Doe');
query.ascending('first_name');

Joining Operators

It is very easy to join multiple queries into one. In order of precedence, the three joining operators are listed below in order of precendence.

  • and joins two or more queries using a logical AND operation.
  • nor joins two or more queries using a logical NOR operation.
  • or joins two or more queries using a logical OR operation.

The example below demonstrates how to join two separate queries.

var query = new Kinvey.Query();
query.equalTo('last_name', 'Doe');
var secondQuery = new Kinvey.Query();
secondQuery.equalTo('last_name', 'Roe')

// Selects all users with last_name “Doe” or “Roe”.
query.or(secondQuery);

Alternatively, the snippet above can be shortened using the join operator inline.

// Selects all users with last_name “Doe” or “Roe”.
var query = new Kinvey.Query();
query.equalTo('last_name', 'Doe').or().equalTo('last_name', 'Roe');

You can build arbitrary complex queries using any join operators. The rule of thumb is to take the precendence order into account when building queries to make sure the correct results are returned.

Counting

To count the number of entities in a collection, call dataStore.count. Optionally, you can pass in a query to only count the entities matching the query.

var stream = dataStore.count(query);
stream.subscribe(function(count) {
  // ...
});

Aggregation/Grouping

Grouping allows you to collect all entities with the same value for a field or fields, and then apply a reduce function (such as count or average) on all those items.

The results are returned as an object literal that represents the list of groups containing the result of the reduce function.

For example, let’s group a collection of companies based on the year they launched.

var stream = dataStore.group(Kinvey.Aggregation.count('launched'));
stream.subscribe(function(result) {
  // Result = [{ count: 1 }] if one company has launched
});

Reduce Function

There are two pre-defined reduce functions.

  • Kinvey.Aggregation.count counts the number of elements in the group.
  • Kinvey.Aggregation.sum sums together the numeric values of the supplied field.

Scoping With Queries

Groups can also take an optional condition. This is a query that acts as a filter that is applied on the server before the reduce function is evaluated. Any limit, skip, and sort modifiers are applied after the reduce function is evaluated.

In our above example, assume we wanted to group only the companies who have offices in Boston. In addition, we want to order the resulting groups by the year they launched and only display the first two. It would look like:

// Build the query and group.
var query = new Kinvey.Query();
query.equalTo('city', 'Boston');
query.ascending('launched')
query.limit = 2;

var aggregation = Kinvey.Aggregation.count('launched');
aggregation.query = query;

var stream = dataStore.group(aggregation);
stream.subscribe(function(result) {
  // ...
});

Location Querying

See the Location guide for information on how to query data by location.

Caching and Offline

A key aspect of good mobile apps is their ability to render responsive UIs, even under conditions of poor or missing network connectivity. The Kinvey library provides caching and offline capabilities for you to easily manage data access and synchronization between the device and the backend.

Kinvey’s DataStore provides configuration options to solve common caching and offline requirements. If you need better control, you can utilize the options described in Granular Control.

Specify DataStore Type

Sync

Configuring your collection as a Sync DataStore Type allows you to pull a copy of your data to the device and work with it completely offline. The library provides APIs to synchronize local data with the backend.

This type of collection is ideal for apps that need to work for long periods without a network connection.

Here is how you'd use a Sync collection:

// Create a collection
var MyCollection = Kinvey.Collection.extend({
  url: 'collection-name',
  dataStoreType: Kinvey.DataStoreType.Sync
});
var collection = new MyCollection([]);

// Pull data from the backend and save it to the cache.
var promise = collection.pullFromKinvey({
  success: function(collection, response, options) {
    // ...
  }
});

// Find entities in the cache.
collection.fetch({
  query: query,
  success: function(collection, response, options) {
    // ...
  }
});

// Save a model to the cache. This model can be synced with
// the backend at a later time.
var model = new Kinvey.Model({});
model.save({
  title: 'MyBook'
}, {
  success: function(collection, response, options) {
    collection.add(model);
    // ...
  }
});

// Sync the collection with the backend.
// This will first push any pending local changes to the backend and
// then pull data from the backend and save it in the cache.
collection.syncWithKinvey({
  success: function(collection, response, options) {
    // ...
  }
});

The pullFromKinvey, pushToKinvey, syncWithKinvey, and syncCount APIs allow you to synchronize data between the application and the backend. This section describes how these APIs work with examples.

Usage

The Sync Collection is a used when: The app is going to be offline for an extended period of time (hours to days) Data service is poor/expensive Mobile data synchronization is a difficult problem to solve and should only be used when necessary.

The two main use cases for Mobile Data Synchronization are: Caching data from the cloud for use during the day Creating data during the data and sending it to the cloud at the end of the day.

Pull Operation

Calling pullFromKinvey retrieves data from the backend and stores it locally in the Sync Collection.

By default, pulling will retrieve the entire collection to the device. Optionally, you can provide a query parameter to pullFromKinvey to restrict what entities are retrieved.

The pullFromKinvey API needs a network connection in order to succeed.

// Create a collection
var MyCollection = Kinvey.Collection.extend({
  url: 'collection-name',
  dataStoreType: Kinvey.DataStoreType.Sync
});
var collection = new MyCollection([]);

// Pull data from the backend and save it in the cache.
collection.pullFromKinvey({
  success: function(collection, response, options) {
    // ...
  }
});

If your Sync Collection has pending local changes, they must be pushed to the backend before pulling data to the collection.

Push Operation

Calling pushToKinvey kicks off a uni-directional push of data from the library to the backend.

The library goes through the following steps to push entities modified locally to the backend -

  • Reads from the "pending writes queue" to determine what entities have been changed locally. The "pending writes queue" maintains a reference for each entity locally that has been modified by the app. For an entity that gets modified multiple times locally, the queue only references the last modification on the entity.

  • Creates a REST API request for each pending change in the queue. The type of request depends on the type of modification that was performed locally on the entity.

    • If a model is newly created, the library builds a POST request.

    • If a model is modified, the library builds a PUT request.

    • If a model is deleted, the library builds a DELETE request.

  • Makes the REST API requests against the backend concurrently. Requests are batched to avoid hitting platform limits on the number of open network requests.

    • For each successful request, the corresponding reference in the queue is removed.

    • For each failed request, the corresponding reference remains persisted in the queue. The library adds information in the push/sync response to indicate that a failure occurred.

  • Returns a response to the application indicating the count of entities that were successfully synced, and a list of errors for entities that failed to sync.

// Create a collection
var MyCollection = Kinvey.Collection.extend({
  url: 'collection-name',
  dataStoreType: Kinvey.DataStoreType.Sync
});
var collection = new MyCollection([]);

// Push data from the cache that has changed to the backend.
// No data is retrieved from the backend.
collection.pushToKinvey({
  success: function(collection, response, options) {
    // ...
  }
});
Sync Operation

Calling sync() on a Sync Collection kicks off a bi-directional synchronization of data between the library and the backend. First, the library calls push to send local changes to the backend. Subsequently, the library calls pull to fetch data in the collection from the backend and stores it on the device.

You can provide a query as a parameter to the syncWithKinvey API, to restrict the data that is pulled from the backend. The query does not affect what data gets pushed to the backend.

// Create a collection
var MyCollection = Kinvey.Collection.extend({
  url: 'collection-name',
  dataStoreType: Kinvey.DataStoreType.Sync
});
var collection = new MyCollection([]);

// Sync the cache and backend.
collection.syncWithKinvey({
  success: function(collection, response, options) {
    // ...
  }
});


// Use a query to sync only a subset of data with the backend.
var query = new Kinvey.Query();
query.equalTo('field', 'value');

collection.syncWithKinvey({
  query: query,
  success: function(collection, response, options) {
    // ...
  }
});
Sync Count Operation

You can retrieve a count of entities modified locally and pending a push to the backend.

// Create a collection
var MyCollection = Kinvey.Collection.extend({
  url: 'collection-name',
  dataStoreType: Kinvey.DataStoreType.Sync
});
var collection = new MyCollection([]);

// Number of models modified offline.
collection.syncCount({
  success: function(collection, response, options) {
    // ...
  }
});

Cache

Configuring your collection as a Cache collection allows you to use the performance optimizations provided by the library. The library is smart enough to determine when to serve data from the cache and when to fetch it from the network. In addition, the cache allows you to work with data when the device goes offline.

This type of collection is ideal for apps that are generally used with an active network, but may experience short periods of network loss.

Here is how you'd use a Cache store -

// Create a collection
var MyCollection = Kinvey.Collection.extend({
  url: 'collection-name',
  dataStoreType: Kinvey.DataStoreType.Cache
});
var collection = new MyCollection([]);

// Pull data from the backend and save it to the cache.
var promise = collection.pullFromKinvey({
  success: function(collection, response, options) {
    // ...
  }
});

// Find entities in the cache and your backend
collection.fetch({
  query: query,
  success: function(collection, response, options) {
    // ...
  }
});

// Save a model to the cache and to your backend
var model = new Kinvey.Model({});
model.save({
  title: 'MyBook'
}, {
  success: function(collection, response, options) {
    collection.add(model);
    // ...
  }
});

The Cache Store executes all CRUD requests against local storage as well as the backend. Any data retrieved from backend is stored in the cache. This allows the app to work offline by fetching data that has been cached from past usage.

The Cache Store also stores pending write operations when the app is offline. However, the developer is required to push these pending operations to the backend when the network resumes. The push API should be used to accomplish this.

Usage

The Cache collection is the default data store type for all Kinvey collections. Cache accelerates application performance and does not have drawbacks for the majority of mobile application use cases.

Offline cache is useful for short periods of disconnection (seconds to minutes). When the phone loses conncetivity basic CRUD operations (find, save, delete) will continue to work. Once connectivity is restored push and pull can be used to get the device back into sync with the cloud if needed.

Network

Configuring your datastore as Network turns off all caching in the library. All requests to fetch and save data are sent to the backend.

We don’t recommend this type of datastore for apps in production, since the app will not work without network connectivity. However, it may be useful in a development scenario to validate backend data without a device cache.

Here is how you would use a Network store -

// Create a collection
var MyCollection = Kinvey.Collection.extend({
  url: 'collection-name',
  dataStoreType: Kinvey.DataStoreType.Network
});
var collection = new MyCollection([]);

// Find entities on your backend.
collection.fetch({
  query: query,
  success: function(collection, response, options) {
    // ...
  }
});

// Save a model to your backend.
var model = new Kinvey.Model({});
model.save({
  title: 'MyBook'
}, {
  success: function(collection, response, options) {
    collection.add(model);
    // ...
  }
});
Usage

The Network Collection is used when: Data has to be up to date at all times Data cannot be stored on the mobile device

Application performance will suffer from network latency and this can be problematic for apps that make many calls. It is recommend that this data store is only used when necessary for business requirements or an application use case.

Granular Control

Selecting a DataStoreType is usually sufficient to solve the caching and offline needs of most apps. However, should you desire more control over how data is managed in your app, you can use the granular configuration options provided by the library. The following sections discuss the advanced options available on the DataStore.

In this section:

Data Reads

There are two main ways to control Data Reads - setting the DataStoreType and setting a time to live (ttl).

Data Store Type

When you read data, the type of datastore you use determines how the data gets read.

For a dataStore of type Sync, data is read from the local cache.

For a dataStore of type Cache, data is read first from the local cache. After data is read from the cache, a request is made to the backend. The data in the response is saved to the local cache and returned to the user.

For a dataStore of type Network, data is read directly from the backend and not saved to the local cache.

Examples

Let's assume that you are using a datastore with caching enabled, but want to load data from the backend without loading data from the cache. This can be achieved by calling pull().

var dataStore = Kinvey.DataStore.collection('books', Kinvey.DataStoreType.Sync);
dataStore.pull()
  .then(function(entities) {
    // entities were loaded from the network and saved to the cache
  })
  .catch(function(error) {
    // ...
  });
TTL

The ttl (time to live) option can be used to control how long entities in the cache are considered fresh. Entities in the cache that are older than the ttl set on the store are considered expired; and are never returned to the app.

The default ttl set on the Cache store is infinity (i.e. entities in the cache never expire).

Example
// Set the TTL for the data store to 1 hour
var dataStore = Kinvey.DataStore.collection('books', Kinvey.DataStoreType.Sync);
dataStore.ttl = 3600; // 1 hour
Delta Set Caching

The library implements a mechanism to optimize the amount of data retrieved from the backend. When you use a Sync or Cache datastore, data requests to the backend only fetch data that has changed since the previous update. We call this mechanism "Delta Set Fetching".

Delta set fetching applies to Sync and Cache data stores. To use delta set fetching, you need to enable it on the store.

// Enable delta set fetching on a data store
dataStore.useDeltaFetch = true;
How it works

Each DataStore maintains metadata per entity such as the last time the entity was updated. When the library requests data from the backend, it makes multiple calls to the backend - first to retrieve the metadata of each entity, and subsequently (if required) to fetch only those entities that changed since the previous update.

Discussion
  • To request a delta fetch from the backend, the library will make the following two API calls -

    • To retrieve the metadata for entities, the original request for data is appended with the fields modifier, as follows:
    # only retrieve the _id and lmt fields for entities that match the query
    ?query={...}&fields=_id,_kmd.lmt
    • To retrieve the specific entities that have changed, the library makes a request with the in query filter. If necessary, the library may batch entities into multiple requests, to avoid hitting platform limits on URL length.
    # retrieve specific entities by id
    ?query={"_id":{"$in": ["<entity1.id>", "<entity2.id>", ...]}}

Delta set fetching requires that your backend support the query modifiers described above - fields and $in. All Kinvey out-of-the-box connectors are built to support fields and $in. To use delta set fetching, custom connectors are required to support these query modifiers.

  • The performance benefits of delta set fetching are most noticeable on large collections that have low rate of change. That said, our tests show that in most cases, performance is improved with delta set fetching enabled.

Data Writes

Write Policy

When you save data, the type of datastore you use determines how the data gets saved.

For a store of type Sync, data is written to your local copy. In addition, the library maintains additional information in a "pending writes queue" to recognize that this data needs to be sent to the backend when you decide to sync or push.

For a store of type Cache, data is written first to your local copy and sent immediately to be written to the backend. If the write to the backend fails (e.g. because of network connectivity), the library maintains information to recognize that this data needs to be sent to the backend when connectivity becomes available again. Due to platform limitations, this does not happen automatically, but needs to be initiated from the user by calling the push() or sync() methods.

For a store of type Network, data is sent directly to the backend. If the write to the backend fails, the library does not persist the data for a future write.

Conflicts

When using sync and cache stores, you need to be aware of situations where multiple users could be working on the same entity simultaneously offline. Consider the following scenario:

  1. User X edits entity A offline.
  2. User Y edits entity A offline.
  3. Network connectivity is restored for X, and A is synchronized with Kinvey.
  4. Network connectivity is restored for Y, and A is synchronized with Kinvey.

In the above scenario, the changes made by user X are overwritten by Y.

The libraries and backend implement a default mechanism of "client wins", which implies that the data in the backend reflects the last client that performed a write. Custom conflict management policies can be implemented with Business Logic.

Timeout

When performing any datastore operations, you can pass a timeout value as an option to stop the datastore operation after some amount of time if it hasn't already completed.

dataStore.save({
  _id: 'optional-id',
  field: 'value'
}, {
  timeout: 5000 // 5 seconds in ms
})
  .then(function(entity) {
    // ...
  })
  .catch(function(error) {
    // ...
  });

The global default timeout in the SDK is set to 60 seconds. You can set the global timeout to your own value when you initialize the SDK.

Kinvey.init({
  appKey: '<appKey>',
  appSecret: '<appSecret>',
  defaultTimeout: 30000 // 30 seconds in ms
});

Related Samples

Got a question?