Caching and Offline

The content in this guide refers to the newly released version 3 of the library. You can still access the version 1.x guides here.

Version 1.x is now deprecated. We will continue to support 1.x through March 31, 2017 (End of Support - EOS). If you are currently using 1.x in your apps, you will need to upgrade your apps to 3.x before the EOS date. We have created a migration guide to help you.

This guide discusses advanced setup for the DataStore. We assume that you have already reviewed the basic setup steps in the DataStore guide.

Selecting a DataStoreType is usually sufficient to solve the caching and offline needs of most apps. However, should you desire more control over how data is managed in your app, you can use the granular configuration options provided by the library. The following sections discuss the advanced options available on the DataStore.

Data Reads

There are two main ways to control Data Reads - setting the DataStoreType and setting a time to live (ttl).

Data Store Type

When you read data, the type of datastore you use determines how the data gets read.

For a dataStore of type Sync, data is read from the local cache.

For a dataStore of type Cache, data is read first from the local cache. After data is read from the cache, a request is made to the backend. The data in the response is saved to the local cache and returned to the user.

For a dataStore of type Network, data is read directly to the backend and not saved to the local cache.

Examples

Let's assume that you are using a datastore with caching enabled, but want to load data from the backend without loading data from the cache. This can be achieved by calling pull().

var dataStore = Kinvey.DataStore.collection('books', Kinvey.DataStoreType.Sync);
dataStore.pull()
  .then(function(entities) {
    // entities were loaded from the network and saved to the cache
  })
  .catch(function(error) {
    // ...
  });

TTL

The ttl (time to live) option can be used to control how long entities in the cache are considered fresh. Entities in the cache that are older than the ttl set on the store are considered expired; and are never returned to the app.

The default ttl set on the Cache store is infinity (i.e. entities in the cache never expire).

Example

// Set the TTL for the data store to 1 hour
var dataStore = Kinvey.DataStore.collection('books', Kinvey.DataStoreType.Sync);
dataStore.ttl = 3600; // 1 hour

Delta Set Caching

The library implements a mechanism to optimize the amount of data retrieved from the backend. When you use a Sync or Cache datastore, data requests to the backend only fetch data that changed since the previous update. We call this mechanism "Delta Set Caching".

Delta set caching applies to Sync and Cache data stores. To use delta set caching, you need to enable it on the store.

// Enable delta set caching on a data store
dataStore.useDeltaFetch = true;

How it works

Each DataStore maintains metadata per entity such as the last time the entity was updated. When the library requests data from the backend, it makes multiple calls to the backend - first to retrieve the metadata of each entity, and subsequently (if required) to fetch only those entities that changed since the previous update.

Discussion

  • To request a delta update from the backend, the library will make the following two API calls -

    • To retrieve the metadata for entities, the original request for data is appended with the fields modifier, as follows:
    # only retrieve the _id and lmt fields for entities that match the query
    ?query={...}&fields=_id,_kmd.lmt
    • To retrieve the specific entities that have changed, the library makes a request with the in query filter. If necessary, the library may batch entities into multiple requests, to avoid hitting platform limits on URL length.
    # retrieve specific entities by id
    ?query={"_id":{"$in": ["<entity1.id>", "<entity2.id>", ...]}}

Delta set caching requires that your backend support the query modifiers described above - fields and $in. All Kinvey out-of-the-box connectors are built to support fields and $in. To use delta set caching, custom connectors are required to support these query modifiers.

  • The performance benefits of delta set caching are most noticeable on large collections that have low rate of change. That said, our tests show that in most cases, performance is improved with delta set caching enabled.

Data Writes

Write Policy

When you save data, the type of datastore you use determines how the data gets saved.

For a store of type Sync, data is written to your local copy. In addition, the library maintains additional information in a "pending writes queue" to recognize that this data needs to be sent to the backend when you decide to sync or push.

For a store of type Cache, data is written first to your local copy and sent immediately to be written to the backend. If the write to the backend fails (e.g. because of network connectivity), the library maintains information to recognize that this data needs to be sent to the backend when connectivity becomes available again. Due to platform limitations, this does not happen automatically, but needs to be initiated from the user by calling the push() or sync() methods.

For a store of type Network, data is sent directly to the backend. If the write to the backend fails, the library does not persist the data for a future write.

Conflicts

When using sync and cache stores, you need to be aware of situations where multiple users could be working on the same entity simultaneously offline. Consider the following scenario:

  1. User X edits entity A offline.
  2. User Y edits entity A offline.
  3. Network connectivity is restored for X, and A is synchronized with Kinvey.
  4. Network connectivity is restored for Y, and A is synchronized with Kinvey.

In the above scenario, the changes made by user X are overwritten by Y.

The libraries and backend implement a default mechanism of "client wins", which implies that the data in the backend reflects the last client that performed a write. Custom conflict management policies can be implemented with Business Logic.

Timeout

When performing any datastore operations, you can pass a timeout value as an option to stop the datastore operation after some amount of time if it hasn't already completed.

var promise = dataStore.save({
    _id: 'optional-id',
    field: 'value'
}, {
    timeout: 5000 // 5 seconds in ms
})
  .then(function(entity) {
    // ...
  })
  .catch(function(error) {
    // ...
  });

The global default timeout in the SDK is set to 60 seconds. You can set the global timeout to your own value when you initialize the SDK.

Kinvey.initialize({
  appKey: '<appKey>',
  appSecret: '<appSecret>',
  defaultTimeout: 30000 // 30 seconds in ms
});
Got a question?