Caching and Offline

This guide discusses advanced setup for the DataStore. We assume that you have already reviewed the basic setup steps in the DataStore guide.

Selecting a DataStoreType is usually sufficient to solve the caching and offline needs of most apps. However, should you desire more control over how data is managed in your app, you can use the granular configuration options provided by the library. The following sections discuss the advanced options available on the DataStore.

Data Reads

The way to control Data Reads is to set a ReadPolicy.

Read Policy

ReadPolicy controls how the library fetches data. When you select a store configuration, the library sets the appropriate ReadPolicy on the store. However, individual reads can override the default read policy of the store by specifying a ReadPolicy.

The following read policies are available:

  • ForceLocal - forces the datastore to read data from local storage. If no valid data is found in local storage, the request fails.
  • ForceNetwork - forces the datastore to read data from the backend. If network is unavailable, the request fails.
  • Both - reads first from the local cache and then attempts to get data from the backend.

Examples

Let's assume that you are using a datastore with caching enabled, but want force a certain find request to fetch data from the backend. This can be achieved by specifying a ReadPolicy when you call the find API on your store.

Delta Set Caching

The library implements a mechanism to optimize the amount of data retrieved from the backend. When you use a Sync or Cache datastore, data requests to the backend only fetch data that changed since the previous update. We call this mechanism "Delta Set Caching".

Delta set caching applies to Sync and Cache data stores. To use delta set caching, you need to enable it on the store.

// Create a data store
DataStore<ToDo> todoStore = DataStore<ToDo>.Collection(collectionName, DataStoreType.SYNC);

// Enable delta set fetching on this store
todoStore.DeltaSetFetchingEnabled = true;

How it works

Each DataStore maintains metadata per entity such as the last time the entity was updated. When the library requests data from the backend, it makes multiple calls to the backend - first to retrieve the metadata of each entity, and subsequently (if required) to fetch only those entities that changed since the previous update.

Discussion

  • To request a delta update from the backend, the library will make the following two API calls -

    • To retrieve the metadata for entities, the original request for data is appended with the fields modifier, as follows:

      # only retrieve the _id and lmt fields for entities that match the query
      ?query={...}&fields=_id,_kmd.lmt
    • To retrieve the specific entities that have changed, the library makes a request with the in query filter. If necessary, the library may batch entities into multiple requests, to avoid hitting platform limits on URL length.

      # retrieve specific entities by id
      ?query={"_id":{"$in": ["<entity1.id>", "<entity2.id>", ...]}}

Delta set caching requires that your backend support the query modifiers described above - fields and $in. All Kinvey out-of-the-box connectors are built to support fields and $in. To use delta set caching, custom connectors are required to support these query modifiers.

  • The performance benefits of delta set caching are most noticeable on large collections that have low rate of change. That said, our tests show that in most cases, performance is improved with delta set caching enabled.

Data Writes

Write Policy

When you save data, the type of datastore you use determines how the data gets saved.

For a store of type Sync, data is written to your local copy. In addition, the library maintains additional information in a "pending writes queue" to recognize that this data needs to be sent to the backend when you decide to sync or push.

For a store of type Cache, data is written first to your local copy and sent immediately to be written to the backend. If the write to the backend fails (e.g. because of network connectivity), the library maintains information to recognize that this data needs to be sent to the backend when connectivity becomes available again. Due to platform limitations, this does not happen automatically, but needs to be initiated from the user by calling the push() or sync() methods.

For a store of type Network, data is sent directly to the backend. If the write to the backend fails, the library does not persist the data for a future write.

Conflicts

When using sync and cache stores, you need to be aware of situations where multiple users could be working on the same entity simultaneously offline. Consider the following scenario:

  1. User X edits entity A offline.
  2. User Y edits entity A offline.
  3. Network connectivity is restored for X, and A is synchronized with Kinvey.
  4. Network connectivity is restored for Y, and A is synchronized with Kinvey.

In the above scenario, the changes made by user X are overwritten by Y.

The libraries and backend implement a default mechanism of "client wins", which implies that the data in the backend reflects the last client that performed a write. Custom conflict management policies can be implemented with Business Logic.

Timeout

When performing any datastore operations, you can pass a timeout value as an option to stop the datastore operation after some amount of time if it hasn't already completed.

Got a question?