Data Store

The simplest use case of Kinvey is storing and retrieving data to and from your cloud backend.

The basic unit of data is an entity and entities of the same kind are organized in collections. An entity is a set of key-value pairs which are stored in the backend in JSON format. Kinvey's libraries automatically translate your native objects to JSON.

Kinvey's data store provides simple CRUD operations on data, as well as powerful filtering and aggregation.

Each DataStore in your application represents a collection on your backend. Create a DataStore to start working with the data in a collection.

// we use the example of a "Book" datastore
DataStore<Book> dataStore = DataStore<Book>.Collection("books");

Through the remainder of this guide, we refer to the instance we just retrieved as dataStore.

By default, the library creates a DataStore of type Cache. To understand the types of datastores and when to use each type, please refer to the section on DataStore Types

Entities

The Kinvey service has the concept of entities, which represent a single resource.

using Newtonsoft.Json;
using Kinvey;

[JsonObject(MemberSerialization.OptIn)]
public class Book : Entity
{
  [JsonProperty("title")]
  public string Title { get; set; }
}

Saving

You can save an entity by calling dataStore.save.

try
{
  Book savedBook = await dataStore.SaveAsync(new Book("MyBook"));
}
catch (KinveyException ke)
{
  // handle error
}

The library uses the _id property of the entity to distinguish between updates and inserts.

  • If the entity has an _id, the library treats it as an update to an existing entity.

  • If the entity does not have an _id, the library treats it as a new entity. The Kinvey backend assigns an automatic _id for a new entity.

Fetching

You can retrieve entities by either looking them up using an id, or by querying a collection.

Fetching by Id

To fetch an (one) entity by id, call dataStore.findById.

entity = await dataStore.FindAsync(t.ID);

Fetching by Query

To fetch all entities in a collection, call dataStore.find.

List<Book> books = new List<Book>();

books = await dataStore.FindAsync();

To fetch multiple entities using a query, call dataStore.find and pass in a query.

List<Book> booksByQuery = new List<Book>();

var query = dataStore.Where(x => x.Title.StartsWith("The"));

booksByQuery = await dataStore.FindAsync(query);

Deleting

To delete an entity, call dataStore.removeById and pass in the entity _id.

KinveyDeleteResponse kdr = await dataStore.RemoveAsync(book.ID);

Deleting Multiple Entities at Once

To delete multiple entities at once, call dataStore.remove. Optionally, you can pass in a query to only delete entities matching the query.

Support for Delete By Query is not currently available, but will be coming soon!

Querying

Queries to the backend are performed through LINQ. LINQ provides a number of methods to add filters and modifiers to the query, although the Kinvey-Xamarin library only supports a subset of these operators. Querying is performed on the DataStore by using the FindAsync(query) method, where the query that is passed into the method can be constructed using LINQ syntax.

Under most circumstances, you want to fetch data from your backend that match a specific set of conditions. LINQ provides a mechanism to build up a query for retrieval of a list of items from a collection. For example, you may want to retrieve a sorted list from your "books" collection:

DataStore<Book> dataStore = DataStore<Book>.Collection("books");

var query = from book in bookStore
  orderby book.Title
  select book;

List<Book> booksFromCache = new List<Book>();

// The KinveyDelegate is used to obtain the intermediate results of a find operation.
// In the cache datastore case, the `onSuccess()` callback of the delegate will be
// called back with results returned from the cache, and then the method will return
// with data from the network.
KinveyDelegate<List<Book>> cacheDelegate = new KinveyDelegate<List<Book>>()
{
  onSuccess = (List<Book> results) => booksFromCache.AddRange(results),
  onError = (Exception e) => Console.WriteLine(e.Message)
};

List<Book> booksFromNetwork = await bookStore.FindAsync(query, cacheDelegate);

Operators

In addition to exact match and null queries, you can query based upon several types of expressions: comparison, set match, string and array operations.

Logical Operators

  • && - matches records where both conditions are true
  • || - matches records where either of the conditions are true

Ordering operators

  • orderby <field> - orders the result set by the ascending value of the specified field.

  • orderby <field> descending - orders the result set by the descending value of the specified field.

Kinvey Query Strings

The Kinvey DataStore also provides a FindWithMongoQueryAsync(string kinveyQuery) method, which takes a raw Kinvey-style query string, as an alternative to using the LINQ syntax. We use the MongoDB query syntax, and this method provides complete flexibility for expressing queryies.

Modifiers

Modifiers can be applied to a query in order to control how query results are presented. This includes returning sections of the results, sorting results, and only returning specific fields of an entity.

Limit and Skip

Depending on the size of the results from a query, you may choose to receive the results in smaller sections of the total result set. This is where the concept of limit and skip come into play. Providing a limit size will ensure that the number of results passed back do not exceed the limit given. In conjunction with this, providing a skip count will allow you to offset where your results start, relative to the total result set. Used together, this can achieve pagination of results.

In LINQ, limit is achieved using the Take operator, and skip is done using, appropriately enough, the Skip operator. The example below shows how you would take the first 30 books from the book store in chunks of 10:

var query = bookStore.Where(x => x.Title.StartsWith("How To")).Skip(0).Take(10);
var query = bookStore.Where(x => x.Title.StartsWith("How To")).Skip(10).Take(10);
var query = bookStore.Where(x => x.Title.StartsWith("How To")).Skip(20).Take(10);

Here is a more generic example that shows how to retrieve a large number of entities, one page at a time.

DataStore<Book> bookStore = DataStore<Book>.Collection("books", DataStoreType.SYNC);
int skipCount = 0;
const int pageSize = 10000;
PullDataStoreResponse<Book> pullResponse = null;
do
{
  var query = bookStore.Skip(skipCount).Take(pageSize);
  pullResponse = await productStore.PullAsync(query);
  skipCount += pageSize;
} while (pullResponse.PullCount == pageSize);

The Kinvey backend imposes a limit of 10k entities on a single request to fetch data stored in Kinvey. If you specify pageSize > 10000 in the example above, the backend may reject your request. For this reason, we strongly recommend fetching your data in pages, if your store is likely to exceed the 10k limit on a single fetch from a client.

Sort

As mentioned above in the Ordering Operators section, results from a query can be modified in order to be returned in sorted order (either ascending or descending), based on a particular field.

// ascending
var query = bookStore.Where(x => x.Title.StartsWith("How To")).OrderBy(x => x.Title);

// descending
var query = bookStore.Where(x => x.Title.StartsWith("How To")).OrderByDescending(x => x.Title);

Field Selection

There may be instances where you do not want to retrieve entire entities, but only partial entites containing the fields that you are interested in. To do this, a field selection modifier can be added to a query. In LINQ, this can be accomplished using the select operator to return only a single field.

// single field selection
var query = from book in bookStore
            where book.Details.StartsWith("How To")
            select book.Title;

List<Book> listOfHowToBooks = await bookStore.FindAsync(query);

In the case where multiple fields are required, this can be accomplished in C# using anonymous types.

// multiple field selection using anonymous types
var query = from book in bookStore
            where book.Details.StartsWith("How To")
            select new { book.Title, book.Author };

List<Book> listOfHowToBooks = await bookStore.FindAsync(query);

Saving entities after retrieving them using Field Selection will result in the loss of all fields not selected. In addition, these partial entities will not be available for use with Caching & Offline Saving.

Counting

To count the number of objects in a collection, use the GetCountAsync() method.

uint count = await dataStore.GetCountAsync();

To count the number of objects in a collection that satisfy a query, use GetCountAsync(query).

DataStore Types

A key aspect of good mobile apps is their ability to render responsive UIs, even under conditions of poor or missing network connectivity. The Kinvey library provides caching and offline capabilities for you to easily manage data access and synchronization between the device and the backend.

Kinvey’s DataStore provides configuration options to solve common caching and offline requirements.

When you get an instance of a datastore in your application, select which scenario from the following list most closely resembles your data requirements:

  1. You want a copy of some (or all) the data from your backend to be available locally on the device and you'd like to sync it periodically with the backend then you would use DataStoreType.Sync

  2. You want data stored on the device to optimize your app’s performance and/or provide offline support for short periods of network loss then you would use DataStoreType.Cache

  3. You want the data in your backend to never be stored locally on the device, even if it means that the app cannot support offline usage then you would use DataStoreType.Network

Sync

Configuring your datastore as a Sync datastore allows you to pull a copy of your data to the device and work with it completely offline. The library provides APIs to synchronize local data with the backend.

This type of datastore is ideal for apps that need to work for long periods without a network connection.

Here is how you should use a Sync store -

// Get an instance
string collectionName = "books";
DataStore<Book> dataStore = DataStore<Book>.Collection(collectionName,
                                                       DataStoreType.SYNC);

// Pull data from your backend and save it locally on the device.
List<Book> books = await dataStore.PullAsync();

// Find data locally on the device.
List<Book> books = new List<Book>();

books = await dataStore.FindAsync();

// Save an entity locally on the device. This will add the item to the
// sync table to be pushed to your backend at a later time.
Book book = new Book("My First Book");
await dataStore.SaveAsync(book);

// Sync local data with the backend
// This will push data that is saved locally on the device to your backend; 
// and then pull any new data on the backend and save it locally.
DataStoreResponse syncResponse = await dataStore.SyncAsync();

The pull, push and sync APIs allow you to to synchronize data between the application and the backend. This section describes how these APIs work with examples.

Pull

Calling pull() retrieves data from the backend and stores it locally in the Sync Store.

By default, pulling will retrieve the entire collection to the device. Optionally, you can provide a query parameter to pull to restrict what entities are retrieved. If you prefer to only retrieve the changes since your last pull operation, you should enable delta set caching.

The pull API needs a network connection in order to succeed.

string collectionName = "books";
DataStore<Book> dataStore = DataStore<Book>.Collection(collectionName,
                                                       DataStoreType.SYNC);

// In this example, we pull all the data in the collection
// from the backend to the Sync Store
try
{
  List<Book> books = await dataStore.PullAsync();
}
catch (KinveyException ke)
{
  // handle exception
}

If your Sync Store has pending local changes, they must be pushed to the backend before pulling data to the store.

Push

Calling push() kicks off a uni-directional push of data from the library to the backend.

The library goes through the following steps to push entities modified in local storage to the backend -

  • Reads from the "pending writes queue" to determine what entities have been changed locally. The "pending writes queue" maintains a reference for each entity in local storage that has been modified by the app. For an entity that gets modified multiple times in local storage, the queue only references the last modification on the entity.

  • Creates a REST API request for each pending change in the queue. The type of request depends on the type of modification that was performed locally on the entity.

    • If an entity is newly created, the library builds a POST request.

    • If an entity is modified, the library builds a PUT request.

    • If an entity is deleted, the library builds a DELETE request.

  • Makes the REST API requests against the backend concurrently. Requests are batched to avoid hitting platform limits on the number of open network requests.

    • For each successful request, the corresponding reference in the queue is removed.

    • For each failed request, the corresponding reference remains persisted in the queue. The library adds information in the push/sync response to indicate that a failure occurred. Push failures are discussed below.

  • Returns a response to the application indicating the count of entities that were successfully synced, and a list of errors for entities that failed to sync.

string collectionName = "books";
DataStore<Book> dataStore = DataStore<Book>.Collection(collectionName,
                                                       DataStoreType.SYNC);

// In this example, we push all the data in the collection
// from the backend to the Sync Store
try
{
  DataStoreResponse pushResponse = await dataStore.PushAsync();
}
catch (KinveyException ke)
{
  // handle exception
}
Handling Failures

The push response contains information about the entities that failed to push to the backend. For each failed entity, the corresponding reference in the pending writes queue is retained. This is to prevent any data loss during the push operation. The developer can decide how to handle failures. Here are some options -

  • Retry pushing your changes at a later time. You can simply call push again on the datastore to attempt again.
  • Ignore the failed changes. You can call purge on the datastore, which will remove all pending writes from the queue. The failed entity remains in your local cache, but the library will not attempt to push it again to the backend.
  • Destroy the local cache. You call clear on the datastore, which destroys the local cache for the store. You will need to pull fresh data from the backend to start using the cache again.

Each of the APIs mentioned in this section are described in more detail in the datastore API reference.

Sync

Calling sync() on a Sync Store kicks off a bi-directional synchronization of data between the library and the backend. First, the library calls push to send local changes to the backend. Subsequently, the library calls pull to fetch data in the collection from the backend and stores it on the device.

You can provide a query as a parameter to the sync API, to restrict the data that is pulled from the backend. The query does not affect what data gets pushed to the backend. If you prefer to only retrieve the changes since your last sync operation, you should enable delta set caching.

string collectionName = "books";
DataStore<Book> dataStore = DataStore<Book>.Collection(collectionName,
                                                       DataStoreType.SYNC);

// In this example, we sync all the data in the collection from the
// backend to the Sync Store, which does a push followed by a pull
try
{
  DataStoreResponse syncResponse = await dataStore.SyncAsync();
}
catch (KinveyException ke)
{
  // handle exception
}


// In this example, we restrict the data that is pulled from the backend
// to local storage by specifying a query in the sync API

var query = dataStore.Where(x => x.Title.StartsWith("The"));

try
{
  DataStoreResponse syncResponse = await dataStore.SyncAsync(query);
}
catch (KinveyException ke)
{
  // handle exception
}

Sync Count

You can retrieve a count of entities modified locally and pending a push to the backend.

// Number of entities modified offline
int syncStoreCount = dataStore.GetSyncCount();

Cache

Configuring your datastore as a Cache datastore allows you to use the performance optimizations provided by the library. In addition, the cache allows you to work with data when the device goes offline.

This type of datastore is ideal for apps that are generally used with an active network, but may experience short periods of network loss.

Here is how you should use a Cache store -

// Get an instance
string collectionName = "books";
DataStore<Book> dataStore = DataStore<Book>.Collection(collectionName,
                                                       DataStoreType.CACHE);

List<Book> books = new List<Book>();

// The KinveyDelegate onSuccess() method will be called with cache results.
// The method will then return with the network results.
KinveyDelegate<List<Book>> cacheDelegate = new KinveyDelegate<List<Book>>()
{
  onSuccess = (List<Book> results) => books.AddRange(results),
  onError = (Exception e) => Console.WriteLine(e.Message)
};

List<Book> networkResults = await dataStore.FindAsync(cacheDelegate);
books.AddRange(networkResults);

// Save an entity. The entity will be saved to the device and your backend. 
// If you do not have a network connection, the entity will be stored in local storage, 
// to get pushed to the backend when network becomes available.
Book book = new Book("My First Book");
await dataStore.SaveAsync(book);

The intermediate results returning cache data is only valid for use with a data store of type DataStoreType.CACHE. Intermediate results are not applicable for the DataStoreType.SYNC and DataStoreType.NETWORK data stores.

The Cache Store executes all CRUD requests against local storage as well as the backend. Any data retrieved from backend is stored in the cache. This allows the app to work offline by fetching data that has been cached from past usage.

The Cache Store also stores pending write operations when the app is offline. However, the developer is required to push these pending operations to the backend when the network resumes. The push API should be used to accomplish this.

Network

Configuring your datastore as Network turns off all caching in the library. All requests to fetch and save data are sent to the backend.

We don’t recommend this type of datastore for apps in production, since the app will not work without network connectivity. However, it may be useful in a development scenario to validate backend data without a device cache.

Here is how you should use a Network store -

// Get an instance
string collectionName = "books";
DataStore<Book> dataStore = DataStore<Book>.Collection(collectionName,
                                                       DataStoreType.NETWORK);

// Retrieve data directly from your backend
List<Book> books = new List<Book>();

books = await dataStore.FindAsync();

// Save an entity directly to your backend
Book book = new Book("My First Book");
await dataStore.SaveAsync(book);
Got a question?