Alex Rivera | Logout

How to improve large data import performance with core data in ios

Asked 2012-01-10T14:30:46.917
11

I'm importing ~18,000 records of various entities that are related to each other from a web service on first launch of an iOS app. The records have their own unique identifiers from the database on the web. I have read and re-read apple's documentation, cimgf's blog entries, Marcus Zarra's books on core data, and watched over and over again the iDeveloper TV's series on core data. I can't figure out a way to import Entity A without keeping Entity B, C, and D in memory or performing expensive fetches and maintain relationships between the entities. If I were not using core data, I'd be able to just use the unique identifiers that are already established and the import would be much faster.

If you have any suggestions, I'm all ears.

We've implemented this strategy in Android and the import takes ~2.5 minutes compared to ~6 minutes on iOS where the hardware on the different devices are comparable. Shortening the import time is crucial to our users so I have not been able to compromise on this issue. Thanks in advance for your help.

EDIT:

Here's how I'm currently doing it - I create an NSOperation which creates it's own context. I import Entities B, C, & D first and keep an array per entity type. I then import Entity A and use predicates to filter the arrays for Entities B, C, & D in order to relate Entity A to the appropriate Entities in B, C, & D. I'm batch saving the context at optimized intervals that are different depending on which entity type I'm currently importing.

I'm not just importing to one table, I'm importing many tables that are related to each other. So If I import Entity B, I have to either keep Entity B in memory or fetch it when I need it to relate Entity B to Entity A. Make sense?

ios sqlite core-data
Edit
Report

2 Answers

10

I will raise the same issue every Core Data question gets, and then move on: Make sure that you really want to use Core Data for this and not just sqlite directly. Core Data is for persisting object graphs, not a general database. If you've already implemented a database for Android, then you may want to use the same schema and design on iOS.

OK, got that out of the way. Let's assume CD is really the best tool for the job here (or you can't change it at this point). My first thought here is to cheat. Granted, that is often one of my early thoughts....

See how fast you can insert these objects with no relationships. If that's fast enough, then here's how you cheat: don't store actual relationships at first. Store a string list of identifiers that describe the relationship. Then, once everything's loaded and the user can start working, over time convert the string relationships into real relationships in the background. Whenever you fetch a record, you need to check if it has a cheater-property still set and if so, you'll need to fetch its relationships by hand (and then clear the cheater-property).

This doesn't make the full import faster, but it gives the illusion that it's faster, and that's the goal in iOS 90% of the time. You might even have to prevent certain operations (like delete) until you've finished gluing everything, but that's probably still better than blocking the user entirely.

answered 2012-01-10T15:07:18.033
0

I don't about the process you are following, but have you considered using blocks and GCD (Grand Central Dispath)? I guess it was in a presentation from an employees of LinkedIn about their iPhone app in iTunesU. If I remember correctly, they are also using GCD.

"...GCD API, which supports the asynchronous execution of operations at the Unix level of the system. You can use this API to manage interactions with file descriptors, Mach ports, signals, or timers..."

https://developer.apple.com/library/ios/#documentation/Performance/Reference/GCD_libdispatch_Ref/Reference/reference.html#//apple_ref/doc/uid/TP40008079

answered 2012-01-10T14:54:14.447

Your Answer