1 Reply Latest reply on Jun 4, 2013 6:05 PM by Lampei

    Local data storage with XML vs SQLLite

    JamieStonehill

      Short version:

       

      I have a mobile app that gathers and stores a large amount of data. The current XML solution (all in one file) is having performance issues as the volume of data increases beyond a certain point.

       

      Can I be confident that using SQLLite and a local database will be a better option and improve performance?

       

      Long version:

       

      I have built a mobile app that is used by my client to collect data.

       

      When we started the project the amount of data was relatively small. Given that, and given the past experiences of my team, we chose to use an XML file to store the collected data. When the app starts it loads the whole xml file into a set of data objects in memory for use by the app. The data is not loaded again for as long as the app stays running. We then run a routine whenever the app wants to save to the HD that converts all of the data back into XML.

       

      The data is split into 'projects' and each project naturally has its own xml node. To improve performance when saving, each instance of the Project class stores its loaded node string. Then, when the save routine is called, if the classes data has not been changed it simply returns the original string instead of going through the whole process of re-compiling it.

       

      Of course this does not change the fact that whenever a save is performed, the whole xml structure must be saved back to the file on the HD.

       

      The app has now been used in anger for quite some time and we are starting to get reports of performance issues. The main culprit is during loading which only happens once but can apparently now take an exeptionally long time and causes problems during startup to the point where users think it has crashed.

       

      I am now tasked with trying to improve the data storage for the app. My original reaction was to try breaking down the xml into multiple files which can then we loaded and saved when needed instead of handling everything all at once. I am worried though about the implications of the app trying to handle in excess of 2000 xml files in some cases. It may not be an issue at all but it just stands out to me as something to be concerned about.

       

      So the best other option that I am aware of is to use SQLLite to save all the data into a local database. I have very little personal experience with sql and databases though so while a lot of the documentation and blog posts I see seem to suggest that this is the way to go for mobile apps with a large amount of data, I cannot be quite certain. The big issue is that I cannot afford to sell the idea of such a radical change (and more importantly the time it would take to implement it) to my client without being quite sure that this will be a valid solution to the problem.

       

      So my question to you, Flex community, is would you continue to use XML or is SQLLite the better way to go for large amounts of data? Do you think that I would see an improvement by using SQLLite? Additionally, do you have any tips or experience with a similar situation that might be helpful to me?

       

      Thank you

       

      Jamie

        • 1. Re: Local data storage with XML vs SQLLite
          Lampei Level 1

          I've been very happy using the internal SQLite database with Flex.  You should definitely get much better performance from it.  Just being able to load your data asynchronously if nothing else you should get a great benefit.  The sheer amount of data in your XML file that is being passed around is probably quite a hit (especially on mobile).  Breaking up the XML shouldn't be too crazy to do (as you're already doing it somewhat, I'd imagine, when you're accessing the information from within your app).  Give switching to SQLite a try, but I'd say that's a great place to start.