2 Replies Latest reply on Oct 15, 2011 11:00 AM by Production Monkey

    Using value() with large arrays

    Listat_jhg

      Hi there!

      I am working with arrays from a txt file. They are really large. I use value to change them into real arrays in lingo.

      Someone can tell me if there is a limit in the length of the string to value because value returns a strange array.

       

      - It works well with 85284 string's characters.

      - It doesn't work with 355391 string's characters.

       

      Thanks.

        • 1. Re: Using value() with large arrays
          Adam-C Level 2

          Value is intended to return the numeric value of a statement contained in a string. This can be useful if one needs to create a calculation dynamically within code. I'm not sure that it's the best way of reading an array from a text file though - I would think that you'd be better-off using the FileIO Xtra to open the file and then parse it into an array character-by-character.

           

          You may find that the "do" command will be more suitable to your current method of working though - it attempts to execute the expression contained in a string. So something like

           

          set myArrayString = "[1, 2, 3, 4]"

          set myArray = void

          do "myArray =" && myArrayString

           

          is a better way of converting strings into arrays (note that "myArray" is initialised prior to the "do" command so that Director's compiler recognises the creation of the variable and doesn't throw a script error when you try to close the script-edit window).

           

          You could also arrange your text-based data in XML format and then convert it into Director data using the xmlparser Xtra (the xmlparser takes a wee bit of experimenting to get the hang of, but is pretty handy once you master it).

           

          Another solution would be to use the BudyFile Xtra (from the same people who do the BuddyApi Xtra - google it to find their site). This has functions that allow you to read and write files containing Director arrays comprised of virtually any Director datatype, with optional encryption if desired.

           

          Hope that helps.

          • 2. Re: Using value() with large arrays
            Production Monkey Level 3

            Value works well for converting strings into lists, however, as you discovered it fails if the string is too large or too complicated.

             

            In the old days I would have recommended the Vlist xtra. You could use it to read and write lists and there was no charge if that was the only functionality you needed. It seems to be discontinued although you can still download it.

             

            Depending on how you interact with the data in the list, using a SQLite database may be a better option. It is blazingly fast if you design your database well. Free xtra here: http://staff.dasdeck.de/valentin/xtras/sqlite_xtra/

             

            If you just want to stick with your list and don't mind long startup times, then the FileIO xtra will work as Adam suggested. I wrote a couple functions to do this. One for loading a text file into a list and one to save the list back out to a text file. On my machine it took 50 seconds to load a 400000 word file and 4 seconds to save it.

             

            To try the code out do the following in the message window:

             

            h = ["One","two","three","four"]

            saveStringList(h, _movie.path & "mydata.txt")

            h = []

            loadStringList(h, _movie.path & "mydata.txt")

            put h

             

             

            Notes: 

             

            The load function expects a space delimited file with the first word indicating how many words are in the file. It is probably best to use the saveStringList() function first to create a file that can be read in.

             

            These functions are design to work with a linear list of strings. Nothing more.

             

            -- Movie script list functions

             

            -- Read SPACE delimited strings into the list DataList

            -- * First string MUST be the string/word count.

            on loadStringList DataList, FilePath

              FileIn = new(xtra "fileIO")

              FileIn.openFile(FilePath, 1)  -- Open the file with read-only access

             

              if FileIn.status() then

                FileIn.closeFile()

                put "Error reading data in handler loadStringList. "

                exit

              end if

             

                WordCnt = Value(FileIn.readWord())

                repeat with cnt = 1 to WordCnt

                  DataList.add(FileIn.readWord())

                end repeat

             

              FileIn.closeFile()

            end loadStringList

             

             

            -- Saves a list of Strings to a file delimited by spaces

            on saveStringList DataList, FilePath

              FileOut = new(xtra "fileIO")

              FileOut.createFile(FilePath)

              FileOut.openFile(FilePath, 2)  -- Open the file with write-only access

             

              if FileOut.status() then

                FileOut.closeFile()

                put " Error writing file in handler saveStringList ."

                exit

              end if

             

              WordCount = DataList.count

              FileOut.writeString(String(WordCount) & Space)

             

              repeat with Val in DataList

                FileOut.writeString(Val & Space)

              end repeat

             

              FileOut.closeFile()

            end saveStringList

            -----------------------------------------