7 Replies Latest reply on Apr 29, 2015 9:26 PM by Faisal Bawa

    CF9 and SOLR indexing

    DanaPayne Level 1

      We are using CF9 64-bit and setting up a SOLR collection for an HR application. The database contains several million records and includes resumes that we want to do full text searches on.

       

      We started out by using cfindex to create the index but it would bomb out after just a few thousand records with an error about "warming threads" (I don't have the exact error handy but can get it later) and the indexing would have to be manually restarted. This wasn't a good solution for a multi-million record operation..

       

      Next, we created a custom Data Import Handler (DIH) outside of CF using the instructions in the SOLR wiki. This index worked great and was very fast. However, the ColdFusion tags (cfsearch, etc.) would not work with this index. We even made sure to duplicate the required nodes (<custom1> <custom2>, etc.) that the cfindex tag would have created. Still cannot search that index.

       

      We'd really rather not reinvent the wheel and have to write custom search code. Obviously, we like using CF and it would be great if we can use the built-in indexing and searching capability.

       

      Any ideas on how we can either 1) make the <cfindex> work without stopping OR 2) go ahead and use the custom DIH and be able to make the <cfsearch> work properly?

       

      Dana

        • 1. Re: CF9 and SOLR indexing
          Adam Cameron. Level 5

          Couldn't you just use the full-text search capabilities of whatever DB it is you're running?  Do you even need to use CF & Solr for this?

           

          --

          Adam

          • 2. Re: CF9 and SOLR indexing
            DanaPayne Level 1

            Thanks, Adam.

             

            Just using SQL is an interesting suggestion but not really a good solution in our case. We'd lose all the built-in advantages of a search engine like SOLR, for example faceting, exposing "near duplicates", etc. Our users are already used to doing "concept searching" but the search engine we were using can't be upgraded to work on our new platform, so we have to replace it. We thought SOLR was just the ticket.

             

            Maybe SOLR integration into CF9 is just too new and still has too many bugs to work out? Are there any insiders at Adobe monitoring this forum who might know?

             

            I'm really hoping we can find a way to make SOLR work for us so we don't have to roll our own concept search from scratch.

             

            Dana

            • 3. Re: CF9 and SOLR indexing
              PolicyDesigner

              I only have just over 500 records that I am trying to index, which they do consist of some large documents, and I try to loop through using the cfindex and I also get this error:

               

              Error_opening_new_searcher_exceeded_limit_of_maxWarmingSearchers4_try_again_later

               

              I found that if I put this in my loop

               

              <cfscript>

                  thread = CreateObject("java", "java.lang.Thread");

                  thread.sleep(1000);

              </cfscript>

               

              then I no longer have the error, but it does take a long long time to index.  I also would like a better solution.

               

              The coldfusion debugger shows that it is erroring out on the custom4 field.  I don't know if the custom fields are struggling more than the main body field.  Anyway, I am continuing to research my options.

              • 4. Re: CF9 and SOLR indexing
                desalvatore.com

                I am having a very similar problem... Did you ever find a resolution for this, other than adding the sleep time?

                • 5. Re: CF9 and SOLR indexing
                  PolicyDesigner Level 1

                  I never did find a solution, I am very surprised at this.  I am just waiting around for an answer.

                  • 6. Re: CF9 and SOLR indexing
                    CyberBlog

                    hmm...thats a shame. I too am having the same problem. Every time I find someone with a similar problem, the solution is never there. Indexing seems to run fast some days and slow the other days. The documentation for SOLR isnt the best either. It seems as though SOLR is nice for smaller searches, but once you start getting into larger amounts of documents it seems to fail miserably. 

                     

                    I'm currently indexing 4,000 documents and its been running for 40 minutes.

                     

                    Also, how come they released a built-in version of SOLR into coldfusion without any crawlers?

                    • 7. Re: CF9 and SOLR indexing
                      Faisal Bawa

                      I am also having the same issue