After working on the photos and rejecting some out of different sub-folders I purged the database of these rejected photos and renamed the whole folder to get the numbering straight again.
How did you do this renaming? Did you use your operating system or Lightroom?
I used Lightroom
Well, I honestly don't have an explanation or a solution for you.
It seems to me that if you do the deleting and renumbering before you do the editing and adding of keywords, you can avoid this problem. Or, better yet, don't rename and then the problem goes away entirely, you can use keywords instead of adding text to the file name, and you can always use Lightroom to search by capture time instead of putting the capture time into the file name.
The mess I achieved with this is that keywords and improvements assigned to a specific image before renumbering are not reassigned correctly to the newly renamed image (eg. after rejecting and purging 20 photos details to image -1020 are with image -1013 after renaming instead of -1000). To make the mess complete this has no logic, so I cannot find a logical sequence, the misassignments are made randomly.
Any idea what can be the problem?
Although you mention that all file and folder "renaming" was done inside LR it sounds like you "renumbered" the image files from outside LR. That's the only way that edits and keywords could become disassociated from the original files.
For example you have a folder with 100 pictures and deleted 50 rejects in some "random" order. You then rename the remaining 50 files outside LR in a sequence of 1-50. So let's say image file #50 is now renumbered #29. When you rename files or folders from outside LR they normally become "lost" inside LR and will appear with a ! or ? symbol. In your case they remain "connected" in the LR catalog because a file with that name is in the catalog. Unfortunately the keywords and edits in the catalog associated with "original" file #29 are now (incorrectly) applied to the #50 original file.
Editing and "culling rejects" is an ongoing process and you'll likely need to "rename" files multiple times if your objective is to maintain consecutive file numbers (1, 2, 3). Don't do that! Your file Import file naming structure yyyymmdd-CAL-####.nef (#### being the numbering of the photos) is fine. What I would suggest is that you simply don't change or modify the file name after they are imported. The files will remain in the same 'file name' sequential sort order without renaming.
Here's a good book on the subject that may be helpful: