Why? Well, we're not Adobe software engineers, we're just other users, and we don't really know. Probably there's a very good reason. In fact, I'm sure there is a very good reason. (By the way Photoshop and InDesign are not databases, that may have something to do with it).
It is a software engineering design decision.
Ages ago that decision was probably made for ease of coding. Now the decision is one they are stuck with. To change it now would probably require some changes in the architecture that knowledgeable folks think is more costly than beneficial.
This is all just speculation based on years of software development experience.
Thanks, MJ. Your answer makes more sense to me than the database rote. I've used databases before (iPhoto, Filemaker and Excel come to mind) and none of them misbehave like Lightroom. I'm not even sure that a Lightroom Catalog qualifies as a "database." In the macOS Photo app, there is an actual database, and you never edit original images. But in Lightroom edits are written to a raw image's XML sidecar file, or somehow embedded in JPGs and newer raw files such as Adobe's own DNG format.
Lr most certainly does use a database (SQLite). It has an .lrcat extension and .lrdata (Lr previews) also uses a compact database.
As to your comments re edits written to XMP sidecar or DNG. This is actually a user decision/action. By default Lr stores edits and all other metadata in the lrcat database, and does so irrespective of whether you save the data to file or not.
I stand corrected.
I guess it is a design decission.
There is no technical reason that you can't have open more than one database at the same time.
1 ) So either they started with lightroom having exactly one database/catalog at first and found it a good idea to have the possibility to have more than one catalog later. Then I guess it was easier to start lightroom with the information of the database/catalog it should open.
2 ) Or it is just because of ressources
But to have the real answer, you have to ask the developpers.
There are many reasons and reasonings for the single user database.
Some of the smartest "Developers" at Adobe made the decision including Thomas Knoll and Mark Hamburg ... Google them if you don't recognise their names.
But I'm sure time will surpass the reasons for a single user database as more and more connectivity is available and we push more and more to have access to everything, everywhere and all of the time.... we might just use the big database in the cloud!!
BTW Photos only allows one database open at a time as well.
That was not the point.
I was not criticising the developers decision. I just wanted to show two possible reasons for the fact that you have to restart lightroom if you switch from on catalog to another.
It has nothing to do with single user database. I can open as many single user databases simultanously (even SQLite databases like it is used as catalog file), as I want. Single user databases just don't handle concurrent multiuser access.