I would do an internet search for programs that convert videos to different formats. (Or maybe someone can jump in here and give an example or two). I used to have such a program, but it was a long time ago and came with some other device that I bought, and I no longer remember the name.
Once it is converted to a format that is compatible with Lightroom, you simply import it, the same as you import any other video or still photo. I prefer a single catalog for many reasons, but I believe that your concerns about catalog size are not real concerns, that the size of your catalog and image/video collection is not a problem for Lightroom.
You might consider using the Video Asset Management or Any File plugins to import the existing video files into your LR catalog. This would let you manage them with LR while you incrementally convert them to a more modern, common format.
In general, LR's support for video is pretty weak, especially for video metadata and for older formats. If you care about preserving your metadata for the long-term, you might consider those plugins.
What format and software for this task do you sugest?
I'm far from an expert on video formats. However, I use Handbrake (free) and Xilisoft Video Converter (paid) for the occasional conversions I need to do.