I've tried using the merge option to delete the empty line, but it has never worked properly.
That's odd; it's pretty much exactly what you are asking after. Can you tell us anything outside of "it has never worked properly"?
Is there anything non-standard about your merge? (I.e. unusual text encoding, images, master pages, n-up merges, et cetera.) Can you take screenshots of your merge dialogs? Or give us a step-by-step rundown of what you are doing? Are you previewing before merge?
I am an infrequent but long-time user of Data Merge, and I've always been able to use the "Remove blank lines for empty fields" checkbox without issue.
If you have a space between the fields, ID doesn't see it as empty, it sees it as containing a space character. I have a directory I do with similar problems, and I have some GREP queries I use to clean it up:
Find ^\s+\r and replace with nothing. This removes those white spaces between null fields and removes the line.
Find \r,\s$ and replace with nothing should remove a "blank" line in the form <lastname>, <firstname>
Looking at these together, I'd probably rewrite the second one as ^,\s\r as I think the first version may change the paragraph style of the line above the blank to the style assigned to teh blank line if it's different.
If you have a space between the fields, ID doesn't see it as empty, it sees it as containing a space character.
Never mind my questions, Peter has the correct answer.