You have to work with layers. Put the text on the topmost layer, the problem is interacting of you text with transparency. That is why you have to have all text above objects with transparency.
Objects with transparency above text will text rasterize or at least convert into outlines. Both look bad in a PDF.
I do have the text on a layer above the transparency tho. Thats what is confusing me
Does it look bad in the PDF on screen only or also when printed on a high resolution device?
If the text is above all transparent objects than it is not a transparency flattening problem, although you did not tell us anything about your PDF export settings.
If it looks good on high resolution print but bad on screen only it might be an issue of the used font only. To be honest, on your screen shot I don't see the problem here what you mentioned. Could you link your problematic INDD and PDF here. Not the original file, it understnd that it is not possible for you, but a small file where you reconstruct this problem with a placeholder image and blind text, one page only to keep things small.