Upgraded FM9 to FM12; now all Symbol font characters are appear as "?"

[jeff_gansberg]

I'm using Frame 12.0.2.389.  I'm using the same font resources that worked for Frame 9, where I had no problems.

For example, I use Ohm for electrical resistance. Normally, I type a W, select it, and assign the Symbol font.  Now, it appears as "?".  All characters in the symbol font show the same result.

There are no missing font alerts when I open files.  The Font pod shows the Symbol font is installed/available.

What am I missing?

Thanks.

[Arnis_Gubins]

Which specific version of the Symbol font are you using? I suspect that you're using the unicode encoded Symbol Std OTF version instead of the older ASCII  encoded Symbol TTF version. In the unicode version the characers are encoded with the unicode values in a private area and W in the standard ASCII range will generate the unknown "?" character.

[Bob_Niland]

> Normally, I type a W, select it, and assign the Symbol font.

We are in the Unicode age. I'd suggest not doing that anymore, or if you must, do it as a Variable, because applying an override like that can have various unhappy side effects (like, put cursor right after that Ω, and anything else you type will also be from the Symbol codepage glyph set).

> Now, it appears as "?".

Reportedly, FM8 and higher, when it opens a document with legacy codepage  text, may attempt to convert instances to Unicode.

I've never seen any technical discussion of this but what appears to have happened in your case is consistent with this scenario.

Your document had a "W" (\x57 with an override to the legacy Symbol font, whether Type1 or TrueType probably doesn't matter).

FM12 converted that \x57 to a Unicode

\u03A9 GREEK CAPITAL LETTER OMEGA

(UTF-8 for that would be CE A9, and you'd see something like "<String `Ω'>" for that in the MIF for the text at that point)

You are seeing a "?" because the font in effect for the character is not populated at code point U+03A9. This could still be your legacy Symbol font, which is definitely not populated above U+00FF, or if FM has deleted the Symbol override, then your body font lacks that code point.

You're probably going to have to Search by Character Format, or by Text & Character Formats on Clipboard to find and correct these instances.

Our Fm7.x shop does all non-roman characters as Variables both to avoid override/Character_Format issues, and to ease our eventual Unicode transition. Once migrated, it might still be useful to do all non-keycap characters as Vars as well.

Pre FM 8 instance:

Var Name: glyph.U+03A9.GREEK CAPITAL LETTER OMEGA

Var def.: <Symbol>W

After upgrade:

Same Name:

Var def.: \u03A9

or possibly:

Var def.: <Arial Unicode MS>\u03A9

or some other Unicode font Unicode known to have code point U+03A9 populated.

I might add that to be pedantic, Unicode has a specific "Ohm" symbol, which is a different code point than the Greek cap Omega, although usually the same glyph strokes:
U+2126
And I would use that for ohm if possible, as it makes the document semantically correct for future AI engines that interpret it.

[Bob_Niland]

> Reportedly, FM8 and higher, when it opens a document with legacy

> codepage text, may attempt to convert instances to Unicode.

I ran a test on that:

1. Create document in FM7.1/Unix, with omega as:
   a. override using Type1 Symbol font
   b. Character Format
   c. Variable
2. Open document in FM9/Win7.
   Exactly none of the instances got converted to Unicode or went "?"
   They were all still <String `W'>.
3. Saved out the MIF9, and verified that no Unicode conversion happened.

The Win system does have the MonoType TT "Symbol Regular" installed. Whether this outcome might be different if the invoked font is not installed is not known to me. A test for another time.

This suggests that the "?" scenario proposed by Arnis is more likely to be the explanation.

[Arnis_Gubins]

FM doesn't do any automatic conversions of characters to unicode, as it doesn't know the codepage/encoding of the characters being used with any particular font. It just looks up the specified character value in the font. The Symbol Std OTF font has all of the glyphs encoded using unicode values, while the Symbol TTF font has those in the 1-255 range. The Symbol Std OTF 1-255 range doesn't contain characters, so FM displays the "?" symbol to let you know that there is some content there.

[jeff_gansberg]

The font version; hm...the file date-stamp is definitely Post Cold War, but I don't know how to check for a formal version number.  In any case, I have resolved my issue.

I found that none of my colleagues were having this problem, so I re-installed the Symbol font.  Wah lah.

A heart-felt thank you to Arnis, Error7103, and all the experts--your contributions to this forum are greatly appreciated.

Best regards,

Jeff

[Bob_Niland]

> FM doesn't do any automatic conversions of characters

> to unicode, as it doesn't know the codepage/encoding of

> the characters being used with any particular font.

I understand the challenge (esp. when the invoked legacy codepage font is not present on the new system), which why is why I've been curious about what, if anything is going on when you drag a document across the FM8 border.

Here's a typical quote of the sort that has me wondering (from the MIF2GO site😞

"... When FrameMaker converts a pre-8.0 file, it converts the content to Unicode in UTF-8 encoding. However, the index markers are not converted ..."

What exactly is being converted? Any low-order (\x20 to \xFE) characters in a pre-8 FM document, if they have a representation in Frame Roman, are shown to you as the glyph, even if hand-typed as \x##, including when viewing MIF. Only [some] of the non-printing hex codes get preserved as \x code strings.

My guess is that a handful of the Frame Roman "standard" characters that aren't in Unicode Basic Latin (ASCII) and Latin 1 Supplement do get converted.

Testing that just now, the natively superscripted trademark sign [™], which is \xaa in Frame Roman is not in Unicode U+0000 to U+00FF space, and does get converted to Unicode (U+2122 I presume). If that's it, that's not a whole lot of conversion going on. FM9 does convert FM7 instances to Unicode whether hand-typed or defined as a Variable. FM9 also converts it if typed into a new FM9 document.

I can imagine other scenarios where FM8+ checks the system localization, and does further conversion, or detects some legacy multi-byte Asian encodings, and re-does those (but that strikes me as something an app would ask permission for, and not just do).