2019 bug? This ECMAScript XML snippet works in 2018 but not 2019?

[TᴀW]

Would anybody know why the following snippet throws an error when the ESTK target is 2019, but works fine when the ESTK target is 2018 (or ESTK itself)?

Is ASCI 22 in fact not kosher for an XML text node?

a = "apples";
b = "pears";
c = String.fromCharCode(22); // ASCII 22 is the character InDesign uses as a table placeholder in a textframe.
x = <food>
<fruit>{a}</fruit>
<fruit>{b}</fruit>
<fruit>{c}</fruit>
</food>;
alert(x);

When targetting 2019 (even though the 2019 DOM is of course not touched by this script), ESTK throws an "XML error in line 4", and highlights line 7.

But it's fine if the target is 2018.

Am I missing something obvious?

Thanks,

Ariel

[Jongware]

Is your code to create XML on the fly inside InDesign? How does that table (without any data inside!) then appear? Perhaps it's a bug that you could actually specify it in earlier versions and that has been fixed ...

[TᴀW]

It's the other way round. This is part of a script that creates an XML object from the contents of text frames.

Until now it worked fine. But in 2019 it started throwing an error in case the text frame contained a table.

So, for example:

t = myTextFrame.contents;

x = <myxml><item1>{t}</item1></myxml>

If myTextFrame contains a table, t will contain ascii 22, and the script will crash on line 2. In 2019. But not in earlier versions.

[TᴀW]

To put it in a nutshell, the following 1-line script throws an error if the target in ESTK is InDesign 2019, but not if it's ESTK, or InDesign 2018.

x = <myxml>{String.fromCharCode(22)}</myxml>

To me it seems very strange the the Javascript interpreter should be affected by the target-app dropdown in ESTK, especially considering that this 1-liner doesn't actually touch the target application in any way.

[Trevor:]

Interesting one

var x, charCode;
for (charCode = 0; charCode < 10000; charCode++) {
    try {
        x = <myxml>{String.fromCharCode(charCode)}</myxml>
    } catch (e) {
        $.writeln(charCode + '\t' + String.fromCharCode(charCode) + '\t' + e);
    }
}

On Illustrator 2019 no problems.

On InDesign 2019

✏

0

1   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

2   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

3   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

4   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

5   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

6   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

7   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

8   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

11   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

12   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

14   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

15   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

16   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

17   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

18   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

19   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

20   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

21   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

22   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

23   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

24   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

25   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

26   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

27   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

28   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

29   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

30   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

31   SyntaxError: XML error in line 1
- Not well-formed (invalid token)

Solution / Workaround ?

xmlRegex = /[\x01\x02\x03\x04\x05\x06\x07\x08\x0b\x0c\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f]/g;
myXMLSafeContents = myContents.replace(xmlRegex, function(r){return '#!@' + r.charCodeAt() + '#!@@';});

[Laubender]

Hi Trevor,

very strange, indeed.

If the target is the ESTK: No error message.

With InDesign CC 2019: All the errors that you are listing.

No error messages with e.g. InDesign CC 2018.1, PhotoShop CC 2019.

Regards,
Uwe

[TᴀW]

So which app is bugging out? It's got to be ID2019, no? Or are we saying that's the only one that's got it right?

[TᴀW]

Hi Trevor,

Thanks. Interesting that you were even able to wrap this in try-catch. Were you running the script in InDesign? When I tried to run it in ESTK with ID2019 as target, even try-catch didn't help. (Actually, maybe I didn't check my debug options, but I think I did tell it not to break.)

[Laubender]

TᴀW  wrote

Hi Trevor,

Thanks. Interesting that you were even able to wrap this in try-catch. Were you running the script in InDesign? When I tried to run it in ESTK with ID2019 as target, even try-catch didn't help. (Actually, maybe I didn't check my debug options, but I think I did tell it not to break.)

Hi Ariel,

I did my tests with the ESTK CS6 3.8.0.12 and also with ESTK CC 4.0.0.1 with target InDesign CC 2019 14.0.1.209 on Windows 10 Pro. Trevor's code with try-catch worked like it should. Had exactly the same results as Trevor is showing in reply 4. Maybe it will turn out differently, try-catch not working, on OS X ?

TᴀW  wrote

So which app is bugging out? It's got to be ID2019, no? Or are we saying that's the only one that's got it right?

Hard to tell. To my tests only InDesign CC 2019 is showing the new behavior.

In the meanwhile I also tested InCopy CC 2019 and Bridge CC 2019 as target. No issue even with InCopy.

Regards,
Uwe

[TᴀW]

Checked again, you're right indeed. Try-catch does work. I must have had the "do not break on guarded exception" unchecked.

So, this is mysterious. Martinho's answer below is a decent workaround, but still.

Maybe worth posting this on the beta testing as a bug, and see what the Adobe engineers have to say...

[Trevor:]

Hi Uwe

Very interesting about InCopy which my understanding was that it's basically InDesign core with a different set of plugins and UI.

Hi Ariel

Martinho's method I think is the way to go, XML literals E4X had not been supported in mainstream JS environments for years and I can't see it holding out for long in AdobeLand. Using methods like appendChild is going to make porting to the new environments a lot more simple as those methods are much more similar to what is going to be available.

You can see what the engineers come up with but it just a matter of time before it breaks again forever even if they do fix it.

[TᴀW]

You're probably right. OTOH. frameworks like React are bringing a very similar syntax back in again...

[Trevor:]

Yes, that's a very good point about react.

[Marc Autret]

Hi all,

TaW made a crucial point:

(…)

To me it seems very strange the the Javascript interpreter should be affected by the target-app dropdown in ESTK, especially considering that this 1-liner doesn't actually touch the target application in any way.

Serious question indeed. Provided that the target app—i.e. its particular DOM—is not involved in the reported error, why should ID CC 2019 behave differently, versus ID CC 2018 or even PS CC 2019?

This enigma led me to investigate and here is what I found:

1. The XML parser which ExtendScript relies on is (very likely) the Expat library designed by James Clark:

libexpat/expat at master · libexpat/libexpat · GitHub

What allows me to make this assumption is that typical XML tokens and error strings appear in ExtendScript's core library (ScCore.dll) and most of them are clearly connected to Expat parameters.

2. Expat is “an XML 1.0 parser”, as stated in its documentation and confirmed in its source code. For example, the header asciitab.h has the folowing lines

which indicates that almost all control codes from 0x00 to 0x1F are considered NONXML—and then should be reported as invalid by the tokenizer.

As already observed by my colleagues, the only control codes allowed in XML 1.0 are U+0009, U+000A, U+000D. Hence, the code 0x16 (decimal 22) which TaW has inserted in its XML stream should have failed from the very beginning, in any target app.

3. However, Expat has, or had, various bugs that made it more or less permissive regarding the XML spec. Many updates and fixes are reported in the changelog from release 1.95.0 (Sept. 2000) to 2.2.6 (Aug. 2018.) I do not understand those technical details, but sentences like “Fixed UTF-8 decoding bug that caused legal UTF-8 to be rejected”, “Check that a UTF-16 encoding in an XML declaration has the right endianness”, or “Mass-fix compilation for XML_UNICODE_WCHAR_T”, etc. sound very much in connection with our topic.

Also, depending on how Expat is configured when compiling the lib, I suppose that the client can get the tool in various flavors. Cf. the section “Configuring Expat Using the Pre-Processor” in the documentation.

4. Now, let's go back to the original question: how is it possible that ID CC 2019 differs from ID CC 2018 regarding XML parsing at the ExtendScript level? My guess is that they may not use the same Expat version (in ScCore.dll.) The following screenshots strongly support that view:

As you can see, InDesign 13.x embeds ExtendScript with Expat 1.95.7 while InDesign 14.x uses Expat 2.2.5.

This answers at least a part of the question: we know that ExtendScript is not the same in both apps, at the very core level.

As the bonus, I also gave a look at ScCore.dll in Photoshop CC 2019.

Guess what, it still uses Expat 1.95.7, as does ID CC 2018.

Maybe just a coincidence, but it fits very well the symptoms!

Best,

Marc

[Laubender]

Hi Marc,

also test with target InCopy CC 2019 where I cannot see the same behavior InDesign CC 2019 is showing.

If I am looking for expat_ in ScCore.dll of InCopy CC 2019 I can see version expat_2.2.5.

Maybe there is more to it than the version of expat alone.

Regards,
Uwe

[Marc Autret]

Damn! Expat was my ideal suspect…

So there is something else, or something more.

[Noob question: I don't know how InCopy works, or whether it may interact with ID at some level… Are the apps fully disconnected?]

@+

Marc

[Harbs.]

Mark,

Whether right or wrong, I always enjoy your analysis.

InCopy uses the same code base as InDesign (and plugins for one should usually compile for the other), but the applications should be disconnected.

[Trevor:]

Regarding the validity of these characterless see Valid characters in XML - Wikipedia

It could explain why they work in one version and not the other but still it's strange.

[Trevor:]

N.b. The regex was a bit careless should not include \x09\x0a\0d ! it try edit it.

[TᴀW]

I saw that, but it's not clear to me which variety of XML our Ecmascript is based on.

[imartinho]

Different approach, but this should work.

var a = ["apples",  "pears", String.fromCharCode(22)];

var x = <food/>;

for (var idx = 0; idx < a.length; idx++) {

    var f = <fruit/>;

    f.appendChild(a[idx]);

    x.appendChild(f);

}

$.writeln(x.fruit[2].toString().charCodeAt());

x.toXMLString();

[TᴀW]

Good idea. That does work around the bug.

Still wonder what's going on with the ESTK parser in ID2019, though.

[Harbs.]

FWIW, I filed a bug report on some similar XML some weeks ago.

Unfortunately, I have no access to that particular bug anymore, so I can't update it with a reference to this thread. The bug number is 4211924.

Harbs

[TᴀW]

Until they fix the bug base, nobody can see anything. I also filed a bug, but following Marc's analysis perhaps the bug was in the previous versions.

[Laubender]

FWIW: This bug is still with InDesign 2020 version 15.0.0.151.

And InCopy 2020 as well.

Regards,
Uwe Laubender

( ACP )

Bug ID - ID-4212309