binary data in a form field gets changed

[DonPedroElBuenoI]

I'm about to migrate a Coldfusion9 application running on SunOS to Coldfusion10 on Suse Linux.

The webpages work perfectly on the new installation.

However there is also kind of a system interface where clients post forms to a CF page where one of the form fields (intentionally) contains binary data mixed with plain text.

This worked perfectly on the old installation.

The formfield is taken out of the request using

<cfset requestContent="#form.fieldt#"/>

On the new installation this works for the plain text part, but the binary data is somehow changed.

Any idea how I can prevent Coldfusion from changing the binary data?

Thanks

Pedro

[BKBK]

Can you please bear with me. What you say confuses me a bit. What do you mean by "This worked perfectly on the old installation"? What failed to work, and what did you expect?

Also, you say, "one of the form fields (intentionally) contains binary data mixed with plain text". However, the variable you have defined, requestContent, is neither binary nor a combination of binary and string. It is just a string.

[DonPedroElBuenoI]

"This worked perfectly on the old installation": What I meant is, that the data in the form field (i.e. binary data mixed with plain text) was in the variable requestContent as it was sent by the client.

"What failed to work": In Coldfusion10 on Suse Linux, the binary data is not anymore as it was sent by the client (altough the plain text part remains correct). Some bytes (of the binary data) are dropped and the others are changed.

"What did you expect": As described for the old installation.

Thanks for your help

Pedro

[BKBK]

Umhum, I see. What might likely have changed between the 2 installations is the encoding. If so, then the following should help:

<cfif isDefined("form.fieldt")>

    <cfset setEncoding("form", "UTF-8")>

    <cfset requestContent="#form.fieldt#">  

</cfif>

Should you wish to read more on encoding, take a look at this article by the inimitable Joel Spolsky: The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About ....

[DonPedroElBuenoI]

Thanks for that, but unfortunately, it didn't make it work.

With the <cfset setEncoding("form", "UTF-8")>, the result was exactly the same as before.

Btw the encoding on the old and on the new installation is set to ISO-8859-1.

This is set in the CF Administrator:

Java File Encoding:    ISO8859_1

Java Default Locale:   en_US

JVM arguments:        -Dfile.encoding=ISO-8859-1

If in the .cfm I put the following:

<cffile action= "write" file="/tmp/somefile.txt" output= "#getEncoding("form")#">

the contents of somefile.txt is "ISO-8859-1" (i.e. of course without the <cfset setEncoding("form", "UTF-8")>).

Any other suggestions?

Thanks

Pedro

[BKBK]

Could you show us the code for your form?

[DonPedroElBuenoI]

What do you mean by show us your from? The data content?

Is there a way I can dump the contents of the form to a file (or anywhere else I can read it)?

Below is just an example from the documentation... (the "Userdata" is the part which may be binary)

POST /DataInterface HTTP/1.0

Content-Type: multipart/form-data; boundary=123

User-Agent: MyClient/1.0

Content-Length: 232

--123

Content-Disposition: form-data; name="field"; filename=""

Content-Type: text/plain

PBRequest/2.0

SHA1-MAC: 977db7bd36625bfb67d1a1bc67e306230ef332be

Timestamp: 20020415T130553+0200

Type: Command

"Userdata"

--123--

[BKBK]

Ah, I had assumed the form is on your site. Your answer suggests otherwise.

In any case, it seems there are at least 2 form fields coming in, a binary (file) and plain text. You could just treat the form post as an upload. If you know the MIME type of the binary, then you could capture it and store it as a file.

Here follows a simple example in which I upload a PDF file (a binary) and some text. Ignore the fact that it is in Windows. What you are interested in are the 2 cffile tags. In your case, fileField = "Userdata" and output="#form.field#"

<cfif isDefined("Form.FileContents") >

   <cffile action = "upload"

       strict="false"

        fileField = "FileContents"

        destination = "c:\temp\upload\uploadFile.pdf"

        accept = "application/pdf, binary/octet-stream"

        nameConflict = "MakeUnique">

   <cffile action="write" file="c:\temp\upload\uploadTxt.txt" output="#form.txtContent#">

Done uploading

<cfelse>

    <form method="post" action=<cfoutput>#cgi.script_name#</cfoutput>

        name="uploadForm" enctype="multipart/form-data">

        <input name="FileContents" type="file">

         <input name="txtContent" type="text">

        <br>

        <input name="submit" type="submit" value="Upload File">

    </form>

</cfif>

[DonPedroElBuenoI]

Seems I was very unclear in post #6... Sorry about that.

What you see there is the data we receive, not what we send...

And it is not about the file upload.

Altough we do also accept file uploads, this is not the part which causes problems.

The problem is only when the binary data is in the post.

[DonPedroElBuenoI]

I made one more observation which is interesting:

After examining the differences more closely in a hex-editor I installed for that, i noticed that the difference is that all null-bytes (Hex 00) are changed to (Hex 20).

There seem to be no other differences between what is sent from the client and what I get in requestContent.

[BKBK]

DonPedroElBuenoI wrote:

I made one more observation which is interesting:

After examining the differences more closely in a hex-editor I installed for that, i noticed that the difference is that all null-bytes (Hex 00) are changed to (Hex 20).

There seem to be no other differences between what is sent from the client and what I get in requestContent.

There you go. My instincts tell me it will come down to encoding. You said:

This is set in the CF Administrator:

Java File Encoding:    ISO8859_1

Java Default Locale:   en_US

JVM arguments:        -Dfile.encoding=ISO-8859-1

They say ISO-8859-1 is byte-preserving, but I wonder what will happen if, just for the sake of testing, you change the encoding to UTF-8.

[DonPedroElBuenoI]

Thanks, but I tried that before.

And I tried both, by setting UTF-8 in the administrator as well as the way you suggest in your post #3 <cfset setEncoding("form", "UTF-8")> in the cfm.

It actually makes it even worse. Not only are the Hex 00 converted to Hex 20, but also after some other binary data an additional byte is inserted.

[BKBK]

Sorry to hear. I wonder whether the problem results from the change from SunOS to Suse Linux, rather than the change from ColdFusion 9 to ColdFusion 10. By the time we apply encoding, it may be too late.

The Linux machine might be converting Hex 00 automatically to Hex 20. Any way to configure the settings on Suse Linux?

[BKBK]

Another suggestion. Could it also be that the form fields are coming in as before, but what you use to open or view them has changed? Again, that might be the application converting Hex 00 automatically to Hex 20. If so, it might be possible to configure it to prevent conversion.

[DonPedroElBuenoI]

All we basically do is to write the contents of the form field to a file.

I already tried FileWrite() with different encodings (i.e. ISO-8859-1, UTF-8 and several others, as well as no encoding at all (which should be the same as ISO-8859-1)).

Of course theoretically the problem could have been in the way this file is read. That's why I installed a hex-editor to make sure this is not the case.

[BKBK]

DonPedroElBuenoI wrote:

That's why I installed a hex-editor to make sure this is not the case.

I scrolled back up and was duly reminded. You've done a good job at excluding the cases.

So, it's time for new ideas. There is always the possibility that this behaviour changed between ColdFusion 9 and 10. We could therefore look a bit more in this direction.

You asked earlier if there is a way to save the form field's content. What about making Coldfusion to treat it as an upload.

<cfif isDefined("Form.Userdata") >

   <cffile action = "upload"

       strict="false"

        fileField = "Userdata"

        destination = "\your-server-path\uploadFile.ext" <!---Ext is a place holder for file extension--->

        accept = "binary/octet-stream" <!--- Even better, use MIME type for the file extension--->

        nameConflict = "MakeUnique">

</cfif>

Second test. For the purposes of elimination I would temporarily install a test version of ColdFusion 10 on SunOS, and see whether the problem occurs there. If it does, then we have enough information to report a ColdFusion 10 bug.

[DonPedroElBuenoI]

Umhum, I don't really understand what the purpose of that file upload is (which may just be due to the fact that I'm far away from being a CF-expert).

Anyway, when I tried it, I got the error "CF: The form field did not contain a file." (of course I changed "Userdata" to the real fieldname in the form).

Another interesting information: I changed the JkLogLevel in Apache from info to debug. This writes the complete data traffic between Apache and coldfusion in hexadecimal to the logfile. There I can see, that the messages are still correct (i.e. they still contain the null bytes).

[DonPedroElBuenoI]

Interesting observation: CF apparently doesn't consider the content of my form field as being binary. When I try the function BinaryEncode() on the form field, I get the error "Parameter 1 of the BinaryEncode function must be a valid binary object."

Is there a way I can tell CF that it is a binary object? I haven't found anything in the manual...

And one more question:

Is there a way I can access the complete post as it was sent by the client (i.e. starting from "POST /DataInterface HTTP/1.0" to the very end) without having CF parsing out the fields?

Parsing out the relevant field by myself would be easy enough.

[DonPedroElBuenoI]

Just an update...

I thought I had found the solution to my last question by myself as I found the function GetHttpRequestData().

I was wrong! When I tried the follwing

  <cfset x = GetHttpRequestData()>

  <cfif isBinary(x.content)>

    <cfscript>

      FileWrite("/tmp/binarycontent", "#x.content#");

      encContent = binaryEncode("#x.content#", "Hex");

      FileWrite("/tmp/encbinarycontent", "#encContent#");

    </cfscript>

  <cfelse>

    <cffile action      = "write"

            file        = "/tmp/nonbinarycontent"

            output      = "#x.content#"

            addNewLine  = "no"

    >

  </cfif>

the files /tmp/binarycontent and /tmp/encbinarycontent were written, but they were both empty.

[BKBK]

DonPedroElBuenoI wrote:

I thought I had found the solution to my last question by myself as I found the function GetHttpRequestData().

I was wrong!

Indeed. GetHttpRequestData().content is a concatenation of all the form fields and their respective values, in the form

field1=value1&field2=value2&field3=value3&...

  <cfset x = GetHttpRequestData()>

  <cfif isBinary(x.content)>

    <cfscript>

      FileWrite("/tmp/binarycontent", "#x.content#");

      encContent = binaryEncode("#x.content#", "Hex");

      FileWrite("/tmp/encbinarycontent", "#encContent#");

    </cfscript>

  <cfelse>

    <cffile action      = "write"

            file        = "/tmp/nonbinarycontent"

            output      = "#x.content#"

            addNewLine  = "no"

    >

  </cfif>

the files /tmp/binarycontent and /tmp/encbinarycontent were written, but they were both empty.

From what I have just said,  x.content is a string, not a binary. Therefore the else-block of this code will run.

[DonPedroElBuenoI]

From what I have just said,  x.content is a string, not a binary. Therefore the else-block of this code will run.

Well actually the manual says the following about GetHttpRequestData().content:

Raw content from form submitted by client, in string or binary format. For content to be considered string data, the FORM request header "CONTENT_TYPE" must start with "text/" or be special case "application/x-www-form-urlencoded". Other types are stored as a binary object.

This does not seem to work, at least not in my case.

And by the way, the if block is executed in my code as you can see from the names of the files being written (unfortunately empty).

[BKBK]

You are right. I have had a look at the manual. With Content-Type = multipart/form-data, ColdFusion indeed promises to return a binary. Since it returns zero output in your case, this can only mean something has gone wrong.

I have decided to take a deeper look into this. Expect feedback from me soon.

[DonPedroElBuenoI]

@BKBK:

Thanks a lot for your support.

This is an important project for us and even more so for our customer. The project is already delayed because of this issue. It sheds a bad light on us and Colfusion, so we want to get this fixed as soon as possible.

We really appreciate your help

[BKBK]

Don Pedro,

I have been busy testing the behaviour of getHttpRequestData(), when a form is submitted with method="post" and enctype="multipart/form-data". I am on ColdFusion 10.0.13.287689.

Now, I can fully understand your frustration. I am, unfortunately, about to dash your hopes even further.

You were correct in expecting getHttpRequestData() to return the complete post that the client sent. That is in fact what its function is. However, I have found that the scenario I describe in the first paragraph leads to a bug.

To reproduce the bug, run the following code:

<cfif isDefined("Form.fieldContent")>

    <cfoutput>

    isBinary(gethttpRequestData().content): #isBinary(gethttpRequestData().content)#<br>

    binaryEncode(gethttpRequestData().content, 'hex'): #binaryEncode(gethttpRequestData().content, 'hex')#<br>

    </cfoutput>

    <cfdump var="#gethttpRequestData().content#" label="gethttpRequestData().content dump">

    <cffile action="write" file="#expandPath('testBinary')#" output="#gethttpRequestData().content#">

</cfif>

<cfoutput><form method="post" action="#cgi.SCRIPT_NAME#" enctype="multipart/form-data"></cfoutput>

<input name="fieldContent" type="text">

<br>

<input name="sbmt" type="submit" value="Send">

</form>

Enter '0123456789' in the text field and submit the form. The result is as follows:

1) isBinary(gethttpRequestData().content) returns 'Yes'.

2) binaryEncode(gethttpRequestData().content, 'hex') returns an empty string.

3) The dump shows no detail, is therefore apparently blank.

4) The file, testBinary, is created, but has 0 bytes.

For a second opinion, I ran the same code on Railo 4.1.2.005. This version corresponds to ColdFusion 10. The Railo result was as follows:

1) isBinary(gethttpRequestData().content) returned 'true'.

2) binaryEncode(gethttpRequestData().content, 'hex') returned the string 2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D32303133303136383935373631330D0A436F6E74656E742D446973706F736974696F6E3A20666F726D2D646174613B206E616D653D226669656C64436F6E74656E74220D0A0D0A303132333435363738390D0A2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D32303133303136383935373631330D0A436F6E74656E742D446973706F736974696F6E3A20666F726D2D646174613B206E616D653D2273626D74220D0A0D0A53656E640D0A2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D2D32303133303136383935373631332D2D0D0A

3) The dump returned a table entitled 'Native Array (byte[])', consisting of 2 rows. The contents of the rows were as follows:

First row

Column 1: Raw   

Column 2: [45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,50,48,49,51,48,49,54,56,57,53,55,54,49,51,13,10,67,111,110,116,101,110,116,45,68,105,115,112,111,115,105,116,105,111,110,58,32,102,111,114,109,45,100,97,116,97,59,32,110,97,109,101,61,34,102,105,101,108,100,67,111,110,116,101,110,116,34,13,10,13,10,48,49,50,51,52,53,54,55,56,57,13,10,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,50,48,49,51,48,49,54,56,57,53,55,54,49,51,13,10,67,111,110,116,101,110,116,45,68,105,115,112,111,115,105,116,105,111,110,58,32,102,111,114,109,45,100,97,116,97,59,32,110,97,109,101,61,34,115,98,109,116,34,13,10,13,10,83,101,110,100,13,10,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,45,50,48,49,51,48,49,54,56,57,53,55,54,49,51,45,45,13,10]

Second row

Column 1: Base64 Encoded   

Column 2: LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0yMDEzMDE2ODk1NzYxMw0KQ29udGVudC1EaXNwb3NpdGlvbjogZm9ybS1kYXRhOyBuYW1lPSJmaWVsZENvbnRlbnQiDQoNCjAxMjM0NTY3ODkNCi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tMjAxMzAxNjg5NTc2MTMNCkNvbnRlbnQtRGlzcG9zaXRpb246IGZvcm0tZGF0YTsgbmFtZT0ic2JtdCINCg0KU2VuZA0KLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0yMDEzMDE2ODk1NzYxMy0tDQo=

4) The file, testBinary, is created. It has 257 bytes, and the following content:

-----------------------------20130168957613

Content-Disposition: form-data; name="fieldContent"

0123456789

-----------------------------20130168957613

Content-Disposition: form-data; name="sbmt"

Send

-----------------------------20130168957613--

[BKBK]

I have just filed a bug report: https://bugbase.adobe.com/index.cfm?event=bug&id=3700163 .