REXML don't support UTF8 BOM on Windows 11

[andryua]

Hi!

After update Puppet aggent to 8.9.0 (they updated REXML from 3.2.5 to 3.3.6 version) - puppet chocolatey provider can't use correct parsing XML witn UTF-8 BOM codding. Only if enable support beta UTF8 WordWide in Windows 10/11 - everything work fine. If disable this feature - puppet send error again. (tried on cyrrilic and english - same issues)

Debug: Gathering sources from 'C:\ProgramData\chocolatey\config\chocolatey.config'.
Error: Could not prefetch chocolateysource provider 'windows': Malformed XML: Content at the start of the document (got '')
Line: 1
Position: 47
Last 80 unconsumed characters:
<?xml version="1.0" encoding="utf-8"?>
Warning: /Stage[main]/Windows::Chocolatey/Chocolateysource[chocolatey]: Skipping because provider prefetch failed
Debug: /Stage[main]/Windows::Chocolatey/Chocolateysource[chocolatey]: Resource is being skipped, unscheduling all events

[naitoh]

I think REXML supports UTF-8 with BOM.

rexml/lib/rexml/source.rb

Lines 193 to 194 in b70388c

	elsif @scanner.scan(/\xef\xbb\xbf/n)
	detected_encoding = "UTF-8"

@andryua

Could you give us an XML file that can reproduce this problem?

[andryua]

https://github.com/user-attachments/files/18328878/chocolatey.txt

[andryua]

if i install last version of Puppet and replace REXML tool from 3.3.6 on other older (3.2.6 or 3.2.5) - all works. if I replace on 3.2.7 and newer - puppet send errors. (UTF8 WorldWide support - disabled!)

[naitoh]

I can't seem to reproduce this in my m1 Mac environment...

$ ruby -v
ruby 3.3.4 (2024-07-09 revision be1089c8ec) [arm64-darwin22]
$ gem list rexml
*** LOCAL GEMS ***

rexml (3.3.6)
$ ruby -r "rexml/document" -e 'REXML::Document.new(File.open("./chocolatey.txt"))' 

[andryua]

on Windows 11 or 10 any build!!!

[andryua]

and try not File.open - try File.read (same using puppetlabs-chocolatey in their code)

[naitoh]

@andryua

I tried it on Windows 11 and it worked fine.
Do the following commands work in your environment?

>ruby -v
ruby 3.3.6 (2024-11-05 revision 75015d4c1f) [x64-mingw-ucrt]

>gem list rexml
rexml (3.3.6)

>ruby -r "rexml/document" -e 'puts REXML::XPath.each(REXML::Document.new(File.read(".\\Documents\\chocolatey.txt")),  "/chocolatey/config/add[@key]").inspect'
#<Enumerator: [<add key='cacheLocation' value='C:\ProgramData\chocolatey\cache' description='Cache location if not TEMP folder. Replaces `$env:TEMP` value for choco.exe process. It is highly recommended this be set to make Chocolatey more deterministic in cleanup.'/>, <add key='containsLegacyPackageInstalls' value='true'/>, <add key='commandExecutionTimeoutSeconds' value='2700' description='Default timeout for command execution. &apos;0&apos; for infinite.'/>, <add key='proxy' value='' description='Explicit proxy location.'/>, <add key='proxyUser' value='' description='Optional proxy user. Requires explicit proxy configured.'/>, <add key='proxyPassword' value='' description='Optional proxy password. Encrypted. Requires explicit proxy and proxyUser configured.'/>, <add key='webRequestTimeoutSeconds' value='30' description='Default timeout for web requests.'/>, <add key='proxyBypassList' value='' description='Optional proxy bypass list. Comma separated. Requires explicit proxy configured.'/>, <add key='proxyBypassOnLocal' value='true' description='Bypass proxy for local connections. Requires explicit proxy configured.'/>, <add key='upgradeAllExceptions' value='' description='A comma-separated list of package names that should not be upgraded when running `choco upgrade all&apos;. Defaults to empty.'/>, <add key='defaultTemplateName' value='' description='Default template name used when running &apos;choco new&apos; command.'/>, <add key='defaultPushSource' value='' description='Default source to push packages to when running &apos;choco push&apos; command.'/>]:each>

[andryua]

ok, thanks! maybe puppet did something wrong.

[andryua]

Hi!
I reopened, because if in read include encodding: 'bom|utf-8' - everuthing work fine, but, if we using read without encodding parameter - we got bom symbols before xml file content and we have errors like in first post. Look on my screenshots, please (i attached screenshots with version of ruby, rexml and codepage of system console in Windows 10 or 11). Thanks!

p.s. maybe problem is in cp866? I using Ukainian language in Windows.

[tompng]

Malformed XML: Content at the start of the document (got '')

'' can be created by this step.

encoding = [Encoding::IBM437, Encoding::IBM865].sample
bom = "\xef\xbb\xbf".b
bom.force_encoding(encoding).encode('utf-8').force_encoding(encoding).encode('utf-8')
#=> ""

So the data passed to REXML is already corrupt by re-encode with multiple encoding several times. I think this is not a bug of REXML.

Since the xml has encoding="utf-8", reading it in ascii, binary or ascii-8bit works fine, but reading with other wrong encoding corrupts the data. Corrupt data can't be repaired by re-encoding or any other process.

If your Encoding.default_external is not ascii, ascii-8bit or it does not match the file encoding, you need to specify the encoding when reading the file.

[naitoh]

@tompng

So the data passed to REXML is already corrupt by re-encode with multiple encoding several times. I think this is not a bug of REXML.

Thanks!
I think so.

@andryua

Do the following commands work in your environment?

>ruby -r "rexml/document" -e 'puts REXML::XPath.each(REXML::Document.new(File.read(".\\Documents\\chocolatey.txt")),  "/chocolatey/config/add[@key]").inspect'