Unicode problem in Linux

[ehinc]

Hi there!
Please consider following snippet:

Code: Select all

wstring wstrUTF16 = L"12345";
string strUTF8;
Poco::UnicodeConverter::toUTF8(wstrUTF16, strUTF8);
if ( strUTF8 != "12345" )
{
   printf("Conversion failed!\n");
}
else
{
   printf("Conversion worked!\n");
}


This code prints "Conversion worked!" on Windows and "Conversion failed!" on Linux!
Further investigating the Linux version I noticed that the string strUTF8 contains "1\02\03\04\05\0".
Is this a bug? How can I fix it?
(I'm using Poco-1.4.3p1)
Regards,
Christoph

[alex]

gcc std::wstring is 32 bit by default. Compile your code with -fshort-wchar, then try this and see what you get:

Code: Select all

wstring wstrUTF16 = L"12345";
string strUTF8;
if ( 16 == sizeof(wchar_t) )
{
  Poco::UnicodeConverter::toUTF8(wstrUTF16, strUTF8);
  if ( strUTF8 != "12345" )
  {
     printf("Conversion failed!\n");
  }
  else
  {
     printf("Conversion worked!\n");
  }
}
else printf("std::wstring is not of appropriate size (%d)!\n", sizeof(wchar_t));

[ehinc]

Yes, I know that Linux uses 4 byte wchar_t by default.
However I cannot and don't want to change this compiler setting in my project.
IMHO this is a bug in POCO.
There should at least be something like this:

Code: Select all

if ( sizeof(wchar_t) == 4 )
{
  doThis();
}
else if ( sizeof(wchar_t) == 2 )
{
  doThat();
}


I solved my immediate problems by "shrinking" the 4 byte wchar_t to 2 bytes before calling UnicodeConverter.
Regards,
Christoph.

[alex]

This way of communicating won't take you far.

If you know wchar_t is 32 bit, why are you trying to deal with as if it was 16 bit and why are you surprised when it does not do what you intend?
If you think this is a bug, why don't you file a bug report? Documentation (and parameter names) clearly states that this is a 16-bit value.
This is not a bug any more than non-standardized wchar_t size is a bug. Right or wrong, this is currently a feature. You can try to fix it by submitting a patch in SourceForge. I will personally look into it and do the best to make it more robust.

Thanks.

[ehinc]

This way of communicating won't take you far.

If you know wchar_t is 32 bit, why are you trying to deal with as if it was 16 bit and why are you surprised when it does not do what you intend?
If you think this is a bug, why don't you file a bug report?

Because I wanted to be sure that it really is a bug. Thus my original post: "Is this a bug?"

Documentation (and parameter names) clearly states that this is a 16-bit value.

Actally it doesn't! Using "wchar_t" determines the size of a single data element, not the type of data you choose to store in it. You could well store UTF32 using multiple "char"s.
There simply is no standard character type which is 16bits wide, so you'll have to use the 32bit wchar_t. Putting shorter info in longer datatypes must work - otherwise it would also be a problem to assign a "char" to an "int" variable.

This is not a bug any more than non-standardized wchar_t size is a bug. Right or wrong, this is currently a feature.

Well, I do expect the function to work as described in the documentation - which it does not.

You can try to fix it by submitting a patch in SourceForge. I will personally look into it and do the best to make it more robust.

I'll look into it.
Regards,
Christoph.

[alex]

There simply is no standard character type which is 16bits wide, so you'll have to use the 32bit wchar_t.

The problem is that there is no standard character type width, period. I understand your widening data handling logic but storing UTF-16 in 32-bit wchar container would be somewhat unusual way of doing business, IMO. We'll definitely appreciate and look into your proposal/contribution here.

Well, I do expect the function to work as described in the documentation - which it does not.

This class is mainly used for working with the Unicode Windows APIs and probably won't be of much use anywhere else.

The code probably should have been #ifdef'd for Windows only and the problem would have been solved (until someone had an itch to port it elsewhere).

I'll look into it.

Thanks.

[ehinc]

The problem is that there is no standard character type width, period. I understand your widening data handling logic but storing UTF-16 in 32-bit wchar container would be somewhat unusual way of doing business, IMO.

I agree that this is not beautiful, but it works!
All std::wstring methods work as well. That's why I was surprised that this method doesn't work.

This class is mainly used for working with the Unicode Windows APIs and probably won't be of much use anywhere else.

If my previous posts sounded disrespectful that was purely unintentional - I love the POCO framework and its readable code!

My suggestion:

Code: Select all

...
typedef basic_string<UInt16> UTF16String;
...
void UnicodeConverter::toUTF8(const std::wstring& utf16String, std::string& utf8String)
{   
   utf8String.clear();
   UTF8Encoding utf8Encoding;
   UTF16Encoding utf16Encoding;
   UTF16String tempUtf16String(utf16String.begin(), utf16String.end()); // here the excess is stripped
   TextConverter converter(utf16Encoding, utf8Encoding);
   converter.convert(tempUtf16String.data(), (int) tempUtf16String.size() * sizeof(UInt16), strUTF8);
}


When using Linux with 32bit wchar_t the excess is stripped, when using Windows 16bit wchar_t its just copied over.

Regards,
Christoph.

[alex]

There is UTF32Encoding in the trunk (this will become 1.5), so if you need full set of conversions between UTF-8,16 and 32, trunk would be the best place to contribute.

[alex]

One of our contributors is currently working on this, see rev. 1904.
If you want to contribute to the effort, let me know and I'll put you in touch with him.