Overview
Features
Download
Documentation
Community
Add-Ons & Services

Stream convert

Please post support and help requests here.

Stream convert

Postby fbraem » 24 Feb 2009, 21:46

I try to convert an UTF-16 file into UTF-8, but when I display the converted data I only see garbage. What am I doing wrong?

Code: Select all
      Poco::FileInputStream fis("c:\\temp\\test.txt");
      Poco::InputStreamConverter isc(fis, Poco::TextEncoding::byName("UTF-16"),
                                          Poco::TextEncoding::byName("UTF-8"));
      Poco::InputLineEndingConverter lec(isc, "\r\n");
      while(! lec.eof() )
      {
        std::string line;
        std::getline(lec, line);
        std::cout << line;
      }
Zumuta!, that's the way to do IT!
fbraem
 
Posts: 101
Joined: 11 Aug 2008, 22:47
Location: Belgium

Re: Stream convert

Postby guenter » 24 Feb 2009, 22:52

Very probably the byte order of your UTF-16 file is not the one expected by the converter.
Create an instance of Poco::UTF16Encoding and set the byte order accordingly.

Poco::UTF16Encoding utf16(Poco::UTF16Encoding::LITTLE_ENDIAN_BYTE_ORDER);
Poco::InputStreamConverter(fis, utf16, Poco::TextEncoding::byName("UTF-8"));

You can also check if your text file contains a byte order mark - the Unicode character 0xFEFF. In this case you can read a 16-bit integer from the file and pass it to the constructor of UTF16Encoding.
guenter
 
Posts: 1107
Joined: 11 Jul 2006, 16:27
Location: Austria

Re: Stream convert

Postby fbraem » 25 Feb 2009, 21:41

Thanks Guenter. The following code works:

Code: Select all
      Poco::FileInputStream fis("c:\\temp\\test.txt");
      char bom[2];
      fis.read(bom, 2);
      Poco::UTF16Encoding utf16(Poco::UTF16Encoding::LITTLE_ENDIAN_BYTE_ORDER);
      Poco::InputStreamConverter isc(fis, utf16,
                                          Poco::TextEncoding::byName("UTF-8"));
      Poco::InputLineEndingConverter lec(isc, "\r\n");
      while(! lec.eof() )
      {
        std::string line;
        std::getline(lec, line);
        std::cout << line;
      }
Zumuta!, that's the way to do IT!
fbraem
 
Posts: 101
Joined: 11 Aug 2008, 22:47
Location: Belgium


Return to Support

Who is online

Users browsing this forum: No registered users and 2 guests