StreamCopier should return a larger type

[tkirby]

I've been using Poco to copy large amounts of data from streams with a StreamCopier. As a sanity check, I wanted to use the return value from copyStream to make sure that I'd gotten what I expected to get. Unfortunately, Poco returns a std::streamsize from the StreamCopier functions, which is a 32 bit value in all of the environments I use. I regularly read more than 4GiB from a single stream, which overflows a 32 bit value, making it useless.

I'd be really nice if Poco returned a larger type from its StreamCopiers, or offered parallel versions of the functions that returned Poco::UInt64.

I've successfully used the following patch to do what I want, but it'd be nice to see an official offering from Poco. I made the diff against version 1.4.1p1, but it's so simple it should apply cleanly to 1.4.2 and later versions as well.

Code: Select all

Index: poco-1.4.1p1-all/Foundation/include/Poco/StreamCopier.h
===================================================================
--- poco-1.4.1p1-all/Foundation/include/Poco/StreamCopier.h     (revision 1604)
+++ poco-1.4.1p1-all/Foundation/include/Poco/StreamCopier.h     (working copy)
@@ -54,16 +54,19 @@
{
public:
     static std::streamsize copyStream(std::istream& istr, std::ostream& ostr, unsigned bufferSize = 8192);
+    static Poco::UInt64 copyStream64(std::istream& istr, std::ostream& ostr, unsigned bufferSize = 8192);
                /// Writes all bytes readable from istr to ostr, using an internal buffer.
                ///
                /// Returns the number of bytes copied.

     static std::streamsize copyStreamUnbuffered(std::istream& istr, std::ostream& ostr);
+    static UInt64 copyStreamUnbuffered64(std::istream& istr, std::ostream& ostr);
                /// Writes all bytes readable from istr to ostr.
                ///
                /// Returns the number of bytes copied.

     static std::streamsize copyToString(std::istream& istr, std::string& str, unsigned bufferSize = 8192);
+    static Poco::UInt64 copyToString64(std::istream& istr, std::string& str, unsigned bufferSize = 8192);
                /// Appends all bytes readable from istr to the given string, using an internal buffer.
                ///
                /// Returns the number of bytes copied.
Index: poco-1.4.1p1-all/Foundation/src/StreamCopier.cpp
===================================================================
--- poco-1.4.1p1-all/Foundation/src/StreamCopier.cpp    (revision 1604)
+++ poco-1.4.1p1-all/Foundation/src/StreamCopier.cpp    (working copy)
@@ -101,5 +101,63 @@
     return len;
}

+Poco::UInt64 StreamCopier::copyStream64(std::istream& istr, std::ostream& ostr, unsigned bufferSize)
+{
+       poco_assert (bufferSize > 0);

+       Buffer<char> buffer(bufferSize);
+       Poco::UInt64 len = 0;
+       istr.read(buffer.begin(), bufferSize);
+       std::streamsize n = istr.gcount();
+       while (n > 0)
+       {
+               len += n;
+               ostr.write(buffer.begin(), n);
+               if (istr && ostr)
+               {
+                       istr.read(buffer.begin(), bufferSize);
+                       n = istr.gcount();
+               }
+               else n = 0;
+       }
+       return len;
+}
+
+Poco::UInt64 StreamCopier::copyToString64(std::istream& istr, std::string& str, unsigned bufferSize)
+{
+       poco_assert (bufferSize > 0);
+
+       Buffer<char> buffer(bufferSize);
+       Poco::UInt64 len = 0;
+       istr.read(buffer.begin(), bufferSize);
+       std::streamsize n = istr.gcount();
+       while (n > 0)
+       {
+               len += n;
+               str.append(buffer.begin(), static_cast<std::string::size_type>(n));
+               if (istr)
+               {
+                       istr.read(buffer.begin(), bufferSize);
+                       n = istr.gcount();
+               }
+               else n = 0;
+       }
+       return len;
+}
+
+Poco::UInt64 StreamCopier::copyStreamUnbuffered64(std::istream& istr, std::ostream& ostr)
+{
+    char c;
+    Poco::UInt64 len = 0;
+    istr.get(c);
+    while (istr && ostr)
+    {
+        ++len;
+        ostr.put(c);
+        istr.get(c);
+    }
+    return len;
+}
+
+
} // namespace Poco


Anyone else feel the need for this improvement? Or should I be taking some other approach to read large amounts of data from a stream?
--
Tim

[guenter]

I've added this to 1.4.3.