Overview
Features
Download
Documentation
Community
Add-Ons & Services

StreamCopier::copyToString() is extremely slow?

Please post support and help requests here.

StreamCopier::copyToString() is extremely slow?

Postby xcdemon05 » 01 Mar 2014, 06:56

When I grab the HTML from a webpage, often I find that I want that HTML I grab to be placed into a string. For that, I use the function StreamCopier::copyToString(). Here is how I am using it:

Code: Select all
      string url = "www.reddit.com";
      string fullPage;
      SocketAddress sa(url, 80);
      StreamSocket socket(sa);
      SocketStream str(socket);
      str << "GET /live/sw7bubeycai6hey4ciytwamw3a HTTP/1.1\r\n"
       "Host: www.reddit.com\r\n"
       "\r\n";
      str.flush();

      StreamCopier::copyToString(str, fullPage);


However, any time I do this (regardless of which website I grab the HTML from) the copyToString function takes FOREVER. I'm not sure if it's because I'm using it improperly, or if it's just the nature of the function and there's really nothing I can do about it.

One thing I can rule out for sure is it being an issue with using a slow computer (32GB ram and intel i7 3770k processor is what i'm working with).

On average the function takes about 5-10 minutes until it gets to the next line of code. I realize that if I were to put the function into a separate thread that I could execute other lines of code while it copies the HTML to the string, however I would still have to wait the 5-10 minutes for the data to be ready.

So I guess my question is: Is there anything I can do to speed up the process of copying the HTML data I get into a string? (including solutions not even involving copyToString() )
xcdemon05
 
Posts: 2
Joined: 22 Feb 2014, 19:03

Re: StreamCopier::copyToString() is extremely slow?

Postby guenter » 02 Mar 2014, 10:29

Use the proper tool for the job. In this case, Poco::Net::HTTPClientSession. The problem is not copyToString(), the problem is the server's persistent connection, keeping the connection open for a very long time after the actual data has been sent. Since copyToString obviously does not know about HTTP, it does not know when the request is completed and waits for the server to close the connection.
Alternatively, you could add a "Connection: close" header to your request. This will ask the server to close the connection after the request is done.
guenter
 
Posts: 1135
Joined: 11 Jul 2006, 16:27
Location: Austria


Return to Support

Who is online

Users browsing this forum: No registered users and 2 guests

cron