download progress

[iozo]

Hello,

I'm trying to use the CountingStream to retrieve and display the remote file size and download progress.
Apparently monitoring could be done from a Poco task but I can't get any counting info in the outputstream or inputstream.
Downloading a file is no problem but finding a way to show the actual download progress is not obvious.
I've see a few post here and there about the issue (with upload) but nothing really straightforward to use with download.


I'm doing something like this

try
{
Poco::URI uri(myremotefile);

std::auto_ptr<std::istream> pStr(Poco::URIStreamOpener::defaultOpener().open(uri));
// I have no idea how to retrieve the file size from this istream, CountingInputStream won't work here
// Any help would be nice for getting this info

Poco::Path p(true);
Poco::FileOutputStream fo(p.temp()+mylocalfilename);

// as streamcopier waits until the full copy is done, I'm trying to count the actual download progress from a Poco Thread
Poco::CountingOutputStream ostr(fo);
MySizeCounter sc(ostr);
Poco::Thread t;

t.start(sc);
Poco::StreamCopier::copyStream(*pStr.get(), fo);
fo.Close();
sc.done(); // stop the counting thread
t.join();
}
catch(Poco::Exception& exc)
{
std::cerr << exc.displayText() << std::endl;
}

This code downloads the file but does not count any progress (countingstream stays at 0).
Any help or even better a sample code would be great if anybody can help.

Thanks in advance.

[alex]

Try adding fo and ostr to a TeeOutputStream and then writing to it.

[iozo]

Yes, it works much better now.

For those who need it, I changed my code from

Code: Select all

Poco::CountingOutputStream ostr(fo);
MySizeCounter sc(ostr);
Poco::Thread t;

t.start(sc);
Poco::StreamCopier::copyStream(*pStr.get(), fo);

to

Code: Select all

Poco::CountingOutputStream ostr(fo);
MySizeCounter sc(ostr);

Poco::TeeOutputStream tee;
tee.addStream(fo);
tee.addStream(ostr);

Poco::Thread t;
t.start(sc);

Poco::StreamCopier::copyStream(*pStr.get(), tee);

and the thread can now count the bytes going through to display the progress bar or whatever indicator.

Now I still need to know the total file size of the remote file. x
The stream I'm getting is coming from this input std::auto_ptr<std::istream> pStr(Poco::URIStreamOpener::defaultOpener().open(uri));
How can I attach a countingstream to it and retrieve the total file size ?

[alex]

How can I attach a countingstream to it and retrieve the total file size ?

You don't need counting stream for that:

Code: Select all

std::streampos size = pStr->tellg();
pStr->seekg(0, std::ios::end);
size = pStr->tellg() - size;

[iozo]

I still have a couple of issues with this code.

1 : When I do the streamcopier with the TeeStream, the file size of the copied file is incorrect (about twice the size).
2 : Computing the size as you suggest was my original thought but the call pStr->seekg(0, std::ios::end) does not work -> ios_base::fail()

What can be wrong ?

[alex]

Counting should work (see code and output below) but finding out the file size will only work with local files. If you are doing FTP or HTTP, you'll have to resort to the respective protocol implementations. Looking through the code, it seems that Poco::FTPClientSession does not suport SIZE command (contributions welcome). If you are doing HTTP, you can use HEAD request to find out the file size from Content-Length.

Code: Select all

URI uri("file:///C:/temp/test.txt");
std::auto_ptr<std::istream> pStr(URIStreamOpener::defaultOpener().open(uri));
std::streampos size = pStr->tellg();
pStr->seekg(0, std::ios::end);
size = pStr->tellg() - size;
std::cout << "File size:" << size << std::endl;
pStr->seekg(0, std::ios::beg);

std::string file = TemporaryFile::tempName("C:\\temp\\");
std::cout << "File name:" << file << std::endl;
FileOutputStream fo(file);

CountingOutputStream co(fo);
TeeOutputStream tout;
tout.addStream(fo);
tout.addStream(co);

StreamCopier::copyStream(*pStr.get(), tout);
std::cout << "Characters transferred:" << co.chars() << std::endl;

Output:

Code: Select all

File size:3
File name:C:\temp\tmp404aaaaaa
Characters transferred:3

[iozo]

I'm doing HTTP. For the file size, I'll try the header option or maybe with a remote script but I was hoping to get a helper function out of the Poco library.

For the counting issue however, the pb is more complex.
I have a test file (Zip file) of a bout 1.5 MB.
If I do a streamcopier from source stream to file output direct, it works. The file I get in my temp is fine.
If I remplace the direct file output with a tee stream and only add the file output to the tee stream it still works, copied file is OK.

now, if I add the counting stream to the tee, it still download the file, I can see the data going through the counting stream, but the file is corrupted. It's size is about twice the normal.
The size counted by the streamcounter seams correct however.

[iozo]

I did another test with a text file that I dropped on the remote server

The file contains a single string
abcdef

with the teestream the file I download contains
aabbccddeeff

[alex]

You are right, you can omit the TeeStream and write straight to CountingOutputStream. This will probably fix the file corruption as well. As for counting in the thread, you can pass the reference to the counting stream there and check the count periodically. Threads are not thread-safe, though. Since you won't be writing in the thread, you should be ok, but it would still be good to serialize access with a mutex, in which case, you'd have to do the actual file streaming by hand, rather than using StreamCopier.

[alex]

I'm doing HTTP. For the file size, I'll try the header option or maybe with a remote script but I was hoping to get a helper function out of the Poco library.

You can use Poco::Net::HTTPClientSession and send HEAD request to to get file size.