Latest Inquiries - Data Extraction Software

Template retried 3 times without being configured to do so

Submitted: 1/12/2016

Hi,

I am trying to find out why VWR is retrying a PageNavigation template (I did not try with other types) multiple times even if I did not set any configuration to do so.

If you open the script I included, you will see that my "next_page" template is modified to go straight to a 404 error. This template is configured to retry 1 time upon error with 5 seconds delay.

Here is the output I get:

15:03 Deleting old data...
15:03 Extracting data with web browser agent v2.
15:03 Processing start URL https://www.google.com
15:03 Processing single link PageNavigation template next_page
15:03 Processing PageNavigation template next_page (page 2)
15:03 Navigation error. The web server returned a status code indicating failure
15:03 next_page action failed. Retrying...
15:03 Navigation error. The web server returned a status code indicating failure
15:03 Error activating page navigation link: http://www.google.com/going_nowhere.html
15:03 Processing PageNavigation template next_page (page 2)
15:03 Navigation error. The web server returned a status code indicating failure
15:03 next_page action failed. Retrying...
15:03 Navigation error. The web server returned a status code indicating failure
15:03 Error activating page navigation link: http://www.google.com/going_nowhere.html
15:03 Processing PageNavigation template next_page (page 2)
15:03 Navigation error. The web server returned a status code indicating failure
15:03 next_page action failed. Retrying...
15:03 Navigation error. The web server returned a status code indicating failure
15:03 Error activating page navigation link: http://www.google.com/going_nowhere.html
15:03 Updating error reporting...
15:03 Starting export...
15:03 Generating export data...
15:03 Data extraction completed

As you can see, VWR attempts the template 3 times and retry each time to recover from the page load error. I did the same thing with "Max retries" set to 0 for the next_page template and I got the same behavior less the "action failed. Retrying...".

I can't find any configuration to avoid this. Is the "3 times retry" by design in VWR or is it a bug? 

Thanks.

Google.rip

Replied: 1/20/2016 4:53:54 AM

Sorry for the delay in getting back to you.

This behavior is actually by design. It happens only in web browser mode and only for pagination elements. The reasoning is that if pagination fails, the whole project is likely to fail. 

The "Retry action" option only retries the action, but if the browser has navigated to an error page, the retry will continue to fail. The pagination element will attempt to move back to it's previous state before retrying, and it will do that 3 times. Unfortunately, it this time there's no way to control the number of these special retries.
Replied: 1/13/2016 10:31:14 AM
Yes, I've set 'max retries.' = 0 for page navigation template, it still keep to retry for 3 times, thought that 's normal, I will deliver this issue to our developer , see if it's really a bug or normal..