Latest Inquiries - Data Extraction Software

Getting 500 and 503 errors, start URL not loading

Submitted: 7/21/2018
Hello,

first of all i have to say im very satisfied with your great product, it helped me a lot with my business. But lately i am getting some problems that are starting to affect my incomes, so i would really appreciate your help.

I am using VWR mainly for extracting buylist data from Crystal Commerce stores, that are notoriously slow and many times return a  500 and 503 errors. Some of these websites:
http://www.capefeargames.com/buylist/magic_singles/8
http://www.icarusmagic.com/buylist/magic_the_gathering-magic_singles/8
http://www.mtgdeals.com/buylist/magic_singles/8
etc.

Attached is my VWR project for Cape Fear Games (CFG). To counter the errors, i am mainly using this setting:
Project Preferences --> Advanced --> Project:
- Retry errors on resume = True
- Retry count on errors = 3

But the problem is that when the project resumes, it often cannot even load the start URL, which then results in an empty export CSV file.

What would be the solution for this ? 

If the start URL is the problem, i am thinking of maybe starting the project with some random URL and then have a link template that would go to the CFG website. Even if the CFG website would not work, the VWR should not delete the previous extracted data. Would that solve it ?


One other problem:
Sometimes when i resume a project, it doesnt load only the error pages, but it goes again from the start. How come ?


Thank you very much





CFG.rip
CFG_log.txt

Replied: 7/23/2018 7:26:00 AM

I ran the project for a while, it doesn't seem to raise page 500 / 503 errors, please check the attached logs.

there is only issue with xpath in page area template, that it couldn't match the result list after 16th page.

CFG_info_18_07_23.log