Latest Inquiries - Data Extraction Software

Program stops

Submitted: 4/4/2017

Hello!


I am having problems running the project attached because it runs fine for 3 pages and then it stops. How can I fix this?

This is the website I want to scrape: https://www.timeshighereducation.com/world-university-rankings/2017/world-ranking#!/page/0/length/25/sort_by/rank/sort_order/asc/cols/stats

and this is the starting URL: https://www.timeshighereducation.com/world-university-rankings/university-of-oxford#ranking-dataset/589595


Thank you in advance

Federica



The World_2017.rip

Replied: 4/11/2017 3:23:47 PM

Hi Federica,

Your agent works very well on our side, probably this is just a connection issue as pointed out by Soeren. Have you also run other agents? Because the way to test if this is a connection issue is to have you create a simple agent and run them in your system. Let me know if you did this.

Best regards,


Replied: 5/8/2017 8:56:52 AM

Hello,

I managed to solve the issue with the program connecting to internet, but I still cannot run the agent since it does not scrape more than one page.


Also the layout of the page is different from the website when it loads on the program.


Here attached the agent and the log


Kind Regards,


Federica



The World_2017 (3).rip
The World_2017 (3)_info_17_05_08.log

Replied: 5/11/2017 10:52:23 AM

Hello,


That's strange because I can see the normal layout if I open it with Chrome

Do you have any suggestion on how I can finally scrape this website?


Best Regards,

Federica

Captura.PNG

Replied: 4/13/2017 2:19:09 PM

Hi Federica,

You can do my suggestions below:

1. Try to recreate your agent all over again and see if the new one will work.

 2. Uninstall and install your software. This will not affect your deactivation number.

3. Try to install the VWR to another computer/machine and try to run your agent. Deactivation number will apply.

4. If all of this did not work, we need to investigate further win your computer via TeamViewer.


Best regards,


Replied: 4/12/2017 10:47:10 AM

Hello,

Yes I have run other agents and this is the only one not working. I checked the connection and disabled the antivirus, but still it does not work


Best Regards,


Federica

Replied: 6/1/2017 10:45:18 AM

Hello,

I still cannot run this agent successfully. The layout of the website was slightly changed and I am trying to modify the agent accordingly (I cannot use Next page link anymore but I need list of links)

The problem remains that when I load the URL into VWR the layout is different and therefore I cannot select any element to capture

I tried, as suggested by you, to load and navigate several and several times, but nothing changed.

Best Regards,

Federica

Replied: 4/7/2017 9:21:22 AM

Hi Federica,

The error message indicates that Visual Web Ripper doesn't have access to the Internet. This is usually caused by anti-virus software blocking Internet access for Visual Web Ripper. You will not to configure your anti-virus software to allow access.

Your project works fine for me, but the website is responsive so it's possible you need to use a sized browser window. Try the attached project.


The World_2017.rip

Replied: 4/10/2017 1:55:43 PM

Hello,

Actually there is no antivirus blocking it, but still I cannot make it work. What are other possible causes of this problem?


Thank you

Federica

Replied: 4/10/2017 3:48:03 PM

Hello,

The program does not scrape anything, therefore also with the Web Browser selected I cannot see anything, because the error is in connecting to the starting URL. You can find the log file attached


Best Regards,

Federica

The World_2017 (2)_info_17_04_10.log

Replied: 4/6/2017 9:01:56 AM

Hello,


here attached the log (17:04_04), but right now I cannot even start to run the project because when I open VWR the following error appears "One or more errors occured while trying to open the start template.". I tried also with another project and it shows the same message (find eclosed also the log for this 17_04_06)


Thank you in advance


Federica

The World_2017_info_17_04_04.log
The World_2017_info_17_04_06.log

Replied: 4/5/2017 8:55:02 AM

Hi,

Can you attached the log file?


Replied: 5/11/2017 10:10:17 AM

Hi Federica,

There is some problem on the website itself. When I try to visit the website in Chrome, it gives me the error you are describing. See attached.

Best regards,

Capture.JPG

Replied: 4/10/2017 2:11:15 PM

Hi Federica,

Try to tick the View Browser while you test run the agent to see what's going on during failure.

Best regards

Replied: 5/12/2017 1:34:44 PM

Hi Federica,

Yeah, it's really strange. When I navigate to the site several times in Chrome, that error did not show up again. So I did similar thing in VWR, I activate the Navigate in Browser button and navigate several times until the right format of the page is loaded. After that, it is now run successfully, see screenshot.

I suspect this error has something to do with cookies and cache of the website. 

Best regards,


Capture.PNG

Replied: 6/7/2017 4:51:02 AM

Hi Federica,

There is nothing wrong on my side when I run the agent, see attached. Try to used the attached agent file.

Best regards,


The World_2017 _3_.rip
The World_2017 _3_.xls
Capture.PNG