Latest Inquiries - Data Extraction Software

Next Page navigation error in asp page

Submitted: 1/26/2017
Hi Team,

I am facing an issue in navigation wherein after few pages it does not navigate further.. I see the page remain in same page but it shows page navigate to N pages. Attached test project and it works fine for 4/5/10 pages max 

Thanks in advance
Agriturismo - Copy.rip

Replied: 1/31/2017 7:40:09 AM
Hi Support Team,

Thank you  for the response.

Can you please help me how to build a web crawler project adn multi threading. A sample project would help me to proceed further.

As per your suggestion, I see that the Id's are not constat, 1-5000. few may have and few don't
Replied: 1/31/2017 8:31:07 AM

Hi,

Attached is the agent. for you to start.

Not all of them have data, yes. That's the way the site works. You can't tell what url has content that's why you are putting them all. Increase the number if you need to. Then sort it later to excel.

Best regards,


Agriturismo.rip

Replied: 2/1/2017 10:47:09 AM

Hi,

I already sent it yesterday. You can actually run it directly actually. I also attached the list of URL you can run.

Best regards,


Agriturismo.rip
inputURLs.csv

Replied: 1/30/2017 12:33:44 AM

Hi,

The website actually put limit on the paginaiton you can visit. I can't even go beyond 20th page. It is also gives you a maximum page you can visit, 5,8,10,20 and they control it using cookies.

The only way I can see to scrape data here is to generate the url of the page. 

http://en.agriturismo.com/dettaglioAgriturismo.asp?idLingua=1&id=738

and then you can generate the id= from 1-5000 or more and use it as an input url. In this method, you can also use web crawler and use multiple threads to make your scrape much faster.


Best regards,


Replied: 2/1/2017 7:23:46 AM
Hi Support Team,

Thank you  for the response.

Can you please help me how to build a web crawler project adn multi threading. A sample project would help me to proceed further.

As per your suggestion, I see that the Id's are not constat, 1-5000. few may have and few don't