Latest Inquiries - Data Extraction Software

Crawling issues when using Proxy Switch

Submitted: 4/28/2016
Hi,

We had "Navigation error. The web server returned a status code indicating failure" error when we turned on the proxy switch.
It will not occur if we choose no proxies.Please assist.

Thanks,
123 (2)_info_16_04_28.log
123 (2).rip

Replied: 4/29/2016 4:46:57 AM

Seems that you configured link transformation in Start URLs section, but the scripts doesn't work for me :

***Error compiling script***
***Error running script***
A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)


With the proxy switch issue, can you please try 'Ignore page load error' = true in Project > Project options > Advanced tab > Action section? sometimes, page can be loaded properly but target site only returns error code.

Replied: 4/29/2016 8:14:18 AM
The start URL is : https://www.linkedin.com/jobs/search?keywords=&locationId=sg%3A0&f_TP=1&trk=jobs_jserp_posted_one_day

We are still getting this error:




Replied: 5/2/2016 1:54:42 AM

I'm guessing that Linkedin has blocked your i/p since you did scraping on lots of requests, and your session could be expired. in case you may try proxies in your project , see if it could resolve this issue. we've provided a free private proxy switch for you:

http://manual.visualwebripper.com/default.aspx?manual_id=2033