Latest Inquiries - Data Extraction Software

Issue in Pagination

Submitted: 1/18/2016

Hi,

https://ca.linkedin.com/jobs/search?locationId=STATES.ca.bc&start=0&count=25&trk=jobs_jserp_pagination_1

For the above URL,only 10 pages are getting selected and the 11th page is not getting selected.

After clicking 10th page only we can able to see 11th page which is not getting selected in page navigation element.

We want to scrape each page by page one after another untill last page. Kindly advise us.

Please find the below attachments for your reference.

test.rip
ScreenShot_2.jpg
ScreenShot_1.jpg

Replied: 1/19/2016 12:23:08 PM

Please you attach your log file for diagnose further, my private linkedin account couldn't run the project for a long time, it's high risky to me.

I'm guessing that Linkedin site has blocked your i/p address, therefore , you couldn't get any data at 50th page, or your logged session has been expired after you scraped 50 pages, it won't be able to return correct data, you can mark check 'view browser' on debugging window, see exactly what happened at that time.

Propose that you divide scraping from 50th page, since it's a direct link, you might try to create a redirect template firstly navigating to the 50th page url, then starting to collect new data again. that's more easily rather than debugging.

Replied: 1/19/2016 6:10:51 AM

Hi,

As you said to set the pagenavigation element to the 'Dynamic list of links' the page is navigated but it is stopped at 50th page. I need to scrap the details from all pages. The page is not navigated to the last page. Kindly advise us.

test1.rip
test1_screenshot.jpg

Replied: 1/19/2016 1:33:26 AM
In case you should mark check 'Dynamic list of links' instead of 'List of links' , see the attached new project, it works fine iterating through more than 10 pages.
test.rip