Latest Inquiries - Data Extraction Software

new website problem

Submitted: 5/24/2017

There is a website that I've been scraping now problem free for about 2 years where I could easily scrape off the price and item UPC (http://festfoods.gsngrocers.com/Shop/Product.aspx?Department=10369)

HOWEVER, they are currently rolling out a new website, (e.g. https://shop.festfoods.com/shop/categories/1119?tags=on_sale) and here the item information is masked:

<a title="Twizzlers Candy, Pull 'n' Peel, Cherry, 14 oz" class="cell-image show" style='background-image: url("https://images01.unata.com/v1/AUTH_55ddb9eda95d4a3394bb819b215c54e3/fes_prod_unata/web/product_medium/8ba06cd399d4dc4fdef87e9f0deb0a23e19e328d.jpg");' ng-class="{'cell-sponsored': product.sponsored &amp;&amp; config.showSponsoredLabel}" analytics-click="['product cell image link', 'click']" ng-click="linkToProductDetails(product.id)" promoted-media="promotedMedia" default-image="product.images.tile.medium" cell-image="" analytics-product-click="[product, location]"><!-- ngIf: product.sponsored && config.showSponsoredLabel --><!-- ngIf: hasOffer && (!hasOfferPoints || !config.hasLoyaltyPoints) --><!-- ngIf: hasOfferPoints && config.hasLoyaltyPoints --><!-- ngIf: product.tags.length --></a>


Do you have any suggestions, or am I screwed?


Thanks.


Replied: 6/7/2017 3:13:57 PM

Brian-

Tried updating from within the application, and got the attached error.

Tried going to the website and downloading the latest version, ran the setup exe, but still showing 3.0.18 in the application.

Any hints?

VWR Update Fail.png

Replied: 6/14/2017 1:44:51 PM

Yes, I think that will work. Please use this as my reference to my time, http://www.worldtimeserver.com/current_time_in_AU-NSW.aspx 

If you are ready, you can send me an email at brian.javier@sequentum.com

Best regards,

Replied: 6/7/2017 1:54:58 PM
Do you have any ideas to make it work on my end, or what could be causing the issue?
Replied: 5/25/2017 4:31:52 AM

Hi

You would need to take a screenshot of what on the website you are trying to scrape or attach the agent. 

Please note if you would like us to find a solution we would have to charge a consulting fee. 

Best Wishes

Kyle Correia

Replied: 6/12/2017 4:28:44 AM

Hi.

We may need a TeamViewer session so I can look at it since we can't replicate your problem on our side. I am available 11AM to 7PM AEST (NSW, Austraila Time). Please let me know your available time.

Best regards,

Replied: 6/8/2017 9:56:45 AM

Hi,

That really strange. Please try attached agent. We need to find out where the point of the problem.

Festfoods3.rip

Replied: 6/7/2017 1:41:56 PM

It doesn't go through all the values for the aisles, etc.

See screenshot and log.

Festfoods3_info_17_06_07.log
Festival1.png

Replied: 5/25/2017 8:38:59 AM

Hi,

You have to actually click to for the product window to appear then from that window, you can get the link for product url. You also need to add a link template to close the window.

Best regards,

Replied: 6/7/2017 3:22:52 PM
And I was on the latest version of IE
Replied: 6/13/2017 3:36:42 PM

Hi Brian-

How about 8PM Wednesday the 14th my time (CST) which I THINK is 11AM on the 15th for you.

Replied: 6/8/2017 1:59:32 PM

That didn't work either--

It seems to be failing in the "submit" template-- the dialog box for selecting the aisles, etc. is not opening up.  Is there a way to force that?


Thanks

Replied: 6/7/2017 5:01:06 PM

Ok, updated the version of VWR and set IE11.


Still missing the aisles, etc.


Attached log

Festfoods3_info_17_06_07.log
Festival Screen Shots.docx

Replied: 6/7/2017 2:36:44 AM

Hi,

I'm not sure if I understand your problem. I open your agent and I see the prices after navigating on the categories, see attached screenshot.

Best regards,

Capture.PNG

Replied: 6/14/2017 8:23:39 PM

Greetings Brian-

Actually we can cancel the call.  Your being able to do the scrape (and the fact that I could look up prices on my cell phone) got me to thinking that they are probably aware that I'm scraping prices from this IP address.  I set up the project using a proxy and was able to get in and do the scrape no problem.  I would think that as long as I set up the proxy to rotate frequently, that I should be able to continue without being blocked. 

Thanks for your help in trying, and have a good week.

Regards,

Karl

Replied: 6/7/2017 2:00:06 PM

Hi,

1. Make sure you are using the latest version of VWR.

2. Make sure that the web browser emulation is set to IE11. Go to Tools > Web Browser Emulation and set it to IE11.

Best regards,


Replied: 6/15/2017 2:30:37 AM
Not a problem Karl. Glad to know it is now working.
Replied: 6/7/2017 1:50:23 PM

Hi,

Your agent works well on my side. I did not change anything but it works like a charm. It is navigating to the Department, Aisle, and Category. 

Captures.PNG
Festfoods3_20170607_204926.csv

Replied: 6/6/2017 2:15:37 PM

Hey Brian-

I went back to my original scraping project (attached), and looked at where it was being hung up.  On a whim, I tried going to the website on my phone to see if perhaps the mobile environment is different, and I WAS able to get to item pricing information (see attached screen shot).

Do you have any insight as to why it would work on my phone and not on a computer (I tried in Chrome, Firefox and Windows IE)?


Thanks.

Festfoods3.rip
Screenshot_2017-06-06-08-12-48.png