Software Manual - Data Extraction Software

The topic below is from the Visual Web Ripper manual.

Submitting Web Forms

Visual Web Ripper can submit web forms repeatedly for any combination of input values. You can specify input values as a static list of values, or you can feed the values from an input data source, such as a database or a CSV file.

FormSubmit templates are used to submit any web forms, such as login forms or search forms.  A FormSubmit template normally has associated FormField content elements. These content elements select the form input elements and specify the form input values.

Follow these two steps to configure a data extraction project to submit a form:

  1. Add a FormSubmit template that selects the form submit button.
  2. Add content elements for each form field for which you want to supply input values.

The FormField content elements have corresponding output data fields containing the input values. If you do not want to write the input values to data output, you can reset the Save Content option.

A FormSubmit template normally selects an input HTML element of type Submit. Visual Web Ripper clicks on the input element to submit the form. Sometimes websites use a JavaScript to submit the form, and in this case the submit HTML element may be any HTML element. If the website does not use an input HTML element as the submit element, Visual Web Ripper may recognize the selected HTML element as a link and you will have to change the template type from Link to FormSubmit.

A FormSubmit template submits a web form for each combination of form field input values. While designing a project, you can set the form field input values in the Visual Web Ripper Capture Window. Each input value should be separated by a new line. For example, if you want to submit five input values for a form, the Capture Window for the FormField content element should look something like this:

When you add a FormField element that selects an HTML drop-down menu, Visual Web Ripper automatically submits the form for all values in the drop-down menu. If you select the FormField option Iterate all list values, you can see all drop-down menu values listed in the Capture Window. If you want to submit the form for only a few values in the drop-down menu, you need to reset the option Iterate all list values and then remove the superfluous entries from the Capture Window.

FormField Actions

Some websites have form fields that automatically submit a web form. For example, a search result page may have a drop down box that lets you set the number of search results displayed on the page, or a drop down box that lets you control the sort order of the result set. Once you select a value in the drop down box, the web form is automatically submitted without you having to click on a submit button.

Follow these steps to configure a project to set a value in a drop down box and then automatically submit the web form.

  1. Add a FormField content element for the drop down box.
  2. Reset the FormField option Iterate all list values, and enter the desired drop down value in the Capture Windows.
  3. Set an Action for the FormField element. This can be a Full page load action or a JavaScript action.
  4. Add a FormSubmit template and set the form type to Auto submit.

Downloading a File on Form Submit

Sometimes a form submit leads to a file download. In this case a "Save As" dialog box will appear when the form submit button is clicked. This dialog box will prevent Visual Web Ripper from processing the website properly.

The following advanced template options are used to configure a project to download the file instead of displaying the "Save As" dialog.

The Fixed file extension option must be set when downloading files on form submit.

The Download file on submit option is only support by the WebCrawler agent and the version 2 WebBrowser agent. You must set the project option Modify web browser behavior when using the version 2 WebBrowser agent.

Uploading Files

Some web forms contains file upload boxes that allow users to upload files to a website. File upload boxes cannot be manipulated by scripting, so they need to be processed differently than other web form input boxes.

Follow these steps to add a file upload FormField to your project.

  1. Select the file upload input box in the web browser and add a FormField content element to your project.
  2. Enter the path of the file you want to upload in the Capture window. You can also load file paths from an input data source.
  3. Set the FormField action to JavaScript and AJAX.
  4. Set the FormField option "File upload" to true in the advanced options tab.
A web form that uploads files cannot be submitted to a new browser window, so such web forms will always open in the current browser window.

Web Form Examples

Related Topics