Significant amounts of useful information can be accessed through web scraping but you might encounter situations where a page is accessible behind a password-protected login.
This can be problematic for some web scraping platforms or custom scraping using Python.
In this tutorial, we will show you how easy it is to scrape data from password-protected websites using Hexomatic, the no-code cloud-based web scraping platform.
To get started, register for a free Hexomatic account.
Then there are two methods to deal with password-protected pages:
Method 1 will use simple click/type actions.
Method 2 for cases when it’s not possible to select these actions in the scraping recipe builder will use advanced click/type custom actions to specify elements using Xpath or CSS selectors.
Not a Hexomatic user yet? Sign up for a free account here to get started.
Method 1: How to use Simple Click and Type actions
In this section, we will show you the easiest way to access a password-protected website right in the Hexomatic scraping recipe builder.
Step 1: Create a new scraping recipe
Create a new scraping recipe.
Step 2: Add the web page URL
Add the web page URL of the password-protected website.
Step 3: Use the click and type actions to log in
Now, let’s see how you can easily log into your account by using click and type actions right in the Hexomatic scraping recipe.
Step 3.1 Use the Click action
Point to the Login button in the scraping recipe and perform the Click action.
After performing this action, a pop-up window with the login and password fields will appear.
Step 3.2 Use the type action to enter login and password
Perform the type action on the Login and Password fields of the pop-up window to input your Login and Password.
Type action for the Login field
Type action for the Password field
Step 3.3 Use the Click Action to Log in
Now, point the mouse to the Login button and perform the Click action.
Method 2: Use Advanced Click and Type actions
In cases when it is not possible to automatically select elements directly, you can use the advanced Click and Type custom actions to specify elements using Xpath or CSS selectors.
Like in the first, method, you need to create a new scraping recipe from your dashboard.
After, follow the steps below.
Step 1: Go to the website you want to scrape
If you are unable to perform the click action right in the scraping recipe builder, then access the page using your Chrome or other web browser and use developer tools to get the XPath/selector of the desired element. In the gif below, we are getting the selector of the Login button using Chrome.
Step 2: Paste the login button selector
Get back to the scraping recipe builder, select Click, and paste the copied Xpath/selector into the window. Then, click Proceed.
Step 3: Use copy and type custom actions to log into the account
After proceeding, a window will appear to fill in the username and password.
Add Actions, choosing Type. Then, add your username and paste the username selector copied from the targeted website.
Then, repeat the actions for inserting the password.
Step 4: Use the Click action to log into the account
After pasting the Xpath/selector, use the Click action again to log into the account right from the scraping recipe builder.
To do that, copy the Xpath/selector of the login button and paste it into the window.
After proceeding, you will successfully log into your account.
Now, you can select the desired elements to scrape.
Automate & scale time-consuming tasks like never before
Hexomatic. The no-code, point and click work automation platform.
Harness the internet as your own data source, build your own scraping bots and leverage ready made automations to delegate time consuming tasks and scale your business.
No coding or PhD in programming required.