How to scrape password-protected websites that require a login

Significant amounts of useful information can be accessed through web scraping but you might encounter situations where a page is accessible behind a password-protected login.

This can be problematic for some web scraping platforms or custom scraping using Python.

In this tutorial, we will show you how easy it is to scrape data from password-protected websites using Hexomatic, the no-code cloud-based web scraping platform.

To get started, register for a free Hexomatic account.

Then there are two methods to deal with password-protected pages:

Method 1 will use simple click/type actions.

Method 2 for cases when it’s not possible to select these actions in the scraping recipe builder will use advanced click/type custom actions to specify elements using Xpath or CSS selectors.

Not a Hexomatic user yet? Sign up for a free account here to get started.

Method 1: How to use Simple Click and Type actions

In this section, we will show you the easiest way to access a password-protected website right in the Hexomatic scraping recipe builder.

Step 1: Create a new scraping recipe

Create a new scraping recipe. 

Step 2: Add the web page URL

Add the web page URL of the password-protected website. 

Step 3: Use the click and type actions to log in 

Now, let’s see how you can easily log into your account by using click and type actions right in the Hexomatic scraping recipe. 

Step 3.1 Use the Click action

Point to the Login button in the scraping recipe and perform the Click action. 

After performing this action, a pop-up window with the login and password fields will appear. 

Step 3.2 Use the type action to enter login and password

Perform the type action on the Login and Password fields of the pop-up window to input your Login and Password. 

Type action for the Login field

Type action for the Password field

Step 3.3 Use the Click Action to Log in

Now, point the mouse to the Login button and perform the Click action. 

Method 2: Use Advanced Click and Type actions

In cases when it is not possible to automatically select elements directly, you can use the advanced Click and Type custom actions to specify elements using Xpath or CSS selectors.

Like in the first, method, you need to create a new scraping recipe from your dashboard. 

After, follow the steps below. 

Step 1: Go to the website you want to scrape

If you are unable to perform the click action right in the scraping recipe builder, then access the page using your Chrome or other web browser and use developer tools to get the XPath/selector of the desired element. In the gif below, we are getting the selector of the Login button using Chrome.

Step 2: Paste the login button selector

Get back to the scraping recipe builder, select Click, and paste the copied Xpath/selector into the window. Then, click Proceed.

Step 3: Use copy and type custom actions to log into the account

After proceeding, a window will appear to fill in the username and password.

Add Actions, choosing Type. Then, add your username and paste the username selector copied from the targeted website. 

Then, repeat the actions for inserting the password.

Step 4: Use the Click action to log into the account

After pasting the Xpath/selector, use the Click action again to log into the account right from the scraping recipe builder.

To do that, copy the Xpath/selector of the login button and paste it into the window.

After proceeding, you will successfully log into your account.

Now, you can select the desired elements to scrape. 

---

Automate & scale time-consuming tasks like never before