Hi there! Here's a brief step-by-step guide to help you find the dept. URL that can be used to scrape product data for the entire category using the Amazon Category API. Let's get started!

Navigate to the department of your interest

For our example we're navigating to the Mens' department with the the Clothing, Shoes, Jewelry & Watches line

Find the sub-department, when necessary

Now in this case, when we navigate to the Men's department we can see a big banner and all these options for further subcategories and so-on.

It's very important to note, that for a `url` to be valid for use with the Amazon Category API, it must have pagination at the bottom.

To get such a url, we have 3 options:

Scroll to the bottom of the department and find the link "See all results"

If such a link is present, clicking it will take you to a page that has pagination at the bottom and you can use it with our Amazon Category API to retrieve all products from the department.

So, as you can see in the image below, for our example of the Mens' department, no such link exists — so we have no choice but to use another option.

However, for the sake of illustrating this option, let's take the Pet Supplies department page.

On this page, on scrolling to the bottom, we can see a "See all results" link.

When we click on that link, Amazon will take us to the real department page that can be used to apply filters, go to the next page, etc.

That's the page you want to copy the URL for, to use for scraping all the products with the Amazon Category API.

You can apply any filters from the column on the left before copying the URL, if you wish to scrape only those filtered products.

Click on a sub-department

Another simple choice is to navigate further to a sub-department, this will take you to a page with products listed on it with pagination at the bottom.

This is a simple choice, but limiting in terms of the number of pages you can scrape from it as sub-departments usually have much fewer products than the main department.

Still, if you need data for a sub department or if you're absolutely unable to find the category URL this is a valid choice, albeit suboptimal.

Finding the paginated URL for main department page using filtered sections

Now while the sub-department URLs work, having to find the URL for each sub-department and then scrape all pages for each URL using the Amazon Category API is less than ideal.

We still do have an option to try and find the paginated product results URL for the main Mens' department

Click on Clothing

Look for filters such as this one "Under $25", as showcased in the image below

Now remove the filter from the subsequent page

And, voila! You have the paginated product results page for the department Mens' Clothing.

As you can see, this page has more results.