How to Scrape Costco Product Details (2025 Guide)

Costco, the third-largest retailer in the world, operates on a unique membership-based model that prioritizes efficiency and bulk pricing. This approach not only keeps costs low for customers but also creates distinct patterns in product availability, pricing strategies, and inventory management.

For data scrapers, Costco's model presents a valuable opportunity to extract insights on pricing trends, limited-time deals, and product rotations that differ from traditional retailers.

However, scraping product details from major retailers has always been challenging. You are likely to encounter bot detection mechanisms like CAPTCHA, API rate limits, and possibly JavaScript rendering.

At first glance, Costco.com seems relatively easier to scrape compared to sites like Walmart and Target. However, Costco uses Akamai's bot protection, making traditional scraping techniques ineffective.

Let's go through some of the ways you can scrape Costco.

Understanding Costco's Website Structure and How It Affects Scraping

Like every other modern e-commerce website, Costco also relies on single-page applications (SPAs), which dynamically load content using XHR/Ajax requests instead of delivering fully rendered HTML from the server. This means that product details are usually fetched from hidden API endpoints in structured JSON format.

or web scraping, the key is to identify the API responsible for delivering product data to the website. By capturing these requests, you can extract product details directly from Costco's backend.

Let's say you want to scrape electronics items. We'll take LG TVs as an example. When you search for LG TV on Costco's website, you can find an API endpoint that provides detailed product information.

Step 1: Search for a Product on Costco.com

Start by visiting Costco.com and searching for LG TV in the search bar.

Step 2: Locate the API Request Using Developer Tools

1. Open Developer Tools (Press F12 or Right Click → Inspect).
2. Click on the Network tab.
3. Filter by XHR/Fetch requests.
4. Look for a request related to product search—it will typically contain the search keyword in its URL with whloc=1-wh, such as: www_costco_com_search?expoption
5. Copy the full request URL, as it includes key query parameters.

Step 3: Example API Request URL

When searching for LG TV, you may find an API request like this:

https://search.costco.com/api/apps/www_costco_com/query/www_costco_com_search?expoption=def&q=LG%20TV&locale=en-US&start=0&expand=false&userLocation=WA&loc=115-bd,1-wh,1250-3pl,1321-wm,1456-3pl,283-wm,561-wm,725-wm,731-wm,758-wm,759-wm,847_0-cor,847_0-cwt,847_0-edi,847_0-ehs,847_0-membership,847_0-mpt,847_0-spc,847_0-wm,847_1-cwt,847_1-edi,847_d-fis,847_lg_n1f-edi,847_NA-cor,847_NA-pharmacy,847_NA-wm,847_ss_u362-edi,847_wp_r458-edi,951-wm,952-wm,9847-wcs&whloc=1-wh&fq={!tag=item_program_eligibility}item_program_eligibility:("ShipIt")&chdcategory=true&chdheader=true

Step 4: Breaking Down the API Parameters

Parameter	Description
q=LG%20TV	Search query (LG TV)
locale=en-US	Language and region settings
start=0	Pagination (starting position)
expand=false	Whether to expand search results
userLocation=WA	User's detected location
whloc=1-wh	Warehouse location ID
fq={!tag=item_program_eligibility}item_program_eligibility:("ShipIt")	Filters products eligible for shipping

By adjusting these parameters, you can refine your search queries and extract specific product details.

However, since Costco uses Akamai bot protection, standard scraping techniques may not work.

Attempting to Fetch Data from Costco's API With Python

Here's a Python script that attempts to fetch data from Costco's API endpoint.

How This Works

• Sets the API URL for Costco's search endpoint.

• Defines query parameters (e.g., searching for "LG TV").

• Adds headers to mimic a real browser and avoid basic bot detection.

• Sends a GET request to fetch product data.

• Parses and prints the JSON response if successful.

Challenges & Next Steps

If you attempt to scrape, you will likely encounter the following error:

Failed to fetch data. Status code: 401  
{"service":"proxy","error":"unauthorized","msg":"Not Authenticated","path":"/api/apps/www_costco_com/query/www_costco_com_search","messages":["Not Authenticated"],"timestamp":"2025-02-16T04:00:12.202276","message":"Not Authenticated"}

This happens because Costco's domain is protected by Akamai which blocks our access to their API. Even though we include headers to mimic a real browser request, traditional scraping techniques fail due to Akamai's advanced bot protection.

Why Selenium, Puppeteer, or Playwright Won't Work

Even if you attempt to use browser automation tools like Selenium, Puppeteer, or Playwright, the request will still result in a 401 Unauthorized error. This is because Costco's backend API is safeguarded by Akamai, which employs sophisticated techniques to identify and block requests from headless browsers.

Since direct API calls and browser automation are ineffective, an alternative approach is needed.

Step-by-Step Guide to Scraping Costco Using Unwrangle API

Why Use Unwrangle API?

Scraping Costco directly is challenging due to Akamai's bot protection. Traditional scraping techniques, such as using requests, Selenium, or Puppeteer, often fail with 401 Unauthorized errors.

To bypass these restrictions efficiently, Unwrangle API offers a seamless way to extract product data, search results, and reviews from Costco without worrying about bot detection, captchas, or IP bans.

Getting Started: Prerequisites

Before you begin, ensure you have the following:

• API Key: Sign up on Unwrangle to get your API key.

• Python Installed: Make sure Python 3.x is installed on your system.

• Requests Library: Install the requests library if you don't already have it by running:

How to Make an API Request

To scrape Costco data, make a GET request to the /api/getter endpoint with the following query parameters:

Parameter	Description
Platform	Specifies the API type (costco_search, costco_category, or costco_reviews).
URL/Search	Provide the product URL, search term, or other details based on the API type.
API Key	Your Unwrangle API key to authenticate the request.
Additional Parameters	Optional values such as store number, zip code, or page number, depending on the API.

Example: Scraping Costco Search Results for "Macbook"

Here's how you can scrape Costco's search results using the Unwrangle API:

Parameter	Required	Description

search	✅	Search query (e.g., "Macbook").

store_no	❌	Costco warehouse/store number (optional).

page	❌	Page number (default: 1).

platform	✅	Must be set to "costco_search".

api_key	✅	Your Unwrangle API key.

Results & Attributes

Each response returns up to 24 products with details like:

• Product Name, URL, Price, Rating
• Brand, Model Number, Availability
• UPC, Categories, Marketing Features

More Ways to Scrape Costco Data

If you are looking for other BestBuy data we have also APIs for:

• Costco Category Search API: Scrape Costco category listings with pagination and filters.

• Costco Product Reviews API: Extract customer reviews, ratings, and purchase insights.

With these, you can Instantly retrieve product details like prices, images, and descriptions. Easily fetch customer reviews, including ratings and review text. Get structured JSON responses that are easy to integrate into your code.