Shopify Product Automation

A Python product importer that populated an entire Shopify store with 263 products across 18+ brands, using three different image-scraping strategies and intelligent variant grouping.

The Problem

Little Luxuries is a personal ecommerce project I was building. I had a curated list of 263 products from 18+ brands that needed to get into Shopify with proper images, descriptions, variant structures, and pricing. Doing this manually through the Shopify admin would have meant creating each product by hand: uploading images, setting up variants for sizes and colors, writing descriptions, tagging by brand and category. At 10-15 minutes per product, that’s 40+ hours of data entry. The product data existed on supplier websites, but in different formats, different page structures, and different image delivery methods.

The Approach

I built a Python importer that reads product data from a structured spreadsheet (managed in Pandas), scrapes images from supplier websites, groups variants intelligently, and pushes everything to Shopify through the Admin API.

The image scraping was the most involved part because every supplier site serves images differently. I ended up building three separate scraping methods:

1. JSON API extraction for sites that load product images through JavaScript, where the image URLs live in structured JSON embedded in the page source.
2. HTML parsing with BeautifulSoup for traditional sites where images sit in predictable DOM structures like product gallery containers.
3. CDN URL construction for sites that follow a consistent CDN naming pattern, where you can derive image URLs from the product ID without scraping at all.

The variant grouping logic handles the fact that suppliers often list each size or color as a separate product, while Shopify expects them as variants under a single product. The importer matches products by name pattern, groups them, and creates the parent product with its variants in a single API call.

Batch processing handles Shopify’s API rate limits with built-in delays and retry logic. The whole import runs as a single script with logging, so I can see exactly which products succeeded, which failed, and why.

Key Decisions

• Three scraping methods over one generic scraper. I tried building a universal scraper first. It was brittle and failed on half the sites. Accepting that different sites need different approaches and writing three focused scrapers was faster and more reliable than trying to make one smart enough to handle everything.

• Pandas for data management over raw CSV parsing. The product spreadsheet had inconsistencies: missing sizes, duplicate names, price formats that varied by brand. Pandas made it straightforward to clean, normalize, and validate the data before it hit the Shopify API. Catching a pricing error in a DataFrame is a lot cheaper than catching it after 263 products are live.

• Single-script execution over a web interface. This was a one-time import with occasional re-runs for new products. Building a UI around it would have been over-engineering. A Python script with good logging and dry-run mode was the right tool for the job.

What I Learned

Web scraping is 20% writing the scraper and 80% handling the exceptions. A site changes its DOM structure, another one lazy-loads images so they’re not in the initial HTML, a third serves different markup to headless requests. Every supplier site taught me a new edge case. The three-method approach wasn’t planned. It was the result of the first method failing on the second site.

The other lesson was about data quality. My spreadsheet had 263 rows and I assumed it was clean. It wasn’t. Duplicate product names with slightly different spacing, prices in different currency formats, sizes listed as “500ml” in one row and “500 ml” in another. The Pandas validation step caught all of this before it became 263 problems in Shopify. Cleaning your data before automating anything is the most boring and most important step.

---

Built for Little Luxuries, a personal ecommerce project. Fully shareable.