Marketplace Archiver

Product listing archiver and image downloader

Cover by Max Suleimanov

This tool archives product listings on various asset / 3D marketplaces. It accepts a marketplace and user to be crawled, then crawls and downloads product listings: images, descriptions, and links. See the list below for supported marketplaces:

  • ArtStation: No official API, but the website exposes HTTP-JSON API calls to paginate and get products.
  • CGTrader: No official API, so the tool parses HTML responses to paginate and get products.
  • CubeBrush: No official API, so the tool parses HTML responses to paginate and get products.
  • Envato: Official API combined with string manipulation to paginate and get products.

Please note: this tool is not made to steal art, only archive product listings. Please also be responsible when using this tool, as modifying the degree of parallelism set by the program can cause a denial of service attack. For example, CubeBrush is a very poorly hosted website and crashes with 500 errors very often; therefore, the tool slows down for CubeBrush so multiple clients won’t crash it.

Compile and run the tool

You can find a link to the source code below. Please feel free to build and run the tool yourself, or extend it to support other marketplaces.

View Source

Command-line readme

ABSTRACT
  marketplace website username [(--destination | -d) path] 
    [(--token | -t) access_token] [(--verbose | -v)]

DESCRIPTION
  Crawls the specified marketplace website for names, descriptions, and images
  from product listings belonging to the specified user. The output folder 
  will be created in the current directory, named after the specified user, if
  the output path left unspecified.

SOURCE
  https://gitlab.com/spirited/marketplace

ARGUMENTS
  website    The website to crawl (artstation, cgtrader, cubebrush, envato)
  username   The username of the marketplace author to be crawled

OPTIONS
  -d, --destination string   Destination to save images to (current directory)
  -t, --token string         Access token for connecting to some marketplaces
  -v, --verbose bool         Enables verbose logging (false)

EXAMPLES
  marketplace artstation spirited
  marketplace artstation spirited --destination ./marketplace