ProxyCrawl Cloud Storage
Easily store and search your crawled or scraped data.
Store. Search. Scale
Are you using Amazon AWS S3 or SQS to temporarily or permanently store your crawled pages?
Do you maintain your own database? Searching your crawled data is becoming a problem the more you scale up your web scraping requests?
ProxyCrawl Storage is a cloud scalable storage solution where you can permanently or temporarily store your HTML pages, screenshots and scraped data.
Storage handles scaling, backing up and cleaning of your cloud space so you can focus on what really matters for your business.
How it works
You can send data to your ProxyCrawl Cloud Storage by either using the "&store=true" API parameter or configuring your Crawler to do so by using the Storage webhook endpoint, or by using the Crawling API with the "&async=true" API parameter.
Smart storage
Storage is designed to solve the problem that you have data to store and you have no reliable or cost efficient solution to scale it.