(2022) Use PHP technologies to crawl and click buttons on websites with GUI. I highly recommend working with Linux (including virtual machines) or MacOs. Laravel 12.
Before you try to scrape any website, go through its robots.txt file. You can access it via domainname/robots.txt. There, you will see a list of pages allowed and disallowed for scraping. You should not violate any terms of service of any website you scrape.
Published Postman API Collection.
cp .env.example .env
# Don't worry when the following step errors related to chromedriver binary, we will install them right after.
composer installMake sure Chromedriver is installed and added to your environment Path.
# install chromedriver for Panther client.
vendor/bin/bdi detect drivers
sudo mv drivers/chromedriver /usr/local/bin/chromedriver
# Or
# chromedriver_mac64
# chromedriver_win32
# See https://chromedriver.storage.googleapis.com
# for drivers list.
wget https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
sudo mv chromedriver /usr/local/bin/chromedriver
chromedriver --versioncomposer install
php artisan key:generate
# Before running the next command:
# Update your database details in .env
php artisan migrate --seed
yarn install
yarn buildDownload Selenium Server jar file.
Run the following in a new terminal.
java -jar selenium-server-4.29.0.jar standalone --override-max-sessions true --max-sessions 10CLI options in the Selenium Grid.
Update the command at ./app/Console/Commands/BrowserScrape.php
php artisan browser:scrapePanther Environment Variables.
php artisan make:crawler TestCrawlerSee Python Selenium web scraper.
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Please make sure to update tests as appropriate.