Mar 12, 2018 · Updated: Jan 30, 2021 · by Tim Kamanin
If you decide to try a new web scraping package requests-html on Mac, you may be stopped by the following error:
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
"What's wrong?" - You might think. Well, you may close this post and figure it out by yourself or read on and probably I'll save you some time.
2) To control Chrome via Python
requests-html downloads Chromium for you behind the scenes.
And in 99% of cases, you get the error because Pyppeteer can't download Chromium due to lacking SSL certificates in your Python installation.
You see, Python 3.6 on MacOS comes with its own private copy of OpenSSL. That means the trust certificates in the system are no longer used as defaults by the Python
ssl module. To fix that, you need to install a
certifi package in your system.
You may try to do it in two ways:
1) Via PIP:
pip install --upgrade certifi
2) If it doesn't work, try to run a
Cerificates.command that comes bundled with Python 3.6 for Mac:
open /Applications/Python\ 3.6/Install\ Certificates.command
One way or another, you should now have certificates installed, and Python should be able to connect via HTTPS without any issues.
Let me know if this helped you.
Hey, if you've found this useful, please share the post to help other folks find it: