Fixing CERTIFICATE_VERIFY_FAILED error when trying requests-html out on Mac12 Mar · by Tim Kamanin · 2 min read
If you decide to try a new web scraping package requests-html on Mac, you may be stopped by a
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
"What's wrong?" - You might think. Well, you may close this post and figure it out by yourself or read on and probably I'll save you some time.
2) To control Chrome via Python
requests-html downloads Chromium for you behind the scenes.
And in 99% of cases, you get the error because Pyppeteer can't download Chromium due to lacking SSL certificates in your Python installation.
You see, Python 3.6 on MacOS comes with its own private copy of OpenSSL. That means the trust certificates in the system are no longer used as defaults by the Python
ssl module. To fix that, you need to install a
certifi package in your system.
You may try to do it in two ways:
1) Via PIP:
pip install --upgrade certifi
2) If it doesn't work, try to run a
Cerificates.command that comes bundled with Python 3.6 for Mac:
open /Applications/Python\ 3.6/Install\ Certificates.command
One way or another, you should now have certificates installed, and Python should be able to connect via HTTPS without any issues.
Let me know if this helped you.