Fixing CERTIFICATE_VERIFY_FAILED error when trying requests-html out on Mac
Mar 12, 2018 · by Tim KamaninIf you decide to try a new web scraping package requests-html on Mac, you may be stopped by a
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
error.
"What's wrong?" - You might think. Well, you may close this post and figure it out by yourself or read on and probably I'll save you some time.
1) requests-html
is so cool that it can scrape javascript pages, but to do so, it uses Chromium which is basically Google Chrome.
2) To control Chrome via Python requests_html
uses Pyppeteer, an unofficial Python port of puppeteer JavaScript Chome automation library.
3) And now the final step: when you scrape a javascript based web page for the first time, requests-html
downloads Chromium for you behind the scenes.
And in 99% of cases, you get the error because Pyppeteer can't download Chromium due to lacking SSL certificates in your Python installation.
You see, Python 3.6 on MacOS comes with its own private copy of OpenSSL. That means the trust certificates in the system are no longer used as defaults by the Python ssl
module. To fix that, you need to install a certifi
package in your system.
You may try to do it in two ways:
1) Via PIP:
pip install --upgrade certifi
2) If it doesn't work, try to run a Cerificates.command
that comes bundled with Python 3.6 for Mac:
open /Applications/Python\ 3.6/Install\ Certificates.command
One way or another, you should now have certificates installed, and Python should be able to connect via HTTPS without any issues.
Let me know if this helped you.