Fixing CERTIFICATE_VERIFY_FAILED error when trying requests-html out on Mac

If you decide to try a new web scraping package requests-html on Mac, you may be stopped by the following error:

urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

"What's wrong?" - You might think. Well, you may close this post and figure it out by yourself or read on and probably I'll save you some time.

1) requests-html is so cool that it can scrape javascript pages, but to do so, it uses Chromium which is basically Google Chrome.

2) To control Chrome via Python requests_html uses Pyppeteer, an unofficial Python port of puppeteer JavaScript Chome automation library.

3) And now the final step: when you scrape a javascript based web page for the first time, requests-html downloads Chromium for you behind the scenes.

And in 99% of cases, you get the error because Pyppeteer can't download Chromium due to lacking SSL certificates in your Python installation.

You see, Python 3.6 on MacOS comes with its own private copy of OpenSSL. That means the trust certificates in the system are no longer used as defaults by the Python ssl module. To fix that, you need to install a certifi package in your system.

You may try to do it in two ways:

1) Via PIP:

pip install --upgrade certifi

2) If it doesn't work, try to run a Cerificates.command that comes bundled with Python 3.6 for Mac:

open /Applications/Python\ 3.6/Install\ Certificates.command

One way or another, you should now have certificates installed, and Python should be able to connect via HTTPS without any issues.

Let me know if this helped you.