[Python] https 로 시작하는 페이지 requests 요청 시 발생하는 SSLCertVerificationError 해결 방법

홈페이지 중에서 https 로 시작하는 페이지의 경우 http와는 다르게 SSL 인증이 필요한 페이지 입니다.

 

스파이더맨 노 웨이홈 예매 시작을 목빠지게 기다리고 있는데

영등위(영상물등급위원회) 에서 아직도 등급 심의 결과가 나오지 않아 예매 오픈이 되지 않고 있습니다. 🥲

 

CGV 영화 예매 오픈 텔레그램 알리미는 만들어서 집 PC에서 실행시켜 두었는데

추가로 영등위 심의 결과 확인 알리미도 만들어볼까 하여

온라인등급분류서비스-등급진행자료조회 - 영화(Spider-Man: No Way Home)

스파이더맨: 노 웨이 홈  (Spider-Man: No Way Home) 등급 분류 결과 링크를 크롤링을 먼저 해보려고 했습니다.

 

크롤링에는 Python과 requests 라이브러리를 활용하려 했는데

import requests

url = "https://ors.kmrb.or.kr/rating/inquiry_mv_view.do?RCV_NO=2288320&KIND=MV_FOR&GUBUN=N&&proc_state_code=onGoing&mv_kind=MV_FOR&vd_kind=VD_DOM&mv_use_title=%EC%8A%A4%ED%8C%8C%EC%9D%B4%EB%8D%94%EB%A7%A8"

url_req = requests.get(url)

url_req.text

 

https로 시작하는 이 페이지를 그냥 requests로 요청을 하려고 하니

아래와 같이 SSLCertVerificationError가 발생했습니다.

---------------------------------------------------------------------------
SSLCertVerificationError                  Traceback (most recent call last)
~/PythonHome/test_python/lib/python3.7/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    705                 headers=headers,
--> 706                 chunked=chunked,
    707             )

~/PythonHome/test_python/lib/python3.7/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    381         try:
--> 382             self._validate_conn(conn)
    383         except (SocketTimeout, BaseSSLError) as e:

~/PythonHome/test_python/lib/python3.7/site-packages/urllib3/connectionpool.py in _validate_conn(self, conn)
   1009         if not getattr(conn, "sock", None):  # AppEngine might not have  `.sock`
-> 1010             conn.connect()
   1011 

~/PythonHome/test_python/lib/python3.7/site-packages/urllib3/connection.py in connect(self)
    420             ssl_context=context,
--> 421             tls_in_tls=tls_in_tls,
    422         )

~/PythonHome/test_python/lib/python3.7/site-packages/urllib3/util/ssl_.py in ssl_wrap_socket(sock, keyfile, certfile, cert_reqs, ca_certs, server_hostname, ssl_version, ciphers, ssl_context, ca_cert_dir, key_password, ca_cert_data, tls_in_tls)
    428         ssl_sock = _ssl_wrap_socket_impl(
--> 429             sock, context, tls_in_tls, server_hostname=server_hostname
    430         )

~/PythonHome/test_python/lib/python3.7/site-packages/urllib3/util/ssl_.py in _ssl_wrap_socket_impl(sock, ssl_context, tls_in_tls, server_hostname)
    471     if server_hostname:
--> 472         return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
    473     else:

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py in wrap_socket(self, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, session)
    411             context=self,
--> 412             session=session
    413         )

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py in _create(cls, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, context, session)
    849                         raise ValueError("do_handshake_on_connect should not be specified for non-blocking sockets")
--> 850                     self.do_handshake()
    851             except (OSError, ValueError):

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py in do_handshake(self, block)
   1107                 self.settimeout(None)
-> 1108             self._sslobj.do_handshake()
   1109         finally:

SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1045)

During handling of the above exception, another exception occurred:

MaxRetryError                             Traceback (most recent call last)
~/PythonHome/test_python/lib/python3.7/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    448                     retries=self.max_retries,
--> 449                     timeout=timeout
    450                 )

~/PythonHome/test_python/lib/python3.7/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    755             retries = retries.increment(
--> 756                 method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
    757             )

~/PythonHome/test_python/lib/python3.7/site-packages/urllib3/util/retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
    572         if new_retry.is_exhausted():
--> 573             raise MaxRetryError(_pool, url, error or ResponseError(cause))
    574 

MaxRetryError: HTTPSConnectionPool(host='ors.kmrb.or.kr', port=443): Max retries exceeded with url: /rating/inquiry_mv_view.do?RCV_NO=2288320&KIND=MV_FOR&GUBUN=N&&proc_state_code=onGoing&mv_kind=MV_FOR&vd_kind=VD_DOM&mv_use_title=%EC%8A%A4%ED%8C%8C%EC%9D%B4%EB%8D%94%EB%A7%A8 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1045)')))

During handling of the above exception, another exception occurred:

SSLError                                  Traceback (most recent call last)
<ipython-input-3-58e71b7e5ac5> in <module>
      3 url = "https://ors.kmrb.or.kr/rating/inquiry_mv_view.do?RCV_NO=2288320&KIND=MV_FOR&GUBUN=N&&proc_state_code=onGoing&mv_kind=MV_FOR&vd_kind=VD_DOM&mv_use_title=%EC%8A%A4%ED%8C%8C%EC%9D%B4%EB%8D%94%EB%A7%A8"
      4 
----> 5 url_req = requests.get(url)
      6 
      7 url_req.text

~/PythonHome/test_python/lib/python3.7/site-packages/requests/api.py in get(url, params, **kwargs)
     74 
     75     kwargs.setdefault('allow_redirects', True)
---> 76     return request('get', url, params=params, **kwargs)
     77 
     78 

~/PythonHome/test_python/lib/python3.7/site-packages/requests/api.py in request(method, url, **kwargs)
     59     # cases, and look like a memory leak in others.
     60     with sessions.Session() as session:
---> 61         return session.request(method=method, url=url, **kwargs)
     62 
     63 

~/PythonHome/test_python/lib/python3.7/site-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    540         }
    541         send_kwargs.update(settings)
--> 542         resp = self.send(prep, **send_kwargs)
    543 
    544         return resp

~/PythonHome/test_python/lib/python3.7/site-packages/requests/sessions.py in send(self, request, **kwargs)
    653 
    654         # Send the request
--> 655         r = adapter.send(request, **kwargs)
    656 
    657         # Total elapsed time of the request (approximately)

~/PythonHome/test_python/lib/python3.7/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    512             if isinstance(e.reason, _SSLError):
    513                 # This branch is for urllib3 v1.22 and later.
--> 514                 raise SSLError(e, request=request)
    515 
    516             raise ConnectionError(e, request=request)

SSLError: HTTPSConnectionPool(host='ors.kmrb.or.kr', port=443): Max retries exceeded with url: /rating/inquiry_mv_view.do?RCV_NO=2288320&KIND=MV_FOR&GUBUN=N&&proc_state_code=onGoing&mv_kind=MV_FOR&vd_kind=VD_DOM&mv_use_title=%EC%8A%A4%ED%8C%8C%EC%9D%B4%EB%8D%94%EB%A7%A8 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1045)')))

이는 SSL 인증을 거쳐야하는 페이지의 특성상 그냥 요청을 하였기에 오류가 나는 부분입니다.

👨🏻‍💻 해결 방법

모든 페이지에서 동작한다고는 장담할 수 없지만 이 문제를 해결할 수 있는 정말 간단한 방법이 존재합니다.

바로 requests 요청을 보낼때 verify=False 라는 값을 넣어 요청을 보내면 방법이 그 방법입니다.

이는 SSL 인증 없이 요청을 보내겠다는 의미입니다.

import requests

url = "https://ors.kmrb.or.kr/rating/inquiry_mv_view.do?RCV_NO=2288320&KIND=MV_FOR&GUBUN=N&&proc_state_code=onGoing&mv_kind=MV_FOR&vd_kind=VD_DOM&mv_use_title=%EC%8A%A4%ED%8C%8C%EC%9D%B4%EB%8D%94%EB%A7%A8"

url_req = requests.get(url, verify=False)

url_req.text

바로 이전의 코드에서 verify=False 항목을 추가하여 다시 요청을 하게 되면 

'\r\n\r\n\r\n\r\n\r\n\r\n\r\n<!DOCTYPE html>\r\n<html lang="ko">\r\n<head>
\r\n\r\n<!--<meta http-equiv="X-UA-Compatible" content="IE=Edge">-->\r
\n<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />\r\
n<meta name="viewport" content="width=device-width,initial-scale=1.0,minimum-scale=1.0,
maximum-scale=1.0" />\r\n<meta name="format-detection" content="telephone=no, 
address=no, email=no" />\r\n\r\n<title>온라인등급분류서비스-등급진행자료조회 - 영화(Spider-Man: No Way Home)
</title>\r\n\r\n<link rel="stylesheet" href="/2017/css/style.css" type="text/css" />\
r\n<link rel="stylesheet" href="/2017/css/layout.css" type="text/css" />\r\n<link rel="stylesheet" 
href="/2017/css/contents.css" type="text/css" />\r\n<link rel="stylesheet" href="/2017/css/board.css" 
type="text/css" />\r\n<link rel="stylesheet" href="/2017/css/swiper.css" type="text/css" />\r\n
<link rel="stylesheet" href="/2017/css/forms.css">\r\n\r\n<link rel="stylesheet" 
href="/css/jquery-ui.css" type="text/css" media="all" />\r\n<link rel="stylesheet" href="/k_js/styles/kendo.common-material.min.css" />\r\n
<link rel="stylesheet" href="/k_js/styles/kendo.material.min.css" />\r\n\r\n<!-- favicon -->\r\n<link rel="shortcut icon" href="/images/favicon.ico" />\r\n
<link rel="icon" href="/images/favicon.ico" />\r\n<link rel="apple-touch-icon" href="/images/favicon.ico" />\r\n\r\n<link rel="stylesheet" href="/assets/css/font.css">
\r\n<link rel="stylesheet" href="/assets/css/reset.css">\r\n<link rel="stylesheet" href="/assets/css/slick-theme.css">\r\n<link rel="stylesheet" href="/assets/css/slick.css">
\r\n<link rel="stylesheet"

오류 없이 결과를 받을 수 있음을 볼 수 있습니다.

 

이외에 header를 추가해야하거나 할 수 도있는데 해당 방법은 직접 그러한 경우를 경험해보면 

추가로 업데이트를 진행해보려합니다.

 

읽어주셔서 감사합니다.