[Python] MissingSchema: Invalid URL : No scheme supplied. Perhaps you meant 원인과 해결방법! ( feat. requests )

🧑🏻‍💻 겪었던 일

최근 솔루션 개발 이후 뭔가 질문 리스트를 솔루션에 넣고 벌크로 테스트를 해보는 기능을 테스트 해보면서

MissingSchema: Invalid URL '<function <lambda> at 0x1072d9430>': No scheme supplied. Perhaps you meant http://<function <lambda> at 0x1072d9430>?

위와 같은 에러 메세지가 발생한 경우가 있었습니다.

 

처음에는 에러메세지에 MissingSchema 라는 단어를 보고

어? 이거는 DB 연동 관련된 부분인가보다! 하고 생각했습니다.

 

그런데 막상 해당 에러 메세지로 검색을 쭉 하다보니 DB 연동쪽 에러 발생이 아니었습니다.

* 발생 에러 메세지 ( 더보기 클릭 )

---------------------------------------------------------------------------
MissingSchema                             Traceback (most recent call last)
Input In [39], in <cell line: 1>()
----> 1 requests.get(FAQ_API_HOST)

File /opt/anaconda3/lib/python3.9/site-packages/requests/api.py:73, in get(url, params, **kwargs)
     62 def get(url, params=None, **kwargs):
     63     r"""Sends a GET request.
     64 
     65     :param url: URL for the new :class:`Request` object.
   (...)
     70     :rtype: requests.Response
     71     """
---> 73     return request("get", url, params=params, **kwargs)

File /opt/anaconda3/lib/python3.9/site-packages/requests/api.py:59, in request(method, url, **kwargs)
     55 # By using the 'with' statement we are sure the session is closed, thus we
     56 # avoid leaving sockets open which can trigger a ResourceWarning in some
     57 # cases, and look like a memory leak in others.
     58 with sessions.Session() as session:
---> 59     return session.request(method=method, url=url, **kwargs)

File /opt/anaconda3/lib/python3.9/site-packages/requests/sessions.py:573, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    560 # Create the Request.
    561 req = Request(
    562     method=method.upper(),
    563     url=url,
   (...)
    571     hooks=hooks,
    572 )
--> 573 prep = self.prepare_request(req)
    575 proxies = proxies or {}
    577 settings = self.merge_environment_settings(
    578     prep.url, proxies, stream, verify, cert
    579 )

File /opt/anaconda3/lib/python3.9/site-packages/requests/sessions.py:484, in Session.prepare_request(self, request)
    481     auth = get_netrc_auth(request.url)
    483 p = PreparedRequest()
--> 484 p.prepare(
    485     method=request.method.upper(),
    486     url=request.url,
    487     files=request.files,
    488     data=request.data,
    489     json=request.json,
    490     headers=merge_setting(
    491         request.headers, self.headers, dict_class=CaseInsensitiveDict
    492     ),
    493     params=merge_setting(request.params, self.params),
    494     auth=merge_setting(auth, self.auth),
    495     cookies=merged_cookies,
    496     hooks=merge_hooks(request.hooks, self.hooks),
    497 )
    498 return p

File /opt/anaconda3/lib/python3.9/site-packages/requests/models.py:368, in PreparedRequest.prepare(self, method, url, headers, files, data, params, auth, cookies, hooks, json)
    365 """Prepares the entire request with the given parameters."""
    367 self.prepare_method(method)
--> 368 self.prepare_url(url, params)
    369 self.prepare_headers(headers)
    370 self.prepare_cookies(cookies)

File /opt/anaconda3/lib/python3.9/site-packages/requests/models.py:439, in PreparedRequest.prepare_url(self, url, params)
    436     raise InvalidURL(*e.args)
    438 if not scheme:
--> 439     raise MissingSchema(
    440         f"Invalid URL {url!r}: No scheme supplied. "
    441         f"Perhaps you meant http://{url}?"
    442     )
    444 if not host:
    445     raise InvalidURL(f"Invalid URL {url!r}: No host supplied")

MissingSchema: Invalid URL '<function <lambda> at 0x1072d9430>': No scheme supplied. Perhaps you meant http://<function <lambda> at 0x1072d9430>?

🧑🏻‍💻 에러 발생 원인 

MissingSchema: Invalid URL : No scheme supplied. Perhaps you meant http:// ~~

마치 DB연동 관련 부분에서 발생한 것만 같은 이 에러는 

Python 의 requests 라는 라이브러리에서 발생하는 에러였습니다.

 

requests 라이브러리를 활용하여 API 를 호출하는 과정에서 

requests 의 메소드 안에 들어가는 파라미터 중 url 이라는 파라미터의 값이 형식에 맞지 않는 경우 발생합니다.

 

테스트 해보면서 확인한

에러가 발생할 수 있는 경우는

- requests 에 보내는 url 파라미터 값 에 http:// 또는 https:// 가 붙어있지 않은 경우 

( 에러 발생 상황 코드 보려면 더보기 클릭 )

실행 코드

import requests

requests.get(url="somjang.com")

발생 에러 메세지

---------------------------------------------------------------------------
MissingSchema                             Traceback (most recent call last)
Input In [2], in <cell line: 3>()
      1 import requests
----> 3 requests.get("somjang.com")

File /opt/anaconda3/lib/python3.9/site-packages/requests/api.py:73, in get(url, params, **kwargs)
     62 def get(url, params=None, **kwargs):
     63     r"""Sends a GET request.
     64 
     65     :param url: URL for the new :class:`Request` object.
   (...)
     70     :rtype: requests.Response
     71     """
---> 73     return request("get", url, params=params, **kwargs)

File /opt/anaconda3/lib/python3.9/site-packages/requests/api.py:59, in request(method, url, **kwargs)
     55 # By using the 'with' statement we are sure the session is closed, thus we
     56 # avoid leaving sockets open which can trigger a ResourceWarning in some
     57 # cases, and look like a memory leak in others.
     58 with sessions.Session() as session:
---> 59     return session.request(method=method, url=url, **kwargs)

File /opt/anaconda3/lib/python3.9/site-packages/requests/sessions.py:573, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    560 # Create the Request.
    561 req = Request(
    562     method=method.upper(),
    563     url=url,
   (...)
    571     hooks=hooks,
    572 )
--> 573 prep = self.prepare_request(req)
    575 proxies = proxies or {}
    577 settings = self.merge_environment_settings(
    578     prep.url, proxies, stream, verify, cert
    579 )

File /opt/anaconda3/lib/python3.9/site-packages/requests/sessions.py:484, in Session.prepare_request(self, request)
    481     auth = get_netrc_auth(request.url)
    483 p = PreparedRequest()
--> 484 p.prepare(
    485     method=request.method.upper(),
    486     url=request.url,
    487     files=request.files,
    488     data=request.data,
    489     json=request.json,
    490     headers=merge_setting(
    491         request.headers, self.headers, dict_class=CaseInsensitiveDict
    492     ),
    493     params=merge_setting(request.params, self.params),
    494     auth=merge_setting(auth, self.auth),
    495     cookies=merged_cookies,
    496     hooks=merge_hooks(request.hooks, self.hooks),
    497 )
    498 return p

File /opt/anaconda3/lib/python3.9/site-packages/requests/models.py:368, in PreparedRequest.prepare(self, method, url, headers, files, data, params, auth, cookies, hooks, json)
    365 """Prepares the entire request with the given parameters."""
    367 self.prepare_method(method)
--> 368 self.prepare_url(url, params)
    369 self.prepare_headers(headers)
    370 self.prepare_cookies(cookies)

File /opt/anaconda3/lib/python3.9/site-packages/requests/models.py:439, in PreparedRequest.prepare_url(self, url, params)
    436     raise InvalidURL(*e.args)
    438 if not scheme:
--> 439     raise MissingSchema(
    440         f"Invalid URL {url!r}: No scheme supplied. "
    441         f"Perhaps you meant http://{url}?"
    442     )
    444 if not host:
    445     raise InvalidURL(f"Invalid URL {url!r}: No host supplied")

MissingSchema: Invalid URL 'somjang.com': No scheme supplied. Perhaps you meant http://somjang.com?

 

- reqeusts 에 보내는 url 파라미터 값 속에 인코딩이 필요한데 인코딩이 되지 않은 문자가 포함되어있는 경우

( 에러 발생 상황 코드 보려면 더보기 클릭 )

실행 코드

import requests

requests.get(url="""https://msi.abc.com/admin/ui/feedbackCSV/reports
/5d14de32309baf0001501fb7 ?reports[]=pageviews&reports[]=
searches&from=Oct 1 2020&to=Oct 2 2020""")

발생 에러 메세지

---------------------------------------------------------------------------
gaierror                                  Traceback (most recent call last)
File /opt/anaconda3/lib/python3.9/site-packages/urllib3/connection.py:174, in HTTPConnection._new_conn(self)
    173 try:
--> 174     conn = connection.create_connection(
    175         (self._dns_host, self.port), self.timeout, **extra_kw
    176     )
    178 except SocketTimeout:

File /opt/anaconda3/lib/python3.9/site-packages/urllib3/util/connection.py:72, in create_connection(address, timeout, source_address, socket_options)
     68     return six.raise_from(
     69         LocationParseError(u"'%s', label empty or too long" % host), None
     70     )
---> 72 for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
     73     af, socktype, proto, canonname, sa = res

File /opt/anaconda3/lib/python3.9/socket.py:954, in getaddrinfo(host, port, family, type, proto, flags)
    953 addrlist = []
--> 954 for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
    955     af, socktype, proto, canonname, sa = res

gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

NewConnectionError                        Traceback (most recent call last)
File /opt/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py:703, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    702 # Make the request on the httplib connection object.
--> 703 httplib_response = self._make_request(
    704     conn,
    705     method,
    706     url,
    707     timeout=timeout_obj,
    708     body=body,
    709     headers=headers,
    710     chunked=chunked,
    711 )
    713 # If we're going to release the connection in ``finally:``, then
    714 # the response doesn't need to know about the connection. Otherwise
    715 # it will also try to release it and we'll have a double-release
    716 # mess.

File /opt/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py:386, in HTTPConnectionPool._make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    385 try:
--> 386     self._validate_conn(conn)
    387 except (SocketTimeout, BaseSSLError) as e:
    388     # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout.

File /opt/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py:1042, in HTTPSConnectionPool._validate_conn(self, conn)
   1041 if not getattr(conn, "sock", None):  # AppEngine might not have  `.sock`
-> 1042     conn.connect()
   1044 if not conn.is_verified:

File /opt/anaconda3/lib/python3.9/site-packages/urllib3/connection.py:358, in HTTPSConnection.connect(self)
    356 def connect(self):
    357     # Add certificate verification
--> 358     self.sock = conn = self._new_conn()
    359     hostname = self.host

File /opt/anaconda3/lib/python3.9/site-packages/urllib3/connection.py:186, in HTTPConnection._new_conn(self)
    185 except SocketError as e:
--> 186     raise NewConnectionError(
    187         self, "Failed to establish a new connection: %s" % e
    188     )
    190 return conn

NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x120b628b0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

MaxRetryError                             Traceback (most recent call last)
File /opt/anaconda3/lib/python3.9/site-packages/requests/adapters.py:489, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    488 if not chunked:
--> 489     resp = conn.urlopen(
    490         method=request.method,
    491         url=url,
    492         body=request.body,
    493         headers=request.headers,
    494         redirect=False,
    495         assert_same_host=False,
    496         preload_content=False,
    497         decode_content=False,
    498         retries=self.max_retries,
    499         timeout=timeout,
    500     )
    502 # Send the request.
    503 else:

File /opt/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py:787, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    785     e = ProtocolError("Connection aborted.", e)
--> 787 retries = retries.increment(
    788     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
    789 )
    790 retries.sleep()

File /opt/anaconda3/lib/python3.9/site-packages/urllib3/util/retry.py:592, in Retry.increment(self, method, url, response, error, _pool, _stacktrace)
    591 if new_retry.is_exhausted():
--> 592     raise MaxRetryError(_pool, url, error or ResponseError(cause))
    594 log.debug("Incremented Retry for (url='%s'): %r", url, new_retry)

MaxRetryError: HTTPSConnectionPool(host='msi.abc.com', port=443): Max retries exceeded with url: /admin/ui/feedbackCSV/reports%0A/5d14de32309baf0001501fb7%20?reports%5B%5D=pageviews&reports%5B%5D=%0Asearches&from=Oct%201%202020&to=Oct%202%202020 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x120b628b0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

During handling of the above exception, another exception occurred:

ConnectionError                           Traceback (most recent call last)
Input In [46], in <cell line: 3>()
      1 import requests
----> 3 requests.get(url="""https://msi.abc.com/admin/ui/feedbackCSV/reports
      4 /5d14de32309baf0001501fb7 ?reports[]=pageviews&reports[]=
      5 searches&from=Oct 1 2020&to=Oct 2 2020""")

File /opt/anaconda3/lib/python3.9/site-packages/requests/api.py:73, in get(url, params, **kwargs)
     62 def get(url, params=None, **kwargs):
     63     r"""Sends a GET request.
     64 
     65     :param url: URL for the new :class:`Request` object.
   (...)
     70     :rtype: requests.Response
     71     """
---> 73     return request("get", url, params=params, **kwargs)

File /opt/anaconda3/lib/python3.9/site-packages/requests/api.py:59, in request(method, url, **kwargs)
     55 # By using the 'with' statement we are sure the session is closed, thus we
     56 # avoid leaving sockets open which can trigger a ResourceWarning in some
     57 # cases, and look like a memory leak in others.
     58 with sessions.Session() as session:
---> 59     return session.request(method=method, url=url, **kwargs)

File /opt/anaconda3/lib/python3.9/site-packages/requests/sessions.py:587, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    582 send_kwargs = {
    583     "timeout": timeout,
    584     "allow_redirects": allow_redirects,
    585 }
    586 send_kwargs.update(settings)
--> 587 resp = self.send(prep, **send_kwargs)
    589 return resp

File /opt/anaconda3/lib/python3.9/site-packages/requests/sessions.py:701, in Session.send(self, request, **kwargs)
    698 start = preferred_clock()
    700 # Send the request
--> 701 r = adapter.send(request, **kwargs)
    703 # Total elapsed time of the request (approximately)
    704 elapsed = preferred_clock() - start

File /opt/anaconda3/lib/python3.9/site-packages/requests/adapters.py:565, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    561     if isinstance(e.reason, _SSLError):
    562         # This branch is for urllib3 v1.22 and later.
    563         raise SSLError(e, request=request)
--> 565     raise ConnectionError(e, request=request)
    567 except ClosedPoolError as e:
    568     raise ConnectionError(e, request=request)

ConnectionError: HTTPSConnectionPool(host='msi.abc.com', port=443): Max retries exceeded with url: /admin/ui/feedbackCSV/reports%0A/5d14de32309baf0001501fb7%20?reports%5B%5D=pageviews&reports%5B%5D=%0Asearches&from=Oct%201%202020&to=Oct%202%202020 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x120b628b0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

requests.exceptions.MissingSchema: Invalid URL Python API Get request

인 것 같습니다.

🧑🏻‍💻 해결 방법

해결방법은 위에서 이야기한 2가지 경우에 대해서 각각 작성해보았습니다.

- requests 에 보내는 url 파라미터 값 에 http:// 또는 https:// 가 붙어있지 않은 경우

먼저, 내가 requests.get / requests.post 등에 보내는 url 값에 http:// 가 붙어있지 않은 것이 확인 해봅니다.

만약! http:// 또는 https:// 가 존재하지 않는 것이 확인 되었다면 

내가 요청을하는 API가 http:// / https:// 중에 어떤 것을 붙여야하는지 확인 후 추가하여 요청하시면 됩니다.

예시 ) url 을 naver.com 으로 해서 에러가 발생한 경우 ( https:// 가 누락 된 경우 )

- 기존 에러 발생 코드 

import requests

requests.get(url="naver.com")

- 수정 코드

import requests

requests.get(url="https://naver.com")

- reqeusts 에 보내는 url 파라미터 값 속에 인코딩이 필요한데 인코딩이 되지 않은 문자가 포함되어있는 경우

예시 ) 파라미터 값들을 모두 url 에 그냥 붙여넣어 에러가 발생하는 경우 ( 출처 : 스택오버플로우 )

- 기존 에러 발생 코드 

import requests

requests.get(url="""https://msi.abc.com/admin/ui/feedbackCSV/reports
/5d14de32309baf0001501fb7 ?reports[]=pageviews&reports[]=
searches&from=Oct 1 2020&to=Oct 2 2020""")

- 수정 코드 ( reqeusts 에 parms 파라미터에 원하는 파라미터 dictionary 넣어주는 방식으로 수정 )

import requests
from requests.auth import HTTPBasicAuth 
import json
import urllib.parse

url = "https://msi.abc.com/admin/ui/feedbackCSV/reports/5d14de32309baf0001501fb7"
payload = {
    'reports[]':'pageviews', 'reports[]':'searches',
    'from':'Oct 1 2020', 'to':'Oct 2 2020' 
}

authParams = HTTPBasicAuth('admin@abc.com', 'Summer2020')

response = requests.get(url,params=payload, auth =authParams ) 
print(response.content)

이런 경우는 urllib.qoute 메소드로 넣어줄 인자값을 인코딩 해주거나

requests 의 param / payload / json / data 중 알맞은 파라미터를 골라

해당 파라미터에 내가 요청 시 같이 보내고자 하는 값들을 알맞은 형식으로 넣어주시면 됩니다.

 

요거에 대한 부분은 다음에 한번 정리해보려합니다.

 

🧑🏻‍💻 끝!

Schema 라고 에러메세지에 적혀있어서 DB 관련 에러인 줄 알았지만 

사실 다른 부분의 에러였던 것을 보면서

아 그냥 단순하게 에러 메세지만 보고서 판단하기 보다는 좀 더 다양하게 고려해보고 파악해보는게 좋겠다

라는 생각이 들었습니다.

 

읽어주셔서 감사합니다!