Python Requests basics to intermediate

1. Basic GET Request

Copy code

import requests response = requests.get('https://jsonplaceholder.typicode.com/posts/1') print(response.status_code) print(response.json())

Explanation:

requests.get(url) sends a GET request to the specified URL.
response.status_code prints the HTTP status code (e.g., 200 for success).
response.json() parses the response to a JSON object.

2. GET Request with Parameters

Copy code

import requests params = {'userId': 1} response = requests.get('https://jsonplaceholder.typicode.com/posts', params=params) print(response.url) print(response.json())

Explanation:

params is a dictionary of query parameters to append to the URL.
response.url shows the full URL after parameters are added.

3. Handling Response Headers

Copy code

import requests response = requests.get('https://jsonplaceholder.typicode.com/posts/1') print(response.headers)

Explanation:

response.headers returns the headers of the response.

4. POST Request with JSON Data

Copy code

import requests data = { 'title': 'foo', 'body': 'bar', 'userId': 1 } response = requests.post('https://jsonplaceholder.typicode.com/posts', json=data) print(response.status_code) print(response.json())

Explanation:

json=data sends JSON data in the POST request.
requests.post(url, json=data) sends a POST request with the specified JSON data.

5. POST Request with Form Data

Copy code

import requests data = { 'username': 'user', 'password': 'pass' } response = requests.post('https://httpbin.org/post', data=data) print(response.json())

Explanation:

data sends form-encoded data in the POST request.

6. PUT Request

Copy code

import requests data = { 'id': 1, 'title': 'updated title', 'body': 'updated body', 'userId': 1 } response = requests.put('https://jsonplaceholder.typicode.com/posts/1', json=data) print(response.json())

Explanation:

requests.put(url, json=data) sends a PUT request with the specified JSON data.

7. DELETE Request

Copy code

import requests response = requests.delete('https://jsonplaceholder.typicode.com/posts/1') print(response.status_code)

Explanation:

requests.delete(url) sends a DELETE request to the specified URL.

8. Handling Timeout

Copy code

import requests try: response = requests.get('https://httpbin.org/delay/5', timeout=2) except requests.Timeout: print('The request timed out')

Explanation:

timeout specifies the maximum number of seconds to wait for a response.
requests.Timeout is raised if the request exceeds the timeout.

9. Custom Headers

Copy code

import requests headers = { 'User-Agent': 'my-app/0.0.1' } response = requests.get('https://httpbin.org/headers', headers=headers) print(response.json())

Explanation:

headers is a dictionary of HTTP headers to send with the request.

10. Authentication

Copy code

import requests response = requests.get('https://httpbin.org/basic-auth/user/pass', auth=('user', 'pass')) print(response.status_code)

Explanation:

auth provides basic authentication credentials.

11. Handling Redirects

Copy code

import requests response = requests.get('http://github.com') print(response.url) print(response.history)

Explanation:

response.url shows the final URL after redirection.
response.history shows a list of response objects that were created to complete the request.

12. Streaming Requests

python

Copy code

import requests response = requests.get('https://jsonplaceholder.typicode.com/photos', stream=True) for chunk in response.iter_content(chunk_size=1024): print(chunk)

Explanation:

stream=True allows handling the content in chunks.
response.iter_content(chunk_size) iterates over the response data in chunks.

13. Downloading Files

python

Copy code

import requests url = 'https://via.placeholder.com/150' response = requests.get(url) with open('image.png', 'wb') as file: file.write(response.content)

Explanation:

response.content returns the raw bytes of the response content.
with open opens a file in write-binary mode to save the content.

14. Uploading Files

python

Copy code

import requests files = {'file': open('image.png', 'rb')} response = requests.post('https://httpbin.org/post', files=files) print(response.json())

Explanation:

files is a dictionary of files to send in the POST request.
open('filename', 'rb') opens a file in read-binary mode.

15. Session Objects

python

Copy code

import requests session = requests.Session() session.get('https://httpbin.org/cookies/set/sessioncookie/123456789') response = session.get('https://httpbin.org/cookies') print(response.json())

Explanation:

requests.Session() creates a session object to persist certain parameters across requests.
session.get(url) sends a GET request within the session.

16. Custom Timeout per Request

python

Copy code

import requests response = requests.get('https://httpbin.org/delay/2', timeout=(3, 5)) print(response.status_code)

Explanation:

timeout=(connect, read) sets the timeout for connect and read operations.

17. Proxies

python

Copy code

import requests proxies = { 'http': 'http://10.10.1.10:3128', 'https': 'http://10.10.1.10:1080' } response = requests.get('https://httpbin.org/ip', proxies=proxies) print(response.json())

Explanation:

proxies is a dictionary of proxy URLs for HTTP and HTTPS.

18. Custom SSL Certificates

python

Copy code

import requests response = requests.get('https://example.com', verify='/path/to/certfile') print(response.status_code)

Explanation:

verify specifies the path to a custom SSL certificate file.

19. Handling Cookies

python

Copy code

import requests url = 'https://httpbin.org/cookies/set/sessioncookie/123456789' response = requests.get(url) print(response.cookies['sessioncookie'])

Explanation:

response.cookies is a dictionary-like object containing cookies sent by the server.

20. Parsing HTML with BeautifulSoup

python

Copy code

import requests from bs4 import BeautifulSoup response = requests.get('https://example.com') soup = BeautifulSoup(response.text, 'html.parser') print(soup.title.string)

Explanation:

response.text returns the response content as a string.
BeautifulSoup parses the HTML content and provides methods to navigate and search the parse tree.

Intermediate examples

1. Handling JSON with Nested Structures

python

Copy code

import requests response = requests.get('https://jsonplaceholder.typicode.com/users/1') data = response.json() print(data['address']['geo'])

Explanation:

This example fetches user data and extracts the nested geo information within the address field.

2. Handling Errors with Try/Except

python

Copy code

import requests try: response = requests.get('https://jsonplaceholder.typicode.com/invalid-url') response.raise_for_status() except requests.exceptions.HTTPError as err: print(f"HTTP error occurred: {err}") except Exception as err: print(f"Other error occurred: {err}")

Explanation:

This code handles HTTP errors using raise_for_status() and catches other exceptions using try/except.

3. Retrying Requests with Backoff

python

Copy code

import requests from requests.adapters import HTTPAdapter from requests.packages.urllib3.util.retry import Retry session = requests.Session() retry = Retry(total=5, backoff_factor=1, status_forcelist=[500, 502, 503, 504]) adapter = HTTPAdapter(max_retries=retry) session.mount('http://', adapter) session.mount('https://', adapter) response = session.get('https://httpbin.org/status/500') print(response.status_code)

Explanation:

This code sets up automatic retries for failed requests using Retry and HTTPAdapter.

4. Using Sessions for Cookie Persistence

python

Copy code

import requests session = requests.Session() session.get('https://httpbin.org/cookies/set/sessioncookie/123456789') response = session.get('https://httpbin.org/cookies') print(response.json())

Explanation:

Sessions keep cookies between requests, useful for maintaining login sessions.

5. Posting JSON Data

python

Copy code

import requests url = 'https://jsonplaceholder.typicode.com/posts' data = { 'title': 'foo', 'body': 'bar', 'userId': 1 } response = requests.post(url, json=data) print(response.json())

Explanation:

This example sends a POST request with JSON data and prints the response.

6. Posting Multipart Form Data

python

Copy code

import requests url = 'https://httpbin.org/post' files = {'file': open('report.xls', 'rb')} response = requests.post(url, files=files) print(response.json())

Explanation:

This code uploads a file using multipart form data in a POST request.

7. Handling Timeouts and Retries

python

Copy code

import requests from requests.adapters import HTTPAdapter from requests.packages.urllib3.util.retry import Retry session = requests.Session() retry = Retry(total=3, backoff_factor=0.3) adapter = HTTPAdapter(max_retries=retry) session.mount('http://', adapter) session.mount('https://', adapter) try: response = session.get('https://httpbin.org/delay/5', timeout=2) except requests.exceptions.Timeout: print("The request timed out")

Explanation:

This code handles request timeouts and retries the request up to 3 times with exponential backoff.

8. Handling SSL Verification

python

Copy code

import requests response = requests.get('https://expired.badssl.com/', verify=False) print(response.status_code)

Explanation:

verify=False disables SSL verification, useful for testing but not recommended for production.

9. Custom Authentication

python

Copy code

import requests from requests.auth import HTTPBasicAuth response = requests.get('https://httpbin.org/basic-auth/user/pass', auth=HTTPBasicAuth('user', 'pass')) print(response.status_code)

Explanation:

This code demonstrates how to use HTTP basic authentication with the HTTPBasicAuth class.

10. Using Custom Headers

python

Copy code

import requests url = 'https://httpbin.org/headers' headers = {'User-Agent': 'my-app/0.0.1'} response = requests.get(url, headers=headers) print(response.json())

Explanation:

Custom headers can be added to the request by passing a dictionary to the headers parameter.

11. Parsing XML Responses

python

Copy code

import requests import xml.etree.ElementTree as ET response = requests.get('https://www.w3schools.com/xml/note.xml') root = ET.fromstring(response.content) for child in root: print(child.tag, child.text)

Explanation:

This example parses an XML response using ElementTree and prints each element's tag and text.

12. Handling Redirects

python

Copy code

import requests response = requests.get('http://github.com') print(response.url) print(response.history)

Explanation:

response.history provides a list of redirect responses that were followed to reach the final URL.

13. Using Query Parameters

python

Copy code

import requests params = {'page': 2, 'count': 10} response = requests.get('https://jsonplaceholder.typicode.com/posts', params=params) print(response.url)

Explanation:

Query parameters can be added to the URL by passing a dictionary to the params parameter.

14. Downloading Large Files

python

Copy code

import requests url = 'https://speed.hetzner.de/100MB.bin' response = requests.get(url, stream=True) with open('large_file.bin', 'wb') as f: for chunk in response.iter_content(chunk_size=8192): f.write(chunk)

Explanation:

stream=True enables streaming, and iter_content allows downloading large files in chunks.

15. Sending Complex JSON Data

python

Copy code

import requests url = 'https://httpbin.org/post' data = { 'user': { 'id': 1, 'name': 'John Doe', 'contacts': [ {'type': 'email', 'value': 'john.doe@example.com'}, {'type': 'phone', 'value': '123-456-7890'} ] } } response = requests.post(url, json=data) print(response.json())

Explanation:

This example sends complex nested JSON data in a POST request.

16. Advanced Error Handling

python

Copy code

import requests try: response = requests.get('https://httpbin.org/status/404') response.raise_for_status() except requests.exceptions.HTTPError as http_err: print(f'HTTP error occurred: {http_err}') except Exception as err: print(f'Other error occurred: {err}') else: print('Success!')

Explanation:

This code demonstrates advanced error handling with raise_for_status() and multiple exception types.

17. Using Hooks

python

Copy code

import requests def response_hook(response, *args, **kwargs): print(response.status_code) response = requests.get('https://httpbin.org/get', hooks={'response': response_hook})

Explanation:

Hooks allow attaching custom callback functions to request events.

18. Session Cookies

python

Copy code

import requests session = requests.Session() session.get('https://httpbin.org/cookies/set/sessioncookie/123456789') response = session.get('https://httpbin.org/cookies') print(response.json())

Explanation:

This example demonstrates how session cookies are preserved across requests.

19. Custom Timeout for Connect and Read

python

Copy code

import requests response = requests.get('https://httpbin.org/delay/1', timeout=(5, 10)) print(response.status_code)

Explanation:

Separate timeouts can be set for the connect and read operations by passing a tuple to the timeout parameter.

20. Custom Proxy Configuration

python

Copy code

import requests proxies = { 'http': 'http://10.10.1.10:3128', 'https': 'http://10.10.1.10:1080' } response = requests.get('https://httpbin.org/ip', proxies=proxies) print(response.json())

Explanation:

Proxies can be configured for both HTTP and HTTPS requests by passing a dictionary to the proxies parameter.

New examples for practice

. Handling JSON with Nested Structures

python

Copy code

import requests response = requests.get('https://jsonplaceholder.typicode.com/users/1') data = response.json() print(data['address']['geo'])

Explanation:

requests.get() sends a GET request to the specified URL.
.json() method converts the response to a Python dictionary.
data['address']['geo'] accesses the nested geo field inside the address dictionary.

2. Handling Errors with Try/Except

python

Copy code

import requests try: response = requests.get('https://jsonplaceholder.typicode.com/invalid-url') response.raise_for_status() except requests.exceptions.HTTPError as err: print(f"HTTP error occurred: {err}") except Exception as err: print(f"Other error occurred: {err}")

Explanation:

The try block attempts to send a GET request.
raise_for_status() will raise an HTTPError if the HTTP request returned an unsuccessful status code.
The except blocks catch and print specific errors.

3. Retrying Requests with Backoff

python

Copy code

import requests from requests.adapters import HTTPAdapter from requests.packages.urllib3.util.retry import Retry session = requests.Session() retry = Retry(total=5, backoff_factor=1, status_forcelist=[500, 502, 503, 504]) adapter = HTTPAdapter(max_retries=retry) session.mount('http://', adapter) session.mount('https://', adapter) response = session.get('https://httpbin.org/status/500') print(response.status_code)

Explanation:

Retry sets up retry logic with a backoff factor.
HTTPAdapter mounts the retry logic to the session.
session.get() uses the retry logic for the request.

4. Using Sessions for Cookie Persistence

python

Copy code

import requests session = requests.Session() session.get('https://httpbin.org/cookies/set/sessioncookie/123456789') response = session.get('https://httpbin.org/cookies') print(response.json())

Explanation:

requests.Session() creates a session object.
Cookies set in one request are persisted and used in subsequent requests.

5. Posting JSON Data

python

Copy code

import requests url = 'https://jsonplaceholder.typicode.com/posts' data = { 'title': 'foo', 'body': 'bar', 'userId': 1 } response = requests.post(url, json=data) print(response.json())

Explanation:

requests.post() sends a POST request with JSON data.
The json parameter is used to send JSON data in the body of the request.

6. Posting Multipart Form Data

python

Copy code

import requests url = 'https://httpbin.org/post' files = {'file': open('report.xls', 'rb')} response = requests.post(url, files=files) print(response.json())

Explanation:

files parameter allows file uploads.
'rb' mode opens the file in binary read mode for uploading.

7. Handling Timeouts and Retries

python

Copy code

import requests from requests.adapters import HTTPAdapter from requests.packages.urllib3.util.retry import Retry session = requests.Session() retry = Retry(total=3, backoff_factor=0.3) adapter = HTTPAdapter(max_retries=retry) session.mount('http://', adapter) session.mount('https://', adapter) try: response = session.get('https://httpbin.org/delay/5', timeout=2) except requests.exceptions.Timeout: print("The request timed out")

Explanation:

timeout=2 sets a timeout for the request.
Retry logic retries the request if it fails.

8. Handling SSL Verification

python

Copy code

import requests response = requests.get('https://expired.badssl.com/', verify=False) print(response.status_code)

Explanation:

verify=False disables SSL verification.
Useful for testing, but not recommended for production due to security risks.

9. Custom Authentication

python

Copy code

import requests from requests.auth import HTTPBasicAuth response = requests.get('https://httpbin.org/basic-auth/user/pass', auth=HTTPBasicAuth('user', 'pass')) print(response.status_code)

Explanation:

HTTPBasicAuth is used to provide basic authentication credentials.

10. Using Custom Headers

python

Copy code

import requests url = 'https://httpbin.org/headers' headers = {'User-Agent': 'my-app/0.0.1'} response = requests.get(url, headers=headers) print(response.json())

Explanation:

Custom headers can be sent with a request using the headers parameter.

11. Parsing XML Responses

python

Copy code

import requests import xml.etree.ElementTree as ET response = requests.get('https://www.w3schools.com/xml/note.xml') root = ET.fromstring(response.content) for child in root: print(child.tag, child.text)

Explanation:

ET.fromstring() parses XML content.
Iterating through the XML tree to print tags and text.

12. Handling Redirects

python

Copy code

import requests response = requests.get('http://github.com') print(response.url) print(response.history)

Explanation:

response.history contains the list of redirect responses.
response.url is the final URL after all redirects.

13. Using Query Parameters

python

Copy code

import requests params = {'page': 2, 'count': 10} response = requests.get('https://jsonplaceholder.typicode.com/posts', params=params) print(response.url)

Explanation:

Query parameters can be passed using the params parameter.
params is a dictionary of query parameters.

14. Downloading Large Files

python

Copy code

import requests url = 'https://speed.hetzner.de/100MB.bin' response = requests.get(url, stream=True) with open('large_file.bin', 'wb') as f: for chunk in response.iter_content(chunk_size=8192): f.write(chunk)

Explanation:

stream=True enables streaming mode.
iter_content() allows downloading in chunks to handle large files.

15. Sending Complex JSON Data

python

Copy code

import requests url = 'https://httpbin.org/post' data = { 'user': { 'id': 1, 'name': 'John Doe', 'contacts': [ {'type': 'email', 'value': 'john.doe@example.com'}, {'type': 'phone', 'value': '123-456-7890'} ] } } response = requests.post(url, json=data) print(response.json())

Explanation:

Complex nested JSON data can be sent using the json parameter.

16. Advanced Error Handling

python

Copy code

import requests try: response = requests.get('https://httpbin.org/status/404') response.raise_for_status() except requests.exceptions.HTTPError as http_err: print(f'HTTP error occurred: {http_err}') except Exception as err: print(f'Other error occurred: {err}') else: print('Success!')

Explanation:

raise_for_status() raises an HTTPError for bad responses.
else block runs if no exception is raised.

17. Using Hooks

python

Copy code

import requests def response_hook(response, *args, **kwargs): print(response.status_code) response = requests.get('https://httpbin.org/get', hooks={'response': response_hook})

Explanation:

Hooks allow attaching custom callback functions to request events.

18. Session Cookies

python

Copy code

import requests session = requests.Session() session.get('https://httpbin.org/cookies/set/sessioncookie/123456789') response = session.get('https://httpbin.org/cookies') print(response.json())

Explanation:

Session cookies are persisted across requests within the session.

19. Custom Timeout for Connect and Read

python

Copy code

import requests response = requests.get('https://httpbin.org/delay/1', timeout=(5, 10)) print(response.status_code)

Explanation:

timeout can be a tuple (connect timeout, read timeout).

20. Custom Proxy Configuration

python

Copy code

import requests proxies = { 'http': 'http://10.10.1.10:3128', 'https': 'http://10.10.1.10:1080' } response = requests.get('https://httpbin.org/ip', proxies=proxies) print(response.json())

Explanation:

Proxies can be configured for both HTTP and HTTPS requests using the proxies parameter.

These examples cover a range of intermediate-level topics and should give you a solid un derstanding of how to use the requests library in Python for more complex tasks.

Python Requests Module: Detailed and Engaging Notes

1. Handling JSON with Nested Structures

Code Example:

python

Copy code

import requests response = requests.get('https://jsonplaceholder.typicode.com/users/1') data = response.json() print(data['address']['geo'])

Explanation:

requests.get('URL') sends a GET request to the specified URL.
The .json() method converts the response to a Python dictionary.
data['address']['geo'] accesses the nested geo field inside the address dictionary.

Key Points:

JSON responses are easily handled by converting them to dictionaries.
Accessing nested structures requires understanding the hierarchy.

Practical Tip: Use JSON data to parse and manipulate responses from web APIs, making it easy to work with structured data.

2. Handling Errors with Try/Except

Code Example:

python

Copy code

import requests try: response = requests.get('https://jsonplaceholder.typicode.com/invalid-url') response.raise_for_status() except requests.exceptions.HTTPError as err: print(f"HTTP error occurred: {err}") except Exception as err: print(f"Other error occurred: {err}")

Explanation:

The try block attempts to send a GET request.
raise_for_status() will raise an HTTPError if the HTTP request returned an unsuccessful status code.
The except blocks catch and print specific errors.

Key Points:

Error handling is crucial for robust programs.
Use try/except to manage exceptions and provide meaningful messages.

Practical Tip: Always handle potential errors in network requests to avoid crashes and ensure a smooth user experience.

3. Retrying Requests with Backoff

Code Example:

python

Copy code

import requests from requests.adapters import HTTPAdapter from requests.packages.urllib3.util.retry import Retry session = requests.Session() retry = Retry(total=5, backoff_factor=1, status_forcelist=[500, 502, 503, 504]) adapter = HTTPAdapter(max_retries=retry) session.mount('http://', adapter) session.mount('https://', adapter) response = session.get('https://httpbin.org/status/500') print(response.status_code)

Explanation:

Retry sets up retry logic with a backoff factor.
HTTPAdapter mounts the retry logic to the session.
session.get() uses the retry logic for the request.

Key Points:

Retries help handle transient errors gracefully.
The backoff_factor controls the delay between retries.

Practical Tip: Implement retry logic to make your application more resilient to temporary network issues.

4. Using Sessions for Cookie Persistence

Code Example:

python

Copy code

import requests session = requests.Session() session.get('https://httpbin.org/cookies/set/sessioncookie/123456789') response = session.get('https://httpbin.org/cookies') print(response.json())

Explanation:

requests.Session() creates a session object.
Cookies set in one request are persisted and used in subsequent requests.

Key Points:

Sessions allow cookies to persist across multiple requests.
Useful for maintaining stateful interactions with a web service.

Practical Tip: Use sessions to handle login cookies and maintain state across multiple requests without needing to manually manage cookies.

5. Posting JSON Data

Code Example:

python

Copy code

import requests url = 'https://jsonplaceholder.typicode.com/posts' data = { 'title': 'foo', 'body': 'bar', 'userId': 1 } response = requests.post(url, json=data) print(response.json())

Explanation:

requests.post() sends a POST request with JSON data.
The json parameter is used to send JSON data in the body of the request.

Key Points:

Posting JSON data is straightforward with the json parameter.
This is commonly used for creating or updating resources in web APIs.

Practical Tip: Use JSON data to communicate structured information with your web service endpoints efficiently.

6. Posting Multipart Form Data

Code Example:

python

Copy code

import requests url = 'https://httpbin.org/post' files = {'file': open('report.xls', 'rb')} response = requests.post(url, files=files) print(response.json())

Explanation:

The files parameter allows file uploads.
'rb' mode opens the file in binary read mode for uploading.

Key Points:

Multipart form data is used for uploading files.
Ensure files are opened in binary mode for compatibility.

Practical Tip: Use multipart form data to upload files to a server, such as images, documents, or any binary data.

7. Handling Timeouts and Retries

Code Example:

python

Copy code

import requests from requests.adapters import HTTPAdapter from requests.packages.urllib3.util.retry import Retry session = requests.Session() retry = Retry(total=3, backoff_factor=0.3) adapter = HTTPAdapter(max_retries=retry) session.mount('http://', adapter) session.mount('https://', adapter) try: response = session.get('https://httpbin.org/delay/5', timeout=2) except requests.exceptions.Timeout: print("The request timed out")

Explanation:

timeout=2 sets a timeout for the request.
Retry logic retries the request if it fails.

Key Points:

Timeouts prevent your application from hanging indefinitely.
Retries with backoff ensure resilience against temporary issues.

Practical Tip: Configure timeouts and retries to improve the reliability and performance of your application.

8. Handling SSL Verification

Code Example:

python

Copy code

import requests response = requests.get('https://expired.badssl.com/', verify=False) print(response.status_code)

Explanation:

verify=False disables SSL verification.
Useful for testing, but not recommended for production due to security risks.

Key Points:

Disabling SSL verification can expose your application to security risks.
Only disable SSL verification in a controlled testing environment.

Practical Tip: Always enable SSL verification in production to ensure secure communication.

9. Custom Authentication

Code Example:

python

Copy code

import requests from requests.auth import HTTPBasicAuth response = requests.get('https://httpbin.org/basic-auth/user/pass', auth=HTTPBasicAuth('user', 'pass')) print(response.status_code)

Explanation:

HTTPBasicAuth is used to provide basic authentication credentials.

Key Points:

Basic authentication sends user credentials with each request.
Ensure sensitive information is transmitted securely.

Practical Tip: Use custom authentication to access protected resources, ensuring credentials are handled securely.

10. Using Custom Headers

Code Example:

python

Copy code

import requests url = 'https://httpbin.org/headers' headers = {'User-Agent': 'my-app/0.0.1'} response = requests.get(url, headers=headers) print(response.json())

Explanation:

Custom headers can be sent with a request using the headers parameter.

Key Points:

Headers can provide additional information to the server.
Common headers include User-Agent, Authorization, and Content-Type.

Practical Tip: Use custom headers to tailor your requests and provide necessary context or authentication tokens.

11. Parsing XML Responses

Code Example:

python

Copy code

import requests import xml.etree.ElementTree as ET response = requests.get('https://www.w3schools.com/xml/note.xml') root = ET.fromstring(response.content) for child in root: print(child.tag, child.text)

Explanation:

ET.fromstring() parses XML content.
Iterating through the XML tree to print tags and text.

Key Points:

XML responses can be parsed using the xml.etree.ElementTree module.
Access elements and attributes using standard XML parsing techniques.

Practical Tip: Use XML parsing to handle responses from APIs that return XML data, such as some legacy systems or specific web services.

12. Handling Redirects

Code Example:

python

Copy code

import requests response = requests.get('http://github.com') print(response.url) print(response.history)

Explanation:

response.history contains the list of redirect responses.
response.url is the final URL after all redirects.

Key Points:

Requests automatically follow redirects by default.
response.history helps trace the sequence of redirects.

Practical Tip: Monitor redirects to understand how requests are being routed and ensure you're reaching the intended endpoint.

13. Using Query Parameters

Code Example:

python

Copy code

import requests params = {'page': 2, 'count': 10} response = requests.get('https://jsonplaceholder.typicode.com/posts', params=params) print(response.url)

Explanation:

Query parameters can be passed using the params parameter.
params is a dictionary of query parameters.

Key Points:

Query parameters are appended to the URL.
Useful for filtering and paginating results.

Practical Tip: Use query parameters to refine your API requests and retrieve specific data subsets.

14. Downloading Large Files

Code Example:

python

Copy code

import requests url = 'https://speed.hetzner.de/100MB.bin' response = requests.get(url, stream=True) with open('large_file.bin', 'wb') as f: for chunk in response.iter_content(chunk_size=8192): f.write(chunk)

Explanation:

stream=True enables streaming mode.
iter_content() iterates over the response in chunks.

Key Points:

Streaming is efficient for downloading large files.
Chunked writing prevents high memory usage.

Practical Tip: Stream and save large files in manageable chunks to avoid memory overload.

15. Uploading Large Files

Code Example:

python

Copy code

import requests url = 'https://httpbin.org/post' files = {'file': ('large_file.bin', open('large_file.bin', 'rb'), 'application/octet-stream')} response = requests.post(url, files=files) print(response.json())

Explanation:

The files parameter handles file uploads.
Specify the file name, file object, and MIME type.

Key Points:

Uploading large files requires specifying appropriate MIME types.
Ensure files are opened in binary mode for compatibility.

Practical Tip: Upload large files using appropriate MIME types to ensure the server handles them correctly.

16. Using Proxies

Code Example:

python

Copy code

import requests proxies = { 'http': 'http://10.10.1.10:3128', 'https': 'http://10.10.1.10:1080', } response = requests.get('https://httpbin.org/ip', proxies=proxies) print(response.json())

Explanation:

The proxies parameter allows requests to go through specified proxy servers.

Key Points:

Proxies can be used for security, caching, or bypassing restrictions.
Ensure proxy settings are correctly configured.

Practical Tip: Use proxies to enhance security, manage traffic, or access restricted content.

17. Custom Timeout Settings

Code Example:

python

Copy code

import requests response = requests.get('https://httpbin.org/delay/2', timeout=1) print(response.status_code)

Explanation:

The timeout parameter specifies the maximum time to wait for a response.

Key Points:

Timeouts prevent indefinite waits.
Can be set for both connection and read operations.

Practical Tip: Set appropriate timeouts to ensure your application remains responsive.

18. Session Object Reuse

Code Example:

python

Copy code

import requests session = requests.Session() session.headers.update({'User-Agent': 'my-app/0.0.1'}) response = session.get('https://httpbin.org/headers') print(response.json())

Explanation:

requests.Session() creates a session object.
Session headers are persisted across requests.

Key Points:

Reusing session objects can improve performance.
Useful for maintaining state and settings across multiple requests.

Practical Tip: Reuse session objects to optimize performance and manage stateful interactions efficiently.

19. Debugging Requests

Code Example:

python

Copy code

import requests import logging logging.basicConfig(level=logging.DEBUG) response = requests.get('https://httpbin.org/get') print(response.text)

Explanation:

logging.basicConfig(level=logging.DEBUG) enables debug logging.
Detailed request and response information is logged.

Key Points:

Debugging provides insights into request/response cycles.
Useful for troubleshooting and optimizing requests.

Practical Tip: Enable debug logging to diagnose issues and ensure your requests are functioning as expected.

20. Advanced Authentication Methods

Code Example:

python

Copy code

import requests from requests.auth import HTTPDigestAuth url = 'https://httpbin.org/digest-auth/auth/user/pass' response = requests.get(url, auth=HTTPDigestAuth('user', 'pass')) print(response.status_code)

Explanation:

HTTPDigestAuth provides digest authentication.
More secure than basic authentication.

Key Points:

Digest authentication adds security by hashing credentials.
Use when interacting with APIs that require enhanced security.

Practical Tip: Opt for advanced authentication methods when dealing with sensitive or secure API endpoints.

These notes provide a comprehensive guide to using the requests module in Python, covering a range of scenarios and best practices. Each example is designed to be practical and engaging, with detailed explanations to help you understand the concepts thoroughly.