mirror of
https://github.com/Sosokker/python-howto.git
synced 2025-12-18 13:34:05 +01:00
Add markdown files about Concurrency and Parallelism
This commit is contained in:
parent
1006ef8faa
commit
316c1efc84
42
asynchronous.md
Normal file
42
asynchronous.md
Normal file
@ -0,0 +1,42 @@
|
||||
## Part 2: Asynchronous Programming with asyncio
|
||||
|
||||
In this section, we'll dive into asynchronous programming using the asyncio module, enabling concurrent execution without relying on multiple threads.
|
||||
|
||||
### 1. Introduction to Asynchronous Programming
|
||||
|
||||
Asynchronous programming allows non-blocking execution of tasks, making efficient use of system resources. The asyncio module provides a framework for working with asynchronous operations. In this simple example, we define an asynchronous function main that prints "Hello," waits for a second using await asyncio.sleep(1), and then prints "World."
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
|
||||
async def main():
|
||||
print("Hello")
|
||||
await asyncio.sleep(1)
|
||||
print("World")
|
||||
|
||||
asyncio.run(main())
|
||||
|
||||
```
|
||||
|
||||
### 2. Getting Started with asyncio
|
||||
|
||||
To work with asyncio, we can use the async and await keywords. In this example, we define an asynchronous function fetch_data that simulates fetching data from a URL asynchronously. The asyncio.gather function is used to run multiple asynchronous tasks concurrently and gather their results.
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
|
||||
async def fetch_data(url):
|
||||
# Simulate fetching data from a URL
|
||||
await asyncio.sleep(2)
|
||||
return f"Data from {url}"
|
||||
|
||||
async def main():
|
||||
tasks = [fetch_data("url1"), fetch_data("url2"), fetch_data("url3")]
|
||||
results = await asyncio.gather(*tasks)
|
||||
print(results)
|
||||
|
||||
asyncio.run(main())
|
||||
|
||||
```
|
||||
|
||||
[Continue to Part 3: Parallelism with multiprocessing](multiprocessing.md)
|
||||
36
combining.md
Normal file
36
combining.md
Normal file
@ -0,0 +1,36 @@
|
||||
## Part 4: Combining Concurrency and Parallelism
|
||||
|
||||
In this final section, we'll discuss how to leverage both concurrency and parallelism to optimize Python applications.
|
||||
### 1. Concurrency vs. Parallelism: When to Use Which?
|
||||
|
||||
Concurrency is suitable for I/O-bound tasks that involve waiting, such as network requests. Parallelism is ideal for CPU-bound tasks that can be divided into smaller units of work. In some cases, combining both approaches can provide the best performance.
|
||||
|
||||
### 2. Real-world Application: Web Scraping with Concurrency and Parallelism
|
||||
|
||||
As a practical example, let's consider web scraping. We can use asynchronous programming (`asyncio`) to handle multiple requests concurrently, and then use parallelism (`multiprocessing`) to process the scraped data in parallel. This combination can significantly speed up the entire scraping process.
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import aiohttp
|
||||
import multiprocessing
|
||||
|
||||
async def fetch_url(session, url):
|
||||
async with session.get(url) as response:
|
||||
return await response.text()
|
||||
|
||||
def process_data(data):
|
||||
# Process the scraped data
|
||||
pass
|
||||
|
||||
async def main():
|
||||
urls = ["url1", "url2", "url3"]
|
||||
async with aiohttp.ClientSession() as session:
|
||||
tasks = [fetch_url(session, url) for url in urls]
|
||||
responses = await asyncio.gather(*tasks)
|
||||
|
||||
with multiprocessing.Pool() as pool:
|
||||
processed_data = pool.map(process_data, responses)
|
||||
|
||||
print("Processed data:", processed_data)
|
||||
|
||||
asyncio.run(main())
|
||||
79
introduction.md
Normal file
79
introduction.md
Normal file
@ -0,0 +1,79 @@
|
||||
# Concurrency and Parallelism in Python
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Concurrency and Parallelism in Python](#concurrency-and-parallelism-in-python)
|
||||
- [Table of Contents](#table-of-contents)
|
||||
- [Part 1: Introduction to Concurrency](#part-1-introduction-to-concurrency)
|
||||
- [1. Global Interpreter Lock (GIL) Explained](#1-global-interpreter-lock-gil-explained)
|
||||
- [2. Threading in Python](#2-threading-in-python)
|
||||
|
||||
---
|
||||
|
||||
## Part 1: Introduction to Concurrency
|
||||
|
||||
In this section, we'll explore the fundamental concepts of concurrency in Python, including the Global Interpreter Lock (GIL) and the `threading` module.
|
||||
|
||||
### 1. Global Interpreter Lock (GIL) Explained
|
||||
|
||||
The Global Interpreter Lock (GIL) is a mutex that prevents multiple native threads from executing Python bytecodes simultaneously in a single process. While it ensures memory safety, it can limit the parallelism of CPU-bound tasks. In this example, we'll create two threads to perform counting concurrently, but due to the GIL, they won't achieve true parallelism.
|
||||
|
||||
```python
|
||||
import threading
|
||||
|
||||
def count_up():
|
||||
for _ in range(1000000):
|
||||
pass
|
||||
|
||||
def count_down():
|
||||
for _ in range(1000000):
|
||||
pass
|
||||
|
||||
thread1 = threading.Thread(target=count_up)
|
||||
thread2 = threading.Thread(target=count_down)
|
||||
|
||||
thread1.start()
|
||||
thread2.start()
|
||||
|
||||
thread1.join()
|
||||
thread2.join()
|
||||
|
||||
print("Counting done!")
|
||||
```
|
||||
|
||||
### 2. Threading in Python
|
||||
|
||||
The threading module allows us to work with threads in Python. In this example, we're using a lock (threading.Lock()) to synchronize access to a shared variable (x) across multiple threads. This ensures that only one thread modifies x at a time, preventing data corruption.
|
||||
|
||||
```python
|
||||
|
||||
import threading
|
||||
|
||||
x = 0
|
||||
lock = threading.Lock()
|
||||
|
||||
def increment():
|
||||
global x
|
||||
for _ in range(1000000):
|
||||
with lock:
|
||||
x += 1
|
||||
|
||||
def decrement():
|
||||
global x
|
||||
for _ in range(1000000):
|
||||
with lock:
|
||||
x -= 1
|
||||
|
||||
thread1 = threading.Thread(target=increment)
|
||||
thread2 = threading.Thread(target=decrement)
|
||||
|
||||
thread1.start()
|
||||
thread2.start()
|
||||
|
||||
thread1.join()
|
||||
thread2.join()
|
||||
|
||||
print("Result:", x)
|
||||
``````
|
||||
|
||||
[Continue to Part 2: Asynchronous Programming with asyncio](asynchronous.md)
|
||||
46
multiprocessing.md
Normal file
46
multiprocessing.md
Normal file
@ -0,0 +1,46 @@
|
||||
## Part 3: Parallelism with multiprocessing
|
||||
|
||||
In this section, we'll explore parallelism in Python using the multiprocessing module, which allows for concurrent execution using multiple processes.
|
||||
### 1. Introduction to Parallelism
|
||||
|
||||
Parallelism involves executing multiple tasks simultaneously, utilizing multiple CPU cores. The multiprocessing module provides a way to achieve true parallelism by creating separate processes. In this example, we define a function square and use the multiprocessing.Pool to parallelize its execution across multiple processes.
|
||||
|
||||
```python
|
||||
import multiprocessing
|
||||
|
||||
def square(number):
|
||||
return number * number
|
||||
|
||||
if __name__ == "__main__":
|
||||
numbers = [1, 2, 3, 4, 5]
|
||||
with multiprocessing.Pool() as pool:
|
||||
results = pool.map(square, numbers)
|
||||
print(results)
|
||||
|
||||
```
|
||||
|
||||
### 2. Using multiprocessing
|
||||
|
||||
The multiprocessing module allows us to create and manage processes in Python. In this example, we define a function worker_function that each process will execute. We create multiple processes and start them using the start method. Finally, we wait for all processes to finish using the join method.
|
||||
|
||||
```python
|
||||
import multiprocessing
|
||||
|
||||
def worker_function(number):
|
||||
print(f"Worker process {number} is executing")
|
||||
|
||||
if __name__ == "__main__":
|
||||
processes = []
|
||||
for i in range(4):
|
||||
process = multiprocessing.Process(target=worker_function, args=(i,))
|
||||
processes.append(process)
|
||||
process.start()
|
||||
|
||||
for process in processes:
|
||||
process.join()
|
||||
|
||||
print("All processes have finished")
|
||||
|
||||
```
|
||||
|
||||
[Continue to Part 4: Combining Concurrency and Parallelism](combining.md)
|
||||
Loading…
Reference in New Issue
Block a user