Advanced Python Programming Build High Performance, Concurrent, and Multi-Threaded Apps with Python Using Proven Design Patterns

• Main Authors: Lanaro, Gabriele, Nguyen, Quan, Kasampalis, Sakis
• Format: eBook
• Language: English
• Published: Birmingham Packt Publishing, Limited 2019; Packt Publishing
• Edition: 1
• Series: Learning path
• Subjects: Application software; Python (Computer program language)
• ISBN: 9781838551216; 1838551212

Create distributed applications with clever design patterns to solve complex problemsKey FeaturesSet up and run distributed algorithms on a cluster using Dask and PySparkMaster skills to accurately implement concurrency in your codeGain practical experience of Python design patterns with real-world...

Table of Contents:

• Other interesting projects -- Summary -- Chapter 6: Implementing Concurrency -- Asynchronous programming -- Waiting for I/O -- Concurrency -- Callbacks -- Futures -- Event loops -- The asyncio framework -- Coroutines -- Converting blocking code into non-blocking code -- Reactive programming -- Observables -- Useful operators -- Hot and cold observables -- Building a CPU monitor -- Summary -- Chapter 7: Parallel Processing -- Introduction to parallel programming -- Graphic processing units -- Using multiple processes -- The Process and Pool classes -- The Executor interface -- Monte Carlo approximation of pi -- Synchronization and locks -- Parallel Cython with OpenMP -- Automatic parallelism -- Getting started with Theano -- Profiling Theano -- Tensorflow -- Running code on a GPU -- Summary -- Chapter 8: Advanced Introduction to Concurrent and Parallel Programming -- Technical requirements -- What is concurrency? -- Concurrent versus sequential -- Example 1 - checking whether a non-negative number is prime -- Concurrent versus parallel -- A quick metaphor -- Not everything should be made concurrent -- Embarrassingly parallel -- Inherently sequential -- Example 2 - inherently sequential tasks -- I/O bound -- The history, present, and future of concurrency -- The history of concurrency -- The present -- The future -- A brief overview of mastering concurrency in Python -- Why Python? -- Setting up your Python environment -- General setup -- Summary -- Questions -- Further reading -- Chapter 9: Amdahl's Law -- Technical requirements -- Amdahl's Law -- Terminology -- Formula and interpretation -- The formula for Amdahl's Law -- A quick example -- Implications -- Amdahl's Law's relationship to the law of diminishing returns -- How to simulate in Python -- Practical applications of Amdahl's Law -- Summary -- Questions -- Further reading
• Cover -- Title Page -- Copyright -- About Packt -- Contributors -- Table of Contents -- Preface -- Chapter 1: Benchmarking and Profiling -- Designing your application -- Writing tests and benchmarks -- Timing your benchmark -- Better tests and benchmarks with pytest-benchmark -- Finding bottlenecks with cProfile -- Profile line by line with line_profiler -- Optimizing our code -- The dis module -- Profiling memory usage with memory_profiler -- Summary -- Chapter 2: Pure Python Optimizations -- Useful algorithms and data structures -- Lists and deques -- Dictionaries -- Building an in-memory search index using a hash map -- Sets -- Heaps -- Tries -- Caching and memoization -- Joblib -- Comprehensions and generators -- Summary -- Chapter 3: Fast Array Operations with NumPy and Pandas -- Getting started with NumPy -- Creating arrays -- Accessing arrays -- Broadcasting -- Mathematical operations -- Calculating the norm -- Rewriting the particle simulator in NumPy -- Reaching optimal performance with numexpr -- Pandas -- Pandas fundamentals -- Indexing Series and DataFrame objects -- Database-style operations with Pandas -- Mapping -- Grouping, aggregations, and transforms -- Joining -- Summary -- Chapter 4: C Performance with Cython -- Compiling Cython extensions -- Adding static types -- Variables -- Functions -- Classes -- Sharing declarations -- Working with arrays -- C arrays and pointers -- NumPy arrays -- Typed memoryviews -- Particle simulator in Cython -- Profiling Cython -- Using Cython with Jupyter -- Summary -- Chapter 5: Exploring Compilers -- Numba -- First steps with Numba -- Type specializations -- Object mode versus native mode -- Numba and NumPy -- Universal functions with Numba -- Generalized universal functions -- JIT classes -- Limitations in Numba -- The PyPy project -- Setting up PyPy -- Running a particle simulator in PyPy
• Introductory example in Python -- An overview of the multiprocessing module -- The process class -- The Pool class -- Determining the current process, waiting, and terminating processes -- Determining the current process -- Waiting for processes -- Terminating processes -- Interprocess communication -- Message passing for a single worker -- Message passing between several workers -- Summary -- Questions -- Further reading -- Chapter 14: Reduction Operators in Processes -- Technical requirements -- The concept of reduction operators -- Properties of a reduction operator -- Examples and non-examples -- Example implementation in Python -- Real-life applications of concurrent reduction operators -- Summary -- Questions -- Further reading -- Chapter 15: Concurrent Image Processing -- Technical requirements -- Image processing fundamentals -- Python as an image processing tool -- Installing OpenCV and NumPy -- Computer image basics -- RGB values -- Pixels and image files -- Coordinates inside an image -- OpenCV API -- Image processing techniques -- Grayscaling -- Thresholding -- Applying concurrency to image processing -- Good concurrent image processing practices -- Choosing the correct way (out of many) -- Spawning an appropriate number of processes -- Processing input/output concurrently -- Summary -- Questions -- Further reading -- Chapter 16: Introduction to Asynchronous Programming -- Technical requirements -- A quick analogy -- Asynchronous versus other programming models -- Asynchronous versus synchronous programming -- Asynchronous versus threading and multiprocessing -- An example in Python -- Summary -- Questions -- Further reading -- Chapter 17: Implementing Asynchronous Programming in Python -- Technical requirements -- The asyncio module -- Coroutines, event loops, and futures -- Asyncio API -- The asyncio framework in action
• Asynchronously counting down -- A note about blocking functions -- Asynchronous prime-checking -- Improvements from Python 3.7 -- Inherently blocking tasks -- concurrent.futures as a solution for blocking tasks -- Changes in the framework -- Examples in Python -- Summary -- Questions -- Further reading -- Chapter 18: Building Communication Channels with asyncio -- Technical requirements -- The ecosystem of communication channels -- Communication protocol layers -- Asynchronous programming for communication channels -- Transports and protocols in asyncio -- The big picture of asyncio's server client -- Python example -- Starting a server -- Installing Telnet -- Simulating a connection channel -- Sending messages back to clients -- Closing the transports -- Client-side communication with aiohttp -- Installing aiohttp and aiofiles -- Fetching a website's HTML code -- Writing files asynchronously -- Summary -- Questions -- Further reading -- Chapter 19: Deadlocks -- Technical requirements -- The concept of deadlock -- The Dining Philosophers problem -- Deadlock in a concurrent system -- Python simulation -- Approaches to deadlock situations -- Implementing ranking among resources -- Ignoring locks and sharing resources -- An additional note about locks -- Concluding note on deadlock solutions -- The concept of livelock -- Summary -- Questions -- Further reading -- Chapter 20: Starvation -- Technical requirements -- The concept of starvation -- What is starvation? -- Scheduling -- Causes of starvation -- Starvation's relationship to deadlock -- The readers-writers problem -- Problem statement -- The first readers-writers problem -- The second readers-writers problem -- The third readers-writers problem -- Solutions to starvation -- Summary -- Questions -- Further reading -- Chapter 21: Race Conditions -- Technical requirements
• The concept of race conditions
• Chapter 10: Working with Threads in Python -- Technical requirements -- The concept of a thread -- Threads versus processes -- Multithreading -- An example in Python -- An overview of the threading module -- The thread module in Python 2 -- The threading module in Python 3 -- Creating a new thread in Python -- Starting a thread with the thread module -- Starting a thread with the threading module -- Synchronizing threads -- The concept of thread synchronization -- The threading.Lock class -- An example in Python -- Multithreaded priority queue -- A connection between real-life and programmatic queues -- The queue module -- Queuing in concurrent programming -- Multithreaded priority queue -- Summary -- Questions -- Further reading -- Chapter 11: Using the with Statement in Threads -- Technical requirements -- Context management -- Starting from managing files -- The with statement as a context manager -- The syntax of the with statement -- The with statement in concurrent programming -- Example of deadlock handling -- Summary -- Questions -- Further reading -- Chapter 12: Concurrent Web Requests -- Technical requirements -- The basics of web requests -- HTML -- HTTP requests -- HTTP status code -- The requests module -- Making a request in Python -- Running a ping test -- Concurrent web requests -- Spawning multiple threads -- Refactoring request logic -- The problem of timeout -- Support from httpstat.us and simulation in Python -- Timeout specifications -- Good practices in making web requests -- Consider the terms of service and data-collecting policies -- Error handling -- Update your program regularly -- Avoid making a large number of requests -- Summary -- Questions -- Further reading -- Chapter 13: Working with Processes in Python -- Technical requirements -- The concept of a process -- Processes versus threads -- Multiprocessing