Python for DevOps
You're viewing a free preview. Sign in free to read every script.
Why this exists A script that reads a whole log into a list works fine on a 5 MB test file and gets killed on a 5 GB production one. The habit that fixes it is to process data one item at a time , and Python builds that…
Python for DevOps: Iterators, Generators and itertools for Data Bigger Than Memory
Process logs, API pages and query results one item at a time: how for-loops really work, generators with yield, lazy pipelines, generator expressions, yield from, single-use generators, the itertools toolbox (batched, pairwise, islice, chain), and reading in chunks.
Sign in free to read this script, along with the rest of the library.
Sign in free