Chapter 144: Processes and Headroom

The cursor in the terminal blinked twice, and Lin Chen pressed Enter.

import multiprocessing

At once, the fan let out a heavy, muffled hum. Dust had built up in the cooling vents of the old ThinkPad, and heat began to creep up along the edges of the keyboard until his fingertips tingled from the burn. He kept his eyes on the screen as he split the 50MB CSV file into four chunks by line and assigned them to four child processes. The theory was simple enough: the main process reads the file, each subprocess handles deduplication and alignment on its own portion, and everything is merged at the end. But reality was less cooperative. Memory usage shot up to 87 percent in an instant, the hard drive light flashed wildly, and the progress bar froze at 12 percent.

He hit Ctrl+C and killed the job. The terminal spat out MemoryError.

Lin Chen leaned back in his chair. The muscles in his left leg twitched without warning. He reached down and pressed a hand hard against his knee, his knuckles whitening with the force. The spasm lasted more than ten seconds before it finally eased. There were only three ibuprofen tablets left in the drawer. He counted them, then left them where they were. Pain was normal. The medicine had to be saved for the times he truly could not move.

He opened his scratch paper again.

Multiprocessing was not as simple as hacking a task into pieces and calling it done. A CSV was plain text with no index; if he split it by byte position, he could slice a row in half and throw the parsing out of alignment. Duplicate primary keys required a global view. If each child process fought its own little war, then duplicates spanning two chunks would inevitably be missed in the final merge. He drew out a flowchart: chunked reading → local deduplication → write temporary files → second merge in the main process → final output. The logic held. But the I/O cost would multiply. His drive was still a mechanical hard disk, reading and writing only a few dozen megabytes per second. Frequent access to small files would make the drive head thrash back and forth and kill the runtime outright.

Footsteps sounded outside the main room.

Wang Guiying came in carrying an enamel mug. Two slices of dried ginger and a few pieces of rock sugar sat at the bottom. “Have some water.” She set it on the corner of the desk without looking at the screen. “There’s sweet potato porridge simmering on the stove. If you get hungry, serve yourself.”

“Mm,” Lin Chen answered.

She stood there for two seconds. Her gaze dropped to his faintly trembling left leg. She did not ask anything. She simply turned, pulled the door shut behind her, and left. The hinge gave a soft creak. The room fell quiet again, filled only with the fan’s hum and the tapping of keys.

Lin Chen picked up the enamel mug. The water was the right temperature. The bite of ginger mixed with the sweetness of sugar and slid down his throat, leaving a little warmth in his stomach. He sat up straight again and tore the code apart to start over.

No more multiprocessing.Pool.

He switched to a Queue for buffering. The main process would read a fixed number of lines at a time—5,000—and push them into the queue. The child processes would pull data from the queue, build an in-memory dictionary for local deduplication, then write the results back to another queue. To avoid another memory blowout, he stripped the dictionary down to the essentials, keeping only the primary key and the latest timestamp. As for duplicates spanning chunks, he would leave those for the final stage and solve them with an external sort.

He gave up on pandas; it was far too hungry for memory. Instead, he switched to Python’s built-in csv module, parsing line by line and handling field splitting and type conversion by hand. The code tripled in size, but the memory footprint dropped into a range he could control.

By the time he finished typing, it was half past one in the morning.

He saved the file and ran it.

Four lines of Process started appeared in the terminal. The fan spun faster, but memory usage held steady at around 62 percent. The progress bar finally began to crawl forward again: 1%... 3%... 8%...

Lin Chen did not sit there waiting.

He opened Python Data Analysis in Practice and flipped to the chapter on “External Sorting and Merge.” The algorithm in the book was written in pseudocode. Following the logic, he worked out the depth of the merge tree on scratch paper. Five hundred thousand rows of data. After chunking, it would take three rounds of merging. Each round meant reading and writing to disk. Could his hard drive handle that? He estimated the time. If everything went smoothly, maybe forty minutes. If it did not—if the drive overheated and throttled down—it could drag on for two hours.

At 2:10 a.m., the progress bar stopped at 47 percent.

The screen dimmed for a split second.

Lin Chen’s chest tightened. Not a power outage. The system had triggered its protection mechanism. The CPU had downclocked. The fan grew quieter. The progress bar stalled completely.

He stood up, pulled an old cardboard box from under the bed, and dug out a small brush and a can of electronics cleaner. He unscrewed the back cover of the laptop, moving carefully so he would not wake Xiaoman in the next room. The cooling fins were matted with a layer of gray-black dust, and the thermal paste had dried and cracked into powder. He cleaned it out bit by bit with the brush, sprayed the cleaner, and wiped it down with cotton swabs. Then he reassembled the cover, tightened the screws, and powered it back on.

The progress bar jumped from 47 percent to 48 percent.

The fan roared back to life.

Lin Chen sat down again on the bamboo chair and rubbed the ache from the back of his neck. He remembered something Zhou Yan had written on his blog: “Engineering isn’t getting the code to run. It’s managing exceptions.”

Exceptions were not only garbled text in the data. They were aging hardware, stifling heat, the limits of the body itself. He had to factor in all of those variables.

He got up and paced slowly through the main room. When his left foot felt weak on the ground, he shifted his weight to the right and walked three steps, then stopped and stretched his calf. Again and again. His body could not be allowed to lock up. Neither could the code.

At 3:40, the progress bar reached 92 percent.

The cramp in his left leg struck again, harder than before. He bit down on his lower lip and made no sound. His fingers curled unconsciously against the tabletop, nails digging into his palm. He drew a deep breath and forced his attention back to the screen. In the background, the log file kept scrolling:

[INFO] Chunk 12 merged. Duplicates removed: 402. [INFO] Timestamp aligned. Missing values filled with median. [WARN] Row 38412: Format mismatch, fallback to string parse.

He stared at those lines. Every one of them was a rule. Rules he had written himself.

A strange calm settled over him.

Poverty, his injured leg, the gap in information—things that had once pressed down on him until he could hardly breathe—had, in this moment, all been broken down into executable steps. He did not need to leap across class lines overnight. He only needed to make these five hundred thousand rows of data line up according to his logic.

At 4:15, the terminal displayed: Done. Total rows: 489,312. Output saved.

Lin Chen closed the terminal and opened the output file.

He spot-checked twenty rows at random. The primary keys were unique. The timestamps were continuous. The cross-table relationship fields were intact. No garbled text. No broken lines. He ran the validation script once more. The error rate came back zero.

He let out a long breath and leaned back. The bamboo chair gave a faint groan beneath him.

Outside the window, the sky had already turned a pale gray-white. The cicadas had not begun yet. There was only the dull sound of an early freight truck passing along the county road in the distance.

He opened his email and packaged the processed CSV file, the log file, and a short explanation of the logic—less than eight hundred words. He named the attachment:

Lin Chen_Baseline Architecture Assessment_Data Alignment and Deduplication Notes.pdf

Then he clicked send.

Recipient: zhouyan_lab@...

Sent successfully.

He closed the laptop.

From the drawer, he took out a hardbound notebook and turned to a fresh page. He wrote:

2012.08.15. Assessment task delivered. Multiprocessing in practice. I/O bottlenecks and external sorting. Hardware maintenance log. Peak memory usage 62%, total runtime 3h12m.

The handwriting was neat, with no extra emotion in it.

He stood up, went to the water jar, scooped up a dipperful, and splashed his face. The cold made his skin tighten and drove off some of the drowsiness. When he returned to the main room, he packed his laptop, charger, resume, ibuprofen, and a change of clothes into a canvas bag. Halfway through pulling the zipper shut, he paused, reached beneath the bed frame, and brought out a tin box.

Inside were the registration notice for the Provincial Institute of Technology training camp and a long-distance bus ticket to the provincial capital. He had asked an acquaintance in town to buy it for him the day before. Hard seat. Forty-two yuan.

His remaining balance was 1,377.3 yuan. After deducting the fare and miscellaneous expenses, he would still have a little over nine hundred left. Enough to last until the training camp ended. Enough to carry him into the next freelance job. He entered it in the ledger, crossed out the old figure, and wrote in the new one.

Wang Guiying pushed the door open and came in holding a flatbread wrapped in oiled paper. “Eat it on the road.” She tucked it into the side pocket of his bag, then paused. “When you get to the provincial capital, don’t pinch pennies too hard. If you should eat, then eat.”

“I know.” Lin Chen pulled the zipper closed. “Mom, I’ll send money back for Xiaoman’s medicine at the start of next month.”

Wang Guiying nodded and said nothing more. She turned and went to light the stove in the kitchen. Firewood crackled. Blue smoke drifted up the chimney.

Lin Chen hoisted the bag onto his shoulder.

His left foot still felt weak when it touched the ground, but his center of gravity had already learned to shift to the right by habit. He stepped out of the main room. Dew dampened the edges of his cloth shoes. The stone path in the yard lay under a veil of morning mist, and the ridge line of the mountains beyond was blurred and indistinct.

He walked slowly.

Every step landed solidly.

The long-distance bus at the town station left at six. He needed to arrive half an hour early to queue for ticket inspection.

His phone vibrated in his pocket.

He took it out. Not Old Zhao. Not Zhou Yan.

It was an automated text message from the Provincial Institute of Technology academic affairs system:

“The Low-Level Architecture Group of the training camp will conduct its first code review at 9:00 a.m. on August 15. Please bring your personal development environment and project documentation. Late arrivals will be marked absent. Also: please read the attached ‘Fundamental Concepts of Distributed Systems’ in advance. Topics may appear in the review.”

The attachment was a 15MB PDF.

Lin Chen opened it and scanned the table of contents: CAP theorem, consistent hashing, the Raft protocol, message queues.

He put the phone away and looked up in the direction of the station.

The fog was already lifting. Wind blew through the mountain pass, carrying the smell of asphalt and diesel.

He quickened his pace.

When his left foot came down, it hurt. When his right leg followed, it held firm.

The road ahead was still long.

But a bus would not wait.