Chapter 149: Offsets and Locks

Two forty in the morning. The dorm building had gone completely quiet. Only the water dispenser at the far end of the corridor occasionally made a low gurgling sound as it heated. Lin Chen stared at the architecture diagram on the screen. The cursor blinked over the “inter-process communication” module like a heartbeat.

In Python 2.4, the GIL was a wall he could not go around. os.fork could duplicate processes, but pipe-based communication between parent and child processes would block easily once the data volume grew large. He needed to cut two million lines of logs into chunks of fifty thousand lines each, let child processes parse them independently, and send the results back. The memory peak had to stay under control. He opened a new text file and began sketching the data flow. The parent process would distribute file offsets. Child processes would read, run regex matches, classify status codes, and write to temporary files. At the end, the parent process would merge the output in order. No shared memory—that would avoid the compatibility traps of the multiprocessing module under 2.4.

He typed the first draft of the pseudocode. def worker(pipe, offset, chunk_size): The logic was clean, but the devil lived in the details. The default pipe buffer was only 4 KB; under high concurrency, writes would hang. He would have to add non-blocking writes in the child process, or switch to direct mapping through file descriptors. He pulled out the Linux Systems Programming manual the training camp had issued them, the paper copy with a cracked spine. Chapter 14, Pipes and FIFOs. He read it word by word, his fingers unconsciously tapping the desktop. His left foot twitched again inside his slipper, the muscle stiff as a block of weathered wood. He stood up, braced himself on the edge of the desk, and slowly took two steps. As blood flowed back, a dense tingling pain spread through him. He sat down and kept going.

Replace the pipe with temporary files plus file locks. fcntl.flock. Supported in 2.4. Child processes would write into temporary files, and the parent process would read and merge them in sequence. Memory usage would drop from linear growth to something closer to constant space. He opened a terminal, used dd to generate a simulated log file with five hundred thousand lines, and ran the prototype once. The terminal scrolled; CPU usage shot instantly to eighty-five percent. Three seconds later, the result came out. He pulled up the system monitor. Peak memory usage: 21.4 MB. Runtime: 4.1 seconds. By linear estimate, two million lines would finish in around sixteen seconds, and memory would stay under 28 MB. Far below the red line of 50 MB. But the latency still wasn’t good enough. Engineer Li’s requirement was three hundred milliseconds per ten thousand lines, which meant total runtime had to stay within sixty seconds. Sixteen seconds looked safe, but a real production environment included network I/O and disk reads and writes. He needed more margin.

He stared at the logs streaming past in the terminal. The bottleneck was file-lock contention. Four child processes were all trying to grab the lock at the same time; context switches were burning too many CPU cycles. He needed lower-level scheduling. Linux 2.6 supported epoll, but the development machine assigned by the training camp was an aging Red Hat 4 box still running a 2.4 kernel. He could not assume an environment upgrade. He flipped to Chapter 22 of the manual, I/O multiplexing. select and poll. Compatible with 2.4. He decided to replace the polling logic around the file locks with poll, and change the temporary files to mmap memory mapping. Fewer system calls.

The clock in the lower-right corner of the screen flipped to 05:15. Outside, the sky had turned a pale gray-white. The clouds were thick, like waterlogged cotton. He saved the document as arch_v1_draft.txt. His phone screen lit up. A text from Wang Guiying: “The electricity’s fixed. The village doctor says Xiaoman’s medicine has to be switched to a different brand—the old one’s out of stock. The new brand costs twelve yuan more per box and has to be paid up front. You’re busy over there, no need to come back.”

Lin Chen stared at the words twelve yuan more. He opened his online banking page. Balance: 1,112.4 yuan. There were a little over forty yuan left on his meal card at the training camp. He transferred three hundred yuan back home. Note: medicine. The transfer fee deducted another two yuan. His balance became 810.4. He closed the page. In the county seat, twelve yuan meant two jin of pork. In Qingshi Village, it meant half a month’s worth of salt. For him, it was a fault-tolerance branch in a script, a single try-except. He couldn’t save that money, and he couldn’t spend recklessly either. He had to get the project bonus. Difficulty-A projects at the training camp came with an extra performance pool from the partner company. Engineer Li hadn’t said so explicitly, but people on the edge of the list could only turn things around that way. Rankings were fixed. Projects were alive.

Eight in the morning. The training camp computer lab. The outside unit of the air conditioner droned beyond the window. Lin Chen inserted his USB drive into a shared computer and submitted the architecture document through the internal network. The system displayed: “Received.” He sat down in front of a spare machine in the corner and waited for a response. The sound of keyboards was sparse. Several students from key provincial universities were discussing a weekend dinner gathering. Lin Chen did not look up. He kept his eyes on the time in the lower-right corner of the screen. At 9:10, a message from Engineer Li popped up on the intranet.

“I’ve read the document. The direction is right, but file locks will contend under concurrent writes. Why didn’t you use epoll in Linux 2.6?”

Lin Chen rested his fingers on the keyboard. The 2.4 kernel didn’t support epoll, and the servers assigned by the training camp were old Red Hat 4 machines. He couldn’t assume an upgrade. He replied: “The production server kernel is 2.6.18, but the development machine assigned by the training camp is 2.4. For compatibility, I’m temporarily using poll polling plus file locks. If a kernel upgrade is allowed, this can be replaced with edge-triggered epoll to reduce context switching.”

Engineer Li replied almost immediately: “The environment can be changed. Wednesday, two p.m., bring me a working prototype. Run two million lines. Keep latency under three hundred milliseconds. If you can’t make it, someone else gets the project.”

The message froze on the screen. Lin Chen knew it was an ultimatum. He pulled out the USB drive and stood up. When his left foot touched the floor, his knee gave way for an instant; he quickly caught himself with his right hand on the corner of the desk. A dull pain climbed from his calf through the gaps in his bones, like rusted gears grinding against each other by force. He walked slowly back to the dorm. In the afternoon, he would have to rewrite the core parsing module—replace poll with something lower-level, or simply write the extension in C. He needed to look up the Python 2.4 C API documentation. The computer section on the third floor of the library had a copy of Python Source Code Analysis, but the borrowing record showed that a graduate student from Provincial Polytechnic had checked it out last week.

Standing downstairs beneath the dormitory, he looked up at the third-floor window. The wind snapped the shirts hanging from the laundry line. He pulled out his phone and sent a text to an old classmate at Provincial Polytechnic: “Brother, can you return that copy of Python Source Code Analysis from the library before Wednesday? I need it urgently.” The message sent successfully. He leaned against the wall and closed his eyes. Wednesday, two p.m. Forty-eight hours left. The pain under his foot was sharp and distinct, like the ticking of a second hand. He opened his eyes, turned, and walked toward the cafeteria. He had to eat first. He was going to write a C extension this afternoon.

The buns in the cafeteria had already gone cold, their skins a little tough. He bought two and swallowed them down with the free soup. Back in the dorm, he pulled open the drawer and found the blank notebook. He turned to a fresh page and wrote: 1. Replace file locks with mmap memory mapping; 2. Wrap the regex-matching core in a C extension; 3. Request a 2.6 kernel environment; 4. Reserve room for the medicine expense gap.

The tip of the pen paused. He drew a circle around the fourth item. Being on the edge of the list meant he could be squeezed out at any time. He needed a harder result. The training camp’s rules were explicit: theory exams counted for thirty percent, practical work forty percent, enterprise-linked projects thirty percent. The first two were already fixed. The only part left that could overturn the ranking was the last one. Difficulty-A projects were something no one had dared take on before. If you took one, you either got full marks or were eliminated outright. There was no middle ground.

He opened the system monitor and pulled up the historical memory and CPU curves. Five hundred thousand lines, 21.4 MB, 4.1 seconds. At two million lines, linear scaling would push memory past 28 MB and runtime past sixteen seconds. Production wouldn’t allow it. He had to cut at the architectural level. Sharded reads, a multiprocess pool, shared-memory queues. Python 2.4 had no ready-made wheels for any of it. He would have to build them himself.

He typed the first line of a comment: # Goal: single-machine concurrent parsing, peak memory <50 MB, latency <300 ms / 10k lines.

His fingers settled on the keyboard. The rhythm was slow, but steady. Beneath the blanket, his left foot twitched slightly, and he did not move. The cold glow of the screen lit his face. In his pupils there was nothing but the characters scrolling downward.

Tomorrow he had to hand in the architecture draft.

He could not break.