Chapter 236: The Edge of VRAM

The cursor blinked in the terminal like a second hand. Lin Chen stared at the dependency tree on the screen and typed pip freeze > requirements.txt. In an offline environment, every wheel had to be forged by hand. He couldn't count on pip install; he had to package PyTorch, the CUDA toolchain, even the underlying C++ compilation libraries into offline .whl files.

Su Man was running quantization scripts at the workstation next door. The keyboard clatter was dense, broken now and then by a pause, then the scroll of the mouse wheel. Neither of them spoke. The office air was filled with the low-frequency hum of server fans and the dehumidifier. Outside the window, the clouds had gathered again, and the sky had gone dark earlier than usual.

Lin Chen peeled the pain-relief patch off his left foot. The edges had already curled, leaving a faint trace of adhesive. He replaced it with a fresh one. The cold mint seeped into his skin, temporarily suppressing the dull ache deep in his ankle. He flexed his toes, making sure they still bent, and only then shifted his weight back into the chair. The spring in the backrest gave a faint creak. He was used to that sound. It was like the beam of an old house, reminding him he was still awake.

requirements.txt finished generating. He began writing the offline loading script. The hospital's test machine would most likely be running Windows Server 2016, with graphics drivers stuck at CUDA 10.2. That meant he couldn't use the latest PyTorch 2.0 directly; he had to roll back to 1.13 and manually compile some of the operators. He opened his notebook of mistakes, flipped to a blank page, and wrote: Environment isolation: do not depend on system PATH; use full relative paths. Dependency conflicts: isolate with venv; strip redundant C++ runtimes during packaging.

Su Man pushed the door open and came in holding a printout of the quantization logs. “INT8 is down to 3.8 gigs. But accuracy dropped by 1.7 points. Clinical alignment recall could get stuck below 85%.” She set the logs on Lin Chen's desk and tapped the last line with her fingertip. “What the director wants is treatment efficacy, not just something that runs. If the drop is too big, the hospital affairs committee will reject it outright.”

Lin Chen glanced over the numbers. A loss of 1.7 points could be ignored in the lab, but in clinical procurement it was the line between life and death. He was silent for a few seconds, then opened the terminal and pulled up the model architecture diagram. “Prune the attention heads in the last two layers. Use dynamic quantization instead of static quantization, and recalibrate the weights at inference time. It'll take another 200 megabytes of VRAM, but we can claw back 0.8 points of accuracy.”

“And the risk?” Su Man asked.

“Latency goes up by 15 milliseconds. The hospital intranet has enough bandwidth. The bottleneck on the terminal side is the GPU, not the CPU. We can gamble on that.” Lin Chen hit Enter and started modifying the configuration parameters.

Su Man didn't ask anything more. She turned and went back to her station, reopening her code editor. By the third year of the startup, they were long past arguing over the “ideal solution.” Reality cared only about results, not process. If it ran, if it shipped, if it passed review, then it was a good solution.

At four in the afternoon, Lin Chen's stomach began to cramp. He pulled open a drawer, fished out half a packet of soda crackers, chewed two dry bites, couldn't swallow them, and washed them down with half a bottle of mineral water. His phone screen lit up. It was a WeChat message from the caregiver: Xiaoman is stable today. She had an EEG this afternoon. The doctor said one round of sedatives can be reduced. There's 1,200 left on deposit, enough to last through Friday.

Lin Chen stared at the line of text, his finger hovering over the screen. He replied: Got it. The remaining payment will come through before Friday. After sending it, he turned the phone face down on the desk. Twelve hundred yuan, plus the eight hundred and some left on his card, was just enough to cover the cloud server bill and Su Man's base salary. If Tuesday's hospital affairs committee didn't approve them, then next month's wages and medicine costs would both break. He had no way back. All he could do was keep pushing the progress bar forward.

At eight that night, the offline package finished building. Lin Chen copied the files onto a portable hard drive and plugged it into the test machine. It was an old laptop he had picked up at the secondhand market for three hundred yuan, with a GTX 1060 3G card whose VRAM sat exactly on the edge. He launched the script and watched the VRAM usage.

The progress bar reached 60 percent and suddenly froze. The terminal threw up a red error: CUDA out of memory. Tried to allocate 256.00 MiB.

Lin Chen frowned. He checked the logs and found that when the data preprocessing module loaded anonymized medical records, it was reading all fifty files into memory at once. The VRAM had blown out.

He killed the script and reopened the code. He couldn't change the model, so he could only change the data flow. He reduced DataLoader's num_workers from 4 to 1, cut batch_size from 8 to 2, and added gradient accumulation. The VRAM usage curve smoothed out again, but the inference speed visibly slowed.

“Can we live with it?” Su Man came over and looked at the runtime statistics on the screen.

“Per-item inference goes from 120 milliseconds to 180. Running all fifty will take about nine minutes,” Lin Chen said. “The hospital affairs committee usually gives fifteen minutes for the demo. That's enough.”

Su Man nodded and said nothing. She pulled over a chair and sat down, starting to write the frontend for the demonstration. It didn't need a flashy UI, just an input box, a run button, and a text box for real-time output logs. The simpler it was, the less likely it was to bug out in an offline environment.

At one in the morning, the test machine's fan was roaring. Lin Chen leaned back in his chair with his eyes closed. The pain in his left foot had become clearer now that he was still, like a fine needle slowly twisting between the bones. He didn't move; he only adjusted his breathing. In his head he ran through tomorrow's process: bring the hard drive, bring the USB stick, bring a backup Ethernet cable, bring a printed operating manual. Security at the IT department was strict. Phones weren't allowed into the intranet zone. All logs had to be captured locally and exported on the spot after the demonstration.

He opened his eyes, picked up a pen, and wrote in his notebook of mistakes: Item 236: Fault tolerance in offline deployment. Risk: old GPUs have fragile VRAM margins; one-shot loading in the data flow causes OOM; the demo environment is uncontrollable. Countermeasure: degrade the data loading strategy, add VRAM monitoring thresholds, reserve a CPU fallback path. Execution: packaging complete, on-site verification tomorrow. Don't trust luck; leave margin.

He closed the notebook. Then he got to his feet and went to the washroom, splashing his face with cold water. The man in the mirror had deep-set eye sockets and bluish stubble, but his gaze was steady. He knew that technology had never been magic. It was certainty, piled up from countless rounds of trial and error.

When he returned to his desk, his phone vibrated. It wasn't the caregiver. It was the hospital IT department's landline number. Lin Chen answered.

“Mr. Lin, the terminal configuration sheet for Tuesday afternoon has been sent to your email.” The voice on the other end was flat with bureaucratic routine. “One more reminder: that machine is isolated on the intranet, and the USB ports have been physically blocked. Your data and scripts can only be copied in using the IT department's dedicated intranet USB drive. Once the demonstration is over, the drive will be formatted on the spot. Don't bring the wrong format.”

Lin Chen tightened his grip on the phone. “Understood. We'll convert the scripts into a single .py file ahead of time and package all dependencies into one zip.”

“Good. Don't be late.” The line went dead.

Lin Chen opened his email and downloaded the configuration sheet. The screen lit up, and he checked it line by line. CPU: i5-8500. Memory: 16G. Graphics card: GTX 1060 3G. System: Win10 LTSC. It was roughly what he had expected. But the final note made his fingers pause slightly: All permissions for installing external drivers have been disabled in the intranet environment. Only the system's built-in basic components are supported.

That meant that if their offline package included any low-level driver that required administrator privileges, the demonstration would error out immediately. He had to either statically compile the CUDA runtime all the way into the Python environment, or switch to a degraded pure-CPU inference plan.

He checked the time. It was 1:20 a.m. There were less than forty-eight hours left until Tuesday at two in the afternoon.

Lin Chen opened the drawer, found the last roll of electrical tape, and wrapped the connector of the portable hard drive twice. Then he reopened the terminal and typed conda activate local_demo.

The cursor on the screen began blinking again. He drew a deep breath, and his fingertips came down. The next line of code was already written. Outside the window, the wind moved through the gaps between the buildings with a low moan. In the office, dust drifted slowly downward in the faint glow of the screen. He knew tomorrow would not be easy, but there was no road except the one you walked step by step.