Distributed File Storage

Architecture

Clients never talk to Dstores directly except to push/pull file bytes once the Controller has told them which ports to use. Every metadata decision (who stores what, whether a file exists yet, whether a store finished) lives in the Controller's Index, guarded by a single lock.

The protocol

Everything is line-delimited text over a persistent socket, which made the whole thing debuggable with nc before any Java existed. A store looks like this:

Client     → Controller   STORE <filename> <filesize>
Controller → Client       STORE_TO <port1> <port2> <port3>
Client     → Dstores      (raw file bytes, direct socket per port)
Dstore     → Controller   STORE_ACK <filename>   (×R)
Controller → Client       STORE_COMPLETE

The Controller picks which R Dstores get a given file by counting, for every connected Dstore, how many files it currently holds, then handing the write to whichever ones hold the fewest. It's a greedy load-balance, recomputed on every STORE rather than maintained incrementally: simple enough to reason about, expensive enough that it would need rethinking well before Dstore counts got large.

Waiting for acknowledgement uses a CountDownLatch per filename, initialized to the replication factor. Each STORE_ACK the Controller receives counts it down; a background thread blocks on latch.await(timeout, ...) and either marks the file STORE_COMPLETE or, on timeout, rolls the entry back out of the index. That rollback matters: without it, a slow or dead Dstore would leave a file permanently wedged in STORE_IN_PROGRESS, invisible to LIST but blocking anyone from retrying the same filename.

The bug: two STOREs, one filename

The index has to answer two questions atomically: "does this file already exist?" and, if not, "reserve this name for me." Early on, the existence check and the index.addFile(...) call were two separate method calls with nothing serializing them against a second thread doing the same thing.

With one client that's invisible. With two clients racing to store the same filename (which happens constantly under the test harness this coursework was marked against, since it fires concurrent operations at the Controller on purpose) both threads could read "file does not exist" before either had written its reservation, and both would proceed to hand out STORE_TO instructions. Two Dstore sets would end up writing to the same filename, and whichever STORE_COMPLETE landed last would silently win, with no error surfaced to the client that lost.

The fix was to wrap the read-check-write as a single critical section under the same lock used everywhere else the index is touched.

synchronized (index) {
    if (index.getFileInformation(filename) != null) {
        sendMessage(controllerSocket, "ERROR_FILE_ALREADY_EXISTS");
        return;
    }
    index.addFile(filename, filesize, selectDstores());
}

The lock only has to cover the check-and-reserve, not the whole store operation: once the filename is reserved, the rest (sending STORE_TO, waiting on the latch) can run unsynchronized, because no other thread can get past the check for that filename anymore. Holding the lock any longer than that would have serialized unrelated stores against each other for no reason.

What I'd change

The load-balancing recompute is a bottleneck: it's O(Dstores × files) on every single store. A running per-Dstore file count, updated incrementally on store/remove instead of rebuilt from the index each time, would make Dstore selection O(Dstores log Dstores) instead. It wasn't worth building for a coursework-scale deployment.

Result

83/100: full protocol correctness and concurrency, marked down on some edge-case error handling around Dstore rebalancing (didn't fully implement correct rebalancing logic).