h1

Java I/O Concurrency

December 2, 2008

One of the requirements for the current project we are working on was to take a file name as input, execute an external script which would convert the file into another format and then return the contents of the output file. Once a file had been converted, subsequent invocations for the same file would just return the previously converted file.

If our application was to run on a single JVM, concurrency control could be performed in Java itself. A very efficient way of achieving this can be ripped off from Java Concurrency In Practice:

	Future<Boolean> future = fileCopiesInProgress.get(toFileLocation);
	if (future == null) {
	    Callable<Boolean> callable = new Callable<Boolean>() {
		public Boolean call() throws Exception {
		    return fileCopier.doFileCopy(fromFileLocation, toFileLocation);
		}
	    };
	    FutureTask<Boolean> futureTask = new FutureTask<Boolean>(callable);
	    future = fileCopiesInProgress.putIfAbsent(toFileLocation, futureTask);
	    if (future == null) {
		future = futureTask;
		futureTask.run();
	    }
	}
        return future.get();

Once the application scales to multiple nodes, file-level locking is required. A quick search introduces Java’s FileChannel and FileLock classes. The documentation implies that platform independent OS-level locking is achieved using fileChannel.lock(). However, a quick test on my Ubuntu host shows that I can vi from a terminal and edit a file that’s locked by the Java process. However, it works perfectly when other Java processes try to acquire the lock, which in our case was all we needed.

	File src = new File(fromFileLocation);
	File dst = new File(toFileLocation);

	InputStream in = null;
	FileOutputStream out = null;
	FileLock fileLock = null;
	try {
	    in = new FileInputStream(src);
	    out = new FileOutputStream(dst);

	    FileChannel fileChannel = out.getChannel();
	    fileLock = fileChannel.lock();

	    byte b;
	    while ((b = (byte)in.read()) != -1) {
		out.write(b);
	    }
	    return true;
	} finally {
	    if (in != null)
		in.close();
	    if (fileLock != null && fileLock.isValid())
		fileLock.release();
	    if (out != null)
		out.close();
	}

There are two big downsides to using FileLock, at least in this case:

  • Copying the contents of the temporary file into the output file is a slow process.
  • The Javadoc states that the behaviour of FileLock is indeterminate while writing onto a NAS – very helpful, that!

It turns out, after all the hoopla, that the easiest approach is to rely on the OS to rename a file. I am unaware of whether the OS-rename operation is guaranteed to be atomic, but Dustin tested it out on Linux by tailing a file on one console while renaming it on another. The rename was successful, while the first console still held a handle onto its pipe and could continue tailing the file – exactly the behaviour we needed. We then completely stripped out the concurrency control in Java as we could now live with a single layer of locking. Not going to a simple rename directly was probably a waste of time, but then how do you learn about all the cool tricks out there that will probably come in handy at some point in the future?

Advertisements

2 comments

  1. About 10 years ago, Adrian Smith and I built a couple of transaction processing systems based on Unix link/unlink, with a directory being a queue. A file could only be in one directory or another. Whilst the technique worked well on vfs, it wasn’t atomic on NFS or NTFS filesystems, a single file appearing in one, both or neither directory during the rename, which the OS sometimes implemented as link(2) followed by an unlink(2). YMMV.


  2. Wow, that’s interesting. How did you come across these issues – was it during some sort of stress test? For now, we are just logging any errors during the rename – if such an issue does occur, then we will have to revisit. It may not be worth spending too much more time on something that might never occur, or very rarely šŸ™‚



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: