Cross-platform file locking support in Python
On occasion, one requires the need to lock a file. Now, this is relatively easy if you’re targeting a specific platform because there is often a function in the library to do it for you. But what if you want to target a larger set of platforms? The following is a solution I wrote up today. It’s lockfile creation is an atomic operation and thus doesn’t suffer from any race conditions. It should work in both Windows and Unix environments.
import os import time import errno class FileLockException(Exception): pass class FileLock(object): """ A file locking mechanism that has context-manager support so you can use it in a with statement. This should be relatively cross compatible as it doesn't rely on msvcrt or fcntl for the locking. """ def __init__(self, file_name, timeout=10, delay=.05): """ Prepare the file locker. Specify the file to lock and optionally the maximum timeout and the delay between each attempt to lock. """ self.is_locked = False self.lockfile = os.path.join(os.getcwd(), "%s.lock" % file_name) self.file_name = file_name self.timeout = timeout self.delay = delay def acquire(self): """ Acquire the lock, if possible. If the lock is in use, it check again every `wait` seconds. It does this until it either gets the lock or exceeds `timeout` number of seconds, in which case it throws an exception. """ start_time = time.time() while True: try: self.fd = os.open(self.lockfile, os.O_CREAT|os.O_EXCL|os.O_RDWR) break; except OSError as e: if e.errno != errno.EEXIST: raise if (time.time() - start_time) >= self.timeout: raise FileLockException("Timeout occured.") time.sleep(self.delay) self.is_locked = True def release(self): """ Get rid of the lock by deleting the lockfile. When working in a `with` statement, this gets automatically called at the end. """ if self.is_locked: os.close(self.fd) os.unlink(self.lockfile) self.is_locked = False def __enter__(self): """ Activated when used in the with statement. Should automatically acquire a lock to be used in the with block. """ if not self.is_locked: self.acquire() return self def __exit__(self, type, value, traceback): """ Activated at the end of the with statement. It automatically releases the lock if it isn't locked. """ if self.is_locked: self.release() def __del__(self): """ Make sure that the FileLock instance doesn't leave a lockfile lying around. """ self.release()
The above class is best used in a context manager fashion through the with statement like in the example below:
with FileLock("test.txt", timeout=2) as lock: print("Lock acquired.") # Do something with the locked file
The largest downside of this is that the directory the file is located in must be writable. I hope this code helps you. Of course, if you have a better recipe, please share it in the comments. ;)
Ville Kaseva wrote,
Is there bug in acquire() method? Can timeout occur without entering Except: ??
Evan wrote,
Ville,
There shouldn’t be any bug in acquire(). It may seem that there would be a race condition, but the following line is actually atomic:
A timeout hasn’t occured outside the Except block when I’ve used it, but if it has for you, I’d love to see the trackback.
Pat wrote,
Could you please indicate the license under which this cross-platform file-locking code is available? Thanks!
Dan wrote,
Thanks for the code. The only change I made was to use os.mkdir and os.rmdir instead of os.open and os.unlink.
Evan wrote,
Pat, the code is lisenced under the BSD license.
Dan, so I’m guessing that you’re using a “lockfolder” instead of a lockfile? Be sure that is an atomic operation otherwise you may run into problems.
John Giotta wrote,
This is a very useful script. Now to make it work for my environments running 2.4
Steve wrote,
Evan,
What would happen when the process dies but never unlocked the file? I am wanting to use your solution but need to make sure that this case is handled.
Matt Chaput wrote,
This (and directory creation, which is also atomic and IMHO makes for clearer code) will leave behind a stale lock file/directory if your code doesn’t delete it (e.g. if you get an error and the process quits), preventing anyone else from locking the resource until you clean it up manually.
Vinay Sajip wrote,
@John Giotta: There’s a simpler version of this which I wrote back in 2007, which works on older Pythons but has no bells/whistles/context manager. See here: http://code.activestate.com/recipes/519626-simple-file-based-mutex-for-very-basic-ipc/
Note that it writes the PID of the locking process to the file, so you can see who’s got it (and this also allows cleanup – e.g. if there’s no such process).
PP wrote,
It works very well!
Many thanks!
PP
Rick Harris wrote,
Evan, thanks for posting this, was a big help. I ended up using your FileLock to build a file-based counting semaphore which is used by http://github.com/rconradharris/filecache.
Great job!
Suan wrote,
Hi, I ended up using this in my LG TV serial control library project too: https://github.com/suan/libLGTV_serial
Thanks for writing this!
Bill Pollifrone wrote,
Hello. I want just wanted to thank you for posting this code. I only had to modify to use a temporary directory for the lock files so that I was sure to be able to write to it. I used it to ensure one process that reads files created by another process would not collide.
Volker wrote,
For the record, the code is not safe for all NFS implementations. See the O_EXCL section of the “open” manpage.