Дата публикации: 04.11.2013 9:44:05
In my work, I got the need to read textual data of a large number of files. So, I decided to try different approaches for JDK 8 on Windows 7, test their performance, and share the results. You can run similar tests for other platforms by using the TestIO class and share your findings.
In this article, I consider the task of reading only text files, because this is a typical use case when you need to process documents in encoding other than ASCII. You can use other classes and methods to process binary files, which are far more faster because these methods do not decode bytes to characters.
1.1. FileReader
The FileReader class is aimed to read text files. However, it performs this in a very inefficient way: every character is requested from the operating system. Not surprisingly, this is the slowest way to read the characters.
Reader reader = new FileReader(file); try { StringBuilder sb = new StringBuilder(); for (int ch = reader.read(); ch != -1; ch = reader.read()) { sb.append((char) ch); } return sb; } finally { reader.close(); }
1.2. BufferedReader
The BufferedReader class buffers the character stream from the FileReader class, substantially reducing the number of system calls.
Reader reader = new BufferedReader(new FileReader(file)); try { StringBuilder sb = new StringBuilder(); for (int ch = reader.read(); ch != -1; ch = reader.read()) { sb.append((char) ch); } return sb; } finally { reader.close(); }
The new mechanism of IO (NIO.2), introduced in Java 7, also creates a buffered stream of characters from the given file by using the specified encoding. As expected, its performance coincides with the previous approach.
Reader reader = Files.newBufferedReader(path, Charset.defaultCharset()); try { StringBuilder sb = new StringBuilder(); for (int ch = reader.read(); ch != -1; ch = reader.read()) { sb.append((char) ch); } return sb; } finally { reader.close(); }
1.4. Files.readAllBytes
The Files.readAllBytes method reads the whole file content at once as a byte array. All you need is to decode these bytes to characters by using the specified encoding. This approach has the only drawback: if the file is huge, the OutOfMemoryError exception will be thrown.
return new String(Files.readAllBytes(path), Charset.defaultCharset());
To read all lines from the text file you can use the Files.readAllLines method, but it is not applicable for my task. This method returns a String array detecting and removing all types of the line breaks.
1.5. FileChannel.open
To resolve а memory issue, you can use a memory mapping technique.
FileChannel channel = FileChannel.open(path, StandardOpenOption.READ); ByteBuffer buffer = channel.map(FileChannel.MapMode.READ_ONLY, 0, channel.size()); return Charset.defaultCharset().decode(buffer);
2.1. File.listFiles
The File.listFiles method returns a list of files and sub-directories in the current directory. To process all nested files, you should process all the sub-directories recursively.
public void visit(String name) { visit(new File(name)); } private void visit(File file) { if (file.isDirectory()) { for (File child : file.listFiles()) { visit(child); } } else if (file.isFile()) { // PROCESS CURRENT FILE } }
2.2. Files.walkFileTree
The new mechanism of IO provides a flexible way to process all the files. It is far more faster than the previous approach and it enables detecting cyclic links in the file system.
public void visit(String name) { try { Files.walkFileTree(new File(name).toPath(), new SimpleFileVisitor() { @Override public FileVisitResult visitFile(T file, BasicFileAttributes attrs) throws IOException { // PROCESS CURRENT FILE return FileVisitResult.CONTINUE; } }); } catch (IOException exception) { exception.printStackTrace(); } }
For my tests, I used JDK sources that include 7,618 files and contains 98,369,854 characters. The results shown in the table below demonstrate that the new mechanism of IO accelerates not only file reading but also processing of the file tree.