Introduction
'File Cleaner' tool is developed to ZIP and/or delete files older than X number of DAYS/MONTHS/YEARS. It is developed for making some extra free space on system or servers where too many LOG/TEXT files are getting generated.
This tool has been successfully tested on Windows XP/Vista/7, Windows Server 2000/2003, Windows Developer Preview (Windows 8), Ubuntu 11, Fedora 14, UNIX - Sun Solaris, Mac OS X Lion.
Background
One of my friends was discussing with me about the production Batch Component servers and the disk space consumed by the batch jobs over the period of time.
For example, 'Server A' is a batch component server and having around 100+ batch jobs configured. Most of the jobs are configured to run daily and consume around 3 GB of disk space per-day which includes INPUT files, OUTPUT files, all kind of LOG files, etc. With this fact, if server has 500 GB of SAN allocated, the disk will be full within 5 months or so.
Problem
Now adding 500 GB of SAN every 5 months is not advisable in a real production environment. Also most of the batch components do generate LOG file(s) but do not cleanup which is good and helps in real production environment to investigate issue(s).
Solution
As an outcome of the above scenario, there is a business need for some file cleanup tool. Most of the time, respective 'Server Support' and/or 'Application Support' teams write operating system specific micro scripts to achieve this which even my friend has done to his production environment.
Now suppose we have 20 servers out of which 10 are Windows, 4 are Linux, 5 UNIX and 1 OS X. With this, there will be at least 3 different versions of operating system specific micro scripts. Being a technical guy, I will not prefer to have 3 different versions of micro scripts, so I developed this small tool using JAVA.
Since most of the operating systems come with basic version of JAVA run-time installed, I choose to develop this tool using Core JAVA.
Using the Tool
The tool is very simple to configure and use. We need to execute JAR file using JAVA run-time and pass the '.properties' file as argument. The '.properties' file contains the configuration or parameters used by the tool. A sample '.properties' file is as below.
sample.properties
# +--------------------------------+
# | FileCleanerLauncher properties |
# +--------------------------------+
# if zip_files is not provided, true will be used
zip_files = true
# if delete_files is not provided, true will be used
delete_files = true
# +-----------------------+
# | FileFilter properties |
# +-----------------------+
directory = D:\\
file_expression = *.txt
# Expected values [days, months, years]
keep_last_type = days
keep_last = 1
# +-----------------------+
# | FileZipper properties |
# +-----------------------+
# if zip_directory is not provided, directory of FileFilter will be used
#zip_directory =
zip_file_format = a-%tF.zip
# if buffer_size is not provided, default value 1024 will be used
#buffer_size = 2048
JAR File Execution
java -jar FileCleaner.jar sample.properties
The Code
This tool is a very basic JAVA console application. The code is the collection of 4 core classes and 2 helper classes each described below.
PropertiesConfiguration Class
This class facilitates reading settings from '.properties' file. The '.properties' file can be default one as 'filecleaner.properties' or custom passed as a command line argument as showed under 'JAR file execution:'
private static String DEFAULT_PROPERTIES_FILENAME = "filecleaner.properties";
private static PropertiesConfiguration instance = null;
public static void init(String propertiesFile) {
if (null == instance) {
try {
instance = new PropertiesConfiguration(propertiesFile);
} catch (Exception e) {
e.printStackTrace();
}
}
}
public static void init() {
init(DEFAULT_PROPERTIES_FILENAME);
}
Helper Class
I was missing a good 'String' function of .NET called 'IsNullOrEmpty' so I just created it as a helper function in this class.
public final class Helper {
public static Boolean hasString(String string) {
return !((null == string) || string.isEmpty());
}
}
FileFilter Class
This class is the first step for this tool. It is responsible for filtering and generating the list of files to be ZIPped and/or deleted. It has four settings to read from '.properties' file. It has two methods which filter the files based on parameters and returns the list as String array.
- directory: Valid directory or folder path to look for files. Path syntax differs based on 'Operating System'.
- file_expression: Valid JAVA string expression to filter files. This can also include wildcard characters like '?' and '*'.
- keep_last_type: Value can be DAYS/MONTHS/YEARS indicating the type of deduction to be done from System Date. If the value is DAYS, number of days will be deducted. If value is MONTHS, number of months will be deducted. If value is YEARS, number of years will be deducted.
- keep_last: Integer value indicating files to be filtered older then 'X' number of days/months/years. While filtering the files, this value is deducted from the current System Date based on value specified under '
keep_last_type'.
public FileFilter() throws Exception {
String value = null;
value = PropertiesConfiguration.getPropertyValue("directory");
if (!Helper.hasString(value)) {
throw new Exception("Invalid property value: directory");
}
this.directory = value;
value = PropertiesConfiguration.getPropertyValue("file_expression");
if (!Helper.hasString(value)) {
throw new Exception("Invalid property value: file_expression");
}
this.fileExpression = value;
value = PropertiesConfiguration.getPropertyValue("keep_last_type");
if (!Helper.hasString(value)) {
throw new Exception("Invalid property value: keep_last_type");
}
value = value.toUpperCase();
if ("DAYS".equals(value)) {
this.keepLastType = Calendar.DATE;
} else if ("MONTHS".equals(value)) {
this.keepLastType = Calendar.MONTH;
} else if ("YEARS".equals(value)) {
this.keepLastType = Calendar.YEAR;
} else {
throw new Exception("Invalid property value: keep_last_type");
}
value = PropertiesConfiguration.getPropertyValue("keep_last");
this.keepLast = Integer.parseInt(value);
}
- getKeepDate - method: Returns the
Date which should be compared with System Date.
protected Date getKeepDate() {
Calendar calendar = Calendar.getInstance();
calendar.add(this.keepLastType, (this.keepLast * -1));
return calendar.getTime();
}
- getFileNames - method: Returns the list of filtered files as
String array. The returned array can be passed to other two steps ZIP and Delete.
public String[] getFileNames() throws FileNotFoundException {
String matchExpression = fileExpression.replace("?", ".?").replace("*", ".*");
Date keepDate = this.getKeepDate();
File dir = new File(this.directory);
if (!dir.isDirectory() || !dir.exists()) {
throw new FileNotFoundException(this.directory);
}
ArrayList<string> filteredFileNames = new ArrayList<string>();
String[] fileNames = dir.list();
for (String fileName : fileNames) {
if (!fileName.matches(matchExpression)) {
continue;
}
File file = new File(dir, fileName);
if (!file.isFile() || !file.exists()) {
continue;
}
Date lastModified = new Date(file.lastModified());
if (lastModified.after(keepDate)) {
continue;
}
filteredFileNames.add(file.toString());
}
return filteredFileNames.toArray(new String[filteredFileNames.size()]);
}
FileZipper Class
This class is the second step for this tool. It is responsible for compressing the list of provided files to a ZIP/tar.gz file. It has three settings to read from '.properties' file. It has two methods to compress and create ZIP/tar.gz file.
- zip_directory: Valid directory or folder path to create compressed file. If the value is not defined or provided, it will use '
directory' settings of 'FileFilter' class. Path syntax differs based on 'Operating System'. - zip_file_format: Valid compressed file name format. For more details on valid formats, please refer to 'java.util.Formatter class'.
- buffer_size: Integer value indicating the size of buffer in byes to be used for read/write files. If not provided, default value 1024 bytes will be used. The performance can be boosted by setting big buffer size for large files.
public FileZipper() throws Exception {
String value = null;
value = PropertiesConfiguration.getPropertyValue("zip_directory");
if (!Helper.hasString(value)) {
value = PropertiesConfiguration.getPropertyValue("directory");
if (!Helper.hasString(value)) {
throw new Exception("Invalid property value: zip_directory");
}
}
this.directory = value;
value = PropertiesConfiguration.getPropertyValue("zip_file_format");
if (!Helper.hasString(value)) {
throw new Exception("Invalid property value: zip_file_format");
}
this.zipFileFormat = value;
value = PropertiesConfiguration.getPropertyValue("buffer_size");
if (!Helper.hasString(value)) {
value = "1024";
}
this.bufferSize = Integer.parseInt(value);
}
- getZipFileName - method: Returns ZIP/tar.gz file path as
string.
protected String getZipFileName() {
Formatter formatter = new Formatter();
formatter.format(zipFileFormat, Calendar.getInstance());
return this.directory.concat(formatter.toString());
}
- zipFiles - method: Compresses the provided files as
String array and returns newly created compress file path as string. It uses 'BEST_COMPRESSION' as compress level which can compress normal text file to around 95%.
public String zipFiles(String[] fileNames) throws IOException {
if (null == fileNames || 0 == fileNames.length) {
return null;
}
String zipFileName = this.getZipFileName();
FileOutputStream fos = new FileOutputStream(zipFileName);
ZipOutputStream zos = new ZipOutputStream(fos);
zos.setLevel(Deflater.BEST_COMPRESSION);
int bytesRead;
byte[] buffer = new byte[this.bufferSize];
CRC32 crc = new CRC32();
for (String fileName : fileNames) {
File file = new File(fileName);
if (!file.isFile() || !file.exists()) {
continue;
}
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(file));
crc.reset();
while ((bytesRead = bis.read(buffer)) != -1) {
crc.update(buffer, 0, bytesRead);
}
bis.close();
bis = new BufferedInputStream(new FileInputStream(file));
ZipEntry entry = new ZipEntry(file.getName());
entry.setCrc(crc.getValue());
zos.putNextEntry(entry);
while ((bytesRead = bis.read(buffer)) != -1) {
zos.write(buffer, 0, bytesRead);
}
bis.close();
}
zos.close();
return zipFileName;
}
FileDeleter Class
This class is the third step for this tool. This is a very simple class and it just deletes the files provided as String array.
public void deleteFileNames(String[] fileNames) {
for (String fileName : fileNames) {
File file = new File(fileName);
if (!file.isFile() || !file.exists()) {
continue;
}
file.delete();
}
}
FileCleanerLauncher Class
This class is the main class. It also has two settings to read from '.properties' file and has four methods.
public static void main(String[] args) {
if (args.length > 0) {
PropertiesConfiguration.init(args[0]);
} else {
PropertiesConfiguration.init();
}
FileCleanerLauncher launcher = new FileCleanerLauncher();
launcher.run();
launcher = null;
}
- zip_files: '
true' if files need to be compressed, 'false' otherwise. If not provided, default value is 'false'. If set as 'true', files will be compressed using 'FileZipper' class. If set as 'false', files will NOT be compressed. - delete_files: '
true' if files need to be deleted, 'false' otherwise. If not provided, default value is 'false'. If set as 'true', files will be deleted using 'FileDelete' class. If set as 'false', files will NOT be deleted.
Note: If both are set to 'false', nothing will be done. In other case, if 'zip_files' is set to 'false' and 'delete_files' set to 'true', files will be deleted without backup and cannot be recovered. And if 'zip_files' is set to 'true' and 'delete_files' set to 'false', files will be compressed but will NOT be deleted which does not free space on disk.
public FileCleanerLauncher() {
String value = null;
value = PropertiesConfiguration.getPropertyValue("zip_files");
if (!Helper.hasString(value)) {
value = "true";
}
this.zipFiles = Boolean.parseBoolean(value);
value = PropertiesConfiguration.getPropertyValue("delete_files");
if (!Helper.hasString(value)) {
value = "true";
}
this.deleteFiles = Boolean.parseBoolean(value);
}
- run - method: It just calls three different methods defined as each step.
public void run() {
try {
String[] fileNames = this.getFileNames();
this.zipFiles(fileNames);
this.deleteFiles(fileNames);
} catch (Exception e) {
e.printStackTrace();
}
}
- getFileNames - method: With help of '
FileFilter' class, it gets the filtered file list as String array.
protected String[] getFileNames() throws Exception {
String[] fileNames = null;
FileFilter filter = new FileFilter();
fileNames = filter.getFileNames();
filter = null;
return fileNames;
}
- zipFiles - method: With help of '
FileZipper' class, it compresses provided file list and returns file path of newly created compressed file. If 'zip_files' is set to 'false', this step will be skipped.
protected String zipFiles(String[] fileNames) throws Exception {
if (!this.zipFiles) {
return null;
}
String zipFileName = null;
FileZipper zipper = new FileZipper();
zipFileName = zipper.zipFiles(fileNames);
zipper = null;
return zipFileName;
}
- deleteFiles - method: With the help of '
FileDelete' class, it deletes provided file list. If 'delete_files' is set to 'false', this step will be skipped.
protected void deleteFiles(String[] fileNames) {
if (!this.deleteFiles) {
return;
}
FileDeleter deleter = new FileDeleter();
deleter.deleteFileNames(fileNames);
deleter = null;
}
Conclusion
I am not much of an expert in the JAVA world, but it was a very good experience understanding ZIP file concept in JAVA. Also testing on multiple platforms was a challenge and it was achieved with the help of 'Virtual Box'.
History
- 15th November, 2011: Initial post
