Compact RVData
Version: 1.0
Author: Exhydra
Date: 7-12-2011
Version History
- <Version 1.0> 07.12.2011 - Original Release
Description
The overall size of the Data directory can become quite significant in large projects or projects that use large maps. To combat this, Compact RVData can shrink the size of your Data folder significantly without any need for third-party EXEs or DLLs! Controlling the size of your Data directory is just one step you can take to optimize your game for faster distribution.
Since .rvdata files are merely glorified text files, they can be compressed and uncompressed very quickly. This also means that they can be compacted quite heavily. For example, a 500x500 map typically takes up 1.5 megabytes with nothing in it -- which is a huge size for a single map -- but once it is compressed it only takes up around 3 kilobytes!
Features
- Compresses .rvdata in the Data directory to save space.
- On-the-fly un-compression from RGSS Encrypted Archives.
- No third-party EXEs or DLLs required.
Screenshots
None
Instructions
See Script Header
Script
#===============================================================================
# Compact RVData
#===============================================================================
# Author : Exhydra
# Version : 1.0
# Last Updated : 07/12/2011
#===============================================================================
# Updates
# -----------------------------------------------------------------------------
# * 07/12/11 Initial Release.
#===============================================================================
# Future Options
# -----------------------------------------------------------------------------
# » Possibly loading GZip data directly from memory instead of temporarily
# writing it to disk. May not be possible.
#===============================================================================
# Instructions
# -----------------------------------------------------------------------------
# To Use :
# » Set RVDATA_BACKUP and COMPACT_ENABLE to 'true' if they are not already and
# then place the following line after 'begin' in your Main Process script :
#
# CompactRVData.compress_files if CompactOptions::COMPACT_ENABLE
#
# Each time that you run your game, a new rvdata backup folder will be created
# and the game will run off of the gzdata compressed files. Edit your game
# normally until you are ready to release it.
#
#
# » When you are ready to release your game, save your project and run your game
# one more time to make sure that the latest copy of your rvdata has been
# compacted. After you have done that, move or remove the files not listed
# in the EXCLUDE_FILES list below along with the rvdata backup directories.
# These files and directories should not be in the Data directory when you
# compress your game data from the File menu.
#
# Also, remember to set RVDATA_BACKUP and COMPACT_ENABLE to 'false' before
# continuing. You may also wish to remove the line that you added to the
# Main Process script area.
#
# After compressing your game data, you can move these files back to their
# original place if you so desire.
#===============================================================================
# Description
# -----------------------------------------------------------------------------
# - The overall size of the Data directory can become quite significant in large
# projects or projects that use large maps. To combat this, Compact RVData
# can shrink the size of your Data folder significantly without any need for
# third-party EXEs or DLLs! Controlling the size of your Data directory is just
# one step you can take to optimize your game for faster distribution.
#===============================================================================
#===============================================================================
# [NOTE:] - Remember that you will need to manually edit any scripts that try
# to use 'load_data' on a file which has been compacted using this
# script. You should only need to change 'load_data' with 'load_gzdata'
# for the script to continue functioning.
#===============================================================================
module CompactOptions
# Setting this to 'true' will backup all the files in your Data directory
# to a subfolder each time you choose to compress your files. The name of
# the folder is the date and the time the backup occurred (dd_mm_yyyy)
# (HH_MM_SS).
# [IMPORTANT] : Remember to remove these backup directories when you are
# releasing your game!
RVDATA_BACKUP = true
# Setting this to 'true' will compact any files in your Data directory
# which are not listed in the EXCLUDE_FILES list below.
# [IMPORTANT] : Remember to set this value to 'false' when you are releasing
# your game!
COMPACT_ENABLE = true
# [IMPORTANT] : Do not remove Scripts.rvdata from this list! The game will not
# load if it has been compressed! MapInfos is included on the
# list as it is generally small and many scripts open it for
# reading.
EXCLUDE_FILES = [
"Scripts.rvdata" ,
"MapInfos.rvdata" # ,
# "<example-name>.rvdata"
]
end # CompactOptions
class CompactRVData
#--------------------------------------------------------------------------
# » compress_files [ New ]
#--------------------------------------------------------------------------
def self.compress_files
self.backup_files if CompactOptions::RVDATA_BACKUP
datadir = self.get_datadir
Dir.entries(datadir).each do |filename|
nm_file = datadir + filename
next if File.ftype(nm_file) == "directory"
next if CompactOptions::EXCLUDE_FILES.include?(filename)
buffer = nil
ext = File.extname(nm_file)
base_f = filename.dup
base_f.slice!(ext)
gz_file = datadir + base_f + ".gzdata"
File.open(nm_file, "rb") { |infile| buffer = infile.read }
Zlib::GzipWriter.open(gz_file) { |gz| gz.write(buffer) }
File.open(gz_file, 'rb') { |marsh_file| save_data(marsh_file.readlines.join, gz_file) }
end
end
#--------------------------------------------------------------------------
# » backup_files [ New ]
#--------------------------------------------------------------------------
def self.backup_files
datadir = self.get_datadir
backdir = datadir + Time.now.strftime("%d_%m_%Y-%H_%M_%S") + "/"
self.make_dir(backdir)
Dir.entries(datadir).each do |filename|
next if File.ftype(datadir + filename) == "directory"
backfile = File.open(datadir + filename, "rb")
f_data = backfile.read
backfile.close
File.open(backdir + filename, 'wb') { |file| file.write(f_data) }
end
end
#--------------------------------------------------------------------------
# » load_gzdata [ New ]
# filename : Path and name of file.
#--------------------------------------------------------------------------
def self.load_gzdata(filename)
buffer = nil
gz_data = load_data(filename)
File.open('gztemp', 'wb') { |file| file.write(gz_data) }
File.open('gztemp', "rb") { |file|
begin
gz = Zlib::GzipReader.new(file)
buffer = gz.read
rescue
file.rewind
buffer = file.read
ensure
gz.finish if gz != nil
end
}
File.open('gztemp', "wb") { |file| file.write(buffer) }
rv_data = load_data('gztemp')
File.delete('gztemp')
return rv_data
end
#--------------------------------------------------------------------------
# » get_basedir [ New ]
#--------------------------------------------------------------------------
def self.get_basedir
basedir = File.expand_path('./Game.exe')
basedir.slice!('Game.exe')
return basedir
end
#--------------------------------------------------------------------------
# » get_datadir [ New ]
#--------------------------------------------------------------------------
def self.get_datadir
basedir = self.get_basedir
return basedir + "Data/"
end
#--------------------------------------------------------------------------
# » make_dir [ New ]
# directory : Full path ending with desired directory name.
#--------------------------------------------------------------------------
def self.make_dir(directory)
Dir.mkdir(directory) unless File.exist?(directory)
end
end # CompactRVData
#===============================================================================
# -> Scene_Title
# -----------------------------------------------------------------------------
# Summary of Changes:
# Re-Written Method(s) - load_database
#===============================================================================
class Scene_Title < Scene_Base
#--------------------------------------------------------------------------
# * Load Database [ Re-Write ]
#--------------------------------------------------------------------------
def load_database
ex_gz = CompactRVData
$data_actors = ex_gz.load_gzdata("Data/Actors.gzdata")
$data_classes = ex_gz.load_gzdata("Data/Classes.gzdata")
$data_skills = ex_gz.load_gzdata("Data/Skills.gzdata")
$data_items = ex_gz.load_gzdata("Data/Items.gzdata")
$data_weapons = ex_gz.load_gzdata("Data/Weapons.gzdata")
$data_armors = ex_gz.load_gzdata("Data/Armors.gzdata")
$data_enemies = ex_gz.load_gzdata("Data/Enemies.gzdata")
$data_troops = ex_gz.load_gzdata("Data/Troops.gzdata")
$data_states = ex_gz.load_gzdata("Data/States.gzdata")
$data_animations = ex_gz.load_gzdata("Data/Animations.gzdata")
$data_common_events = ex_gz.load_gzdata("Data/CommonEvents.gzdata")
$data_system = ex_gz.load_gzdata("Data/System.gzdata")
$data_areas = ex_gz.load_gzdata("Data/Areas.gzdata")
end
end # Scene_Title
#===============================================================================
# -> Game_Map
# -----------------------------------------------------------------------------
# Summary of Changes:
# Re-Written Method(s) - setup
#===============================================================================
class Game_Map
#--------------------------------------------------------------------------
# * Setup [ Re-Write ]
#--------------------------------------------------------------------------
def setup(map_id)
ex_gz = CompactRVData
@map_id = map_id
# [ Altered ]
@map = ex_gz.load_gzdata(sprintf("Data/Map%03d.gzdata", @map_id))
@display_x = 0
@display_y = 0
@passages = $data_system.passages
referesh_vehicles
setup_events
setup_scroll
setup_parallax
@need_refresh = false
end
end # Game_Map
Credit
Thanks
- TOMY and KGC for their scripts on using Ruby's GZip.
Support
- Please post in this thread for support.
Known Compatibility Issues
- Since this alters the way that information is obtained from .rvdata files, there is a high chance of breaking other scripts which rely on reading them. This can be corrected with minor intervention from the game creator.
Demo
See Attachment
Author's Notes
- This script includes re-writes for Scene_Title (load_database) and Game_Map (setup), as those are the basic necessities to allow this script to work with a new project. The script as it stands right now could definitely become more compatible, but I'm not quite sure how to do so. If anyone has ideas, please let me know!
Also, at the moment, each file that is loaded with 'load_gzdata' is dumped to the hard drive briefly, unencrypted and uncompressed. I've been trying to find a way to do everything in memory, but I can't seem to get that to work.
- GzipWriter Documentation (http://www.ruby-doc.org/stdlib/libdoc/zlib/rdoc/classes/Zlib/GzipWriter.html)
GzipReader Documentation (http://www.ruby-doc.org/stdlib/libdoc/zlib/rdoc/classes/Zlib/GzipReader.html)
Restrictions
- Free for use in non-commercial and commercial games.
Hi Exhydra,
I like some of your concepts in your script :)
Woo, it's Zeriab! :)
I suggest you use Zlib::Deflate.deflate and Zlib::Inflate.inflate for compression as that will allow you to do in memory decompression.
I also suggest that you add the decompression logic to the load_data method as that is the only way to load marshalled objects from the encrypted archive.
Originally, I tried using Deflate and Inflate, but I kept getting errors about how I couldn't compress an Array/Hash into a String (which is what Deflate and Inflate try and do; turn the input into a string to compress). I dug around in the Ruby docs to try and find a way around what was going on, but I could never find it. I got frustrated and decided to dump the Gzipped files to the hard drive for a moment to un-compress and load them instead of doing it in memory like I wanted.
Here is my very alpha code when I was trying to load compressed files into memory. It's really sloppy, but I remember Deflate giving me errors about how it couldn't convert the Array to a String :
def compress_data #(filename)
buf_nrm = nil
buf_zlb = nil
buf_str = ""
buf_nrm = load_data('Data\Actors.rvdata')
buf_zlb = Zlib::Deflate.deflate(buf_nrm)
save_data(buf_zlb, 'Data\Actors.gzdata')
end
def decompress_data(data_stream)
buf = []
buf = Zlib::Inflate.inflate(data_stream)
return buf
end
Right now I'm already using 'load_data' to take the marshalled files from the encrypted archive. Unless I'm not understanding what you're suggesting. I tried to find the code for the 'load_data' method, as putting decompression code in there would increase compatibility, but I couldn't find it. :(
To get around the problems I was having, I put a Marshall 'wrapper' around the GZipped file. This is pretty inefficient, but it works. So right now the script is operating kind of like this :
< Compression (Done before Encrypted Archive is created) >
Buffer << File (Local Drive)
Deflate >> File (Local Drive)
Save_Data >> File (Local Drive)
-> File Structure [ Marshall > GZip > Marshall ]
< Decompression >
Buffer << Load Data | Encrypted Archive
Buffer >> File (Local Drive)
Inflate >> File (Local Drive)
Buffer << Load_Data | Marshalled File
<< Delete File
Note that the compression ratios will most likely not be as big in practice. I did an experiment where I compressed the rxdata for 5 commercial RMXP projects and found that difference was smaller than what the test data indicated, but very much significant if you looked at the data folders alone. On the downside was that the reducing in total project size was only 3-4% which is still better than nothing.
Yeah, I know that typically the Data directory really isn't that large. I was just thinking that it would work nicely on some bigger projects (50+ maps) or projects that use sizable maps (not that there are many who do that), it would be more of a boon. But to the normal RPG Maker user, it probably won't squeeze a whole lot out ... but every little bit helps, I guess! :)
EDIT: By the way, I put your old Remove Comments (http://bb.xieke.com/files/Remove_Comments.zip) project in the Project Optimization (http://rmrk.net/index.php/topic,43125.msg491773.html) thread I've been slowly building. It works really nicely. :)