Compact RVData [1.0]

Compact RVData
Version: 1.0
Author: Exhydra
Date: 7-12-2011

Version History

• <Version 1.0> 07.12.2011 - Original Release

Description

The overall size of the Data directory can become quite significant in large projects or projects that use large maps. To combat this, Compact RVData can shrink the size of your Data folder significantly without any need for third-party EXEs or DLLs! Controlling the size of your Data directory is just one step you can take to optimize your game for faster distribution.

Since .rvdata files are merely glorified text files, they can be compressed and uncompressed very quickly. This also means that they can be compacted quite heavily. For example, a 500x500 map typically takes up 1.5 megabytes with nothing in it -- which is a huge size for a single map -- but once it is compressed it only takes up around 3 kilobytes!

Features

• Compresses .rvdata in the Data directory to save space.
• On-the-fly un-compression from RGSS Encrypted Archives.
• No third-party EXEs or DLLs required.

Screenshots

None

Instructions

See Script Header

Script

Code: [Select]

#===============================================================================
# Compact RVData
#===============================================================================
# Author       : Exhydra
# Version      : 1.0
# Last Updated : 07/12/2011
#===============================================================================
# Updates
# -----------------------------------------------------------------------------
# * 07/12/11 Initial Release.
#===============================================================================
# Future Options
# -----------------------------------------------------------------------------
# » Possibly loading GZip data directly from memory instead of temporarily
#   writing it to disk. May not be possible.
#===============================================================================
# Instructions
# -----------------------------------------------------------------------------
# To Use :
# » Set RVDATA_BACKUP and COMPACT_ENABLE to 'true' if they are not already and
#   then place the following line after 'begin' in your Main Process script :
#
#   CompactRVData.compress_files if CompactOptions::COMPACT_ENABLE
#
#   Each time that you run your game, a new rvdata backup folder will be created
#   and the game will run off of the gzdata compressed files. Edit your game
#   normally until you are ready to release it.
#
#
# » When you are ready to release your game, save your project and run your game
#   one more time to make sure that the latest copy of your rvdata has been
#   compacted. After you have done that, move or remove the files not listed
#   in the EXCLUDE_FILES list below along with the rvdata backup directories.
#   These files and directories should not be in the Data directory when you
#   compress your game data from the File menu.
#
#   Also, remember to set RVDATA_BACKUP and COMPACT_ENABLE to 'false' before
#   continuing. You may also wish to remove the line that you added to the
#   Main Process script area.
#
#   After compressing your game data, you can move these files back to their
#   original place if you so desire.
#===============================================================================
# Description
# -----------------------------------------------------------------------------
# - The overall size of the Data directory can become quite significant in large
# projects or projects that use large maps. To combat this, Compact RVData
# can shrink the size of your Data folder significantly without any need for
# third-party EXEs or DLLs! Controlling the size of your Data directory is just
# one step you can take to optimize your game for faster distribution.
#===============================================================================

#===============================================================================
# [NOTE:] - Remember that you will need to manually edit any scripts that try
#           to use 'load_data' on a file which has been compacted using this
#           script. You should only need to change 'load_data' with 'load_gzdata'
#           for the script to continue functioning.
#===============================================================================

module CompactOptions
 
  # Setting this to 'true' will backup all the files in your Data directory
  # to a subfolder each time you choose to compress your files. The name of
  # the folder is the date and the time the backup occurred (dd_mm_yyyy)
  # (HH_MM_SS).
  # [IMPORTANT] : Remember to remove these backup directories when you are
  #             releasing your game!
  RVDATA_BACKUP      = true
 
  # Setting this to 'true' will compact any files in your Data directory
  # which are not listed in the EXCLUDE_FILES list below.
  # [IMPORTANT] : Remember to set this value to 'false' when you are releasing
  #             your game!
  COMPACT_ENABLE     = true
 
 
  # [IMPORTANT] : Do not remove Scripts.rvdata from this list! The game will not
  #             load if it has been compressed! MapInfos is included on the
  #             list as it is generally small and many scripts open it for
  #             reading.
  EXCLUDE_FILES      = [
                        "Scripts.rvdata"       ,
                        "MapInfos.rvdata"    # ,
  #                     "<example-name>.rvdata"
                       ]

end # CompactOptions



class CompactRVData
 
  #--------------------------------------------------------------------------
  # » compress_files                                                  [ New ]
  #--------------------------------------------------------------------------
  def self.compress_files
    self.backup_files if CompactOptions::RVDATA_BACKUP

    datadir = self.get_datadir
   
    Dir.entries(datadir).each do |filename|
      nm_file = datadir + filename
      next if File.ftype(nm_file) == "directory"
      next if CompactOptions::EXCLUDE_FILES.include?(filename)

      buffer  = nil
      ext     = File.extname(nm_file)
      base_f  = filename.dup
      base_f.slice!(ext)
      gz_file = datadir + base_f + ".gzdata"

      File.open(nm_file, "rb") { |infile| buffer = infile.read }
      Zlib::GzipWriter.open(gz_file) { |gz| gz.write(buffer) }
      File.open(gz_file, 'rb') { |marsh_file| save_data(marsh_file.readlines.join, gz_file) }
    end
  end
 
  #--------------------------------------------------------------------------
  # » backup_files                                                    [ New ]
  #--------------------------------------------------------------------------
  def self.backup_files
    datadir = self.get_datadir
    backdir = datadir + Time.now.strftime("%d_%m_%Y-%H_%M_%S") + "/"
    self.make_dir(backdir)
   
    Dir.entries(datadir).each do |filename|
      next if File.ftype(datadir + filename) == "directory"

      backfile = File.open(datadir + filename, "rb")
        f_data   = backfile.read
      backfile.close

      File.open(backdir + filename, 'wb') { |file| file.write(f_data) }
    end
  end
 
  #--------------------------------------------------------------------------
  # » load_gzdata                                                     [ New ]
  #     filename     : Path and name of file.
  #--------------------------------------------------------------------------
  def self.load_gzdata(filename)
    buffer = nil
    gz_data = load_data(filename)
    File.open('gztemp', 'wb') { |file| file.write(gz_data) }
    File.open('gztemp', "rb") { |file|
      begin
        gz     = Zlib::GzipReader.new(file)
        buffer = gz.read
      rescue
        file.rewind
        buffer = file.read
      ensure
        gz.finish if gz != nil
      end
    }
    File.open('gztemp', "wb") { |file| file.write(buffer) }
    rv_data = load_data('gztemp')
    File.delete('gztemp')
    return rv_data
  end

  #--------------------------------------------------------------------------
  # » get_basedir                                                     [ New ]
  #--------------------------------------------------------------------------
  def self.get_basedir
    basedir = File.expand_path('./Game.exe')
    basedir.slice!('Game.exe')
    return basedir
  end

  #--------------------------------------------------------------------------
  # » get_datadir                                                     [ New ]
  #--------------------------------------------------------------------------
  def self.get_datadir
    basedir = self.get_basedir
    return basedir + "Data/"
  end
 
  #--------------------------------------------------------------------------
  # » make_dir                                                        [ New ]
  #     directory    : Full path ending with desired directory name.
  #--------------------------------------------------------------------------
  def self.make_dir(directory)
    Dir.mkdir(directory) unless File.exist?(directory)
  end
 
end # CompactRVData



#===============================================================================
# -> Scene_Title
# -----------------------------------------------------------------------------
# Summary of Changes:
#    Re-Written Method(s) - load_database
#===============================================================================
class Scene_Title < Scene_Base
 
  #--------------------------------------------------------------------------
  # * Load Database                                              [ Re-Write ]
  #--------------------------------------------------------------------------
  def load_database
    ex_gz               = CompactRVData
    $data_actors        = ex_gz.load_gzdata("Data/Actors.gzdata")
    $data_classes       = ex_gz.load_gzdata("Data/Classes.gzdata")
    $data_skills        = ex_gz.load_gzdata("Data/Skills.gzdata")
    $data_items         = ex_gz.load_gzdata("Data/Items.gzdata")
    $data_weapons       = ex_gz.load_gzdata("Data/Weapons.gzdata")
    $data_armors        = ex_gz.load_gzdata("Data/Armors.gzdata")
    $data_enemies       = ex_gz.load_gzdata("Data/Enemies.gzdata")
    $data_troops        = ex_gz.load_gzdata("Data/Troops.gzdata")
    $data_states        = ex_gz.load_gzdata("Data/States.gzdata")
    $data_animations    = ex_gz.load_gzdata("Data/Animations.gzdata")
    $data_common_events = ex_gz.load_gzdata("Data/CommonEvents.gzdata")
    $data_system        = ex_gz.load_gzdata("Data/System.gzdata")
    $data_areas         = ex_gz.load_gzdata("Data/Areas.gzdata")
  end

end # Scene_Title



#===============================================================================
# -> Game_Map
# -----------------------------------------------------------------------------
# Summary of Changes:
#    Re-Written Method(s) - setup
#===============================================================================
class Game_Map

  #--------------------------------------------------------------------------
  # * Setup                                                      [ Re-Write ]
  #--------------------------------------------------------------------------
  def setup(map_id)
    ex_gz = CompactRVData
    @map_id = map_id
   
    # [ Altered ]
    @map = ex_gz.load_gzdata(sprintf("Data/Map%03d.gzdata", @map_id))
   
    @display_x = 0
    @display_y = 0
    @passages = $data_system.passages
    referesh_vehicles
    setup_events
    setup_scroll
    setup_parallax
    @need_refresh = false
  end
 
end # Game_Map


Credit

• Exhydra

Thanks

• TOMY and KGC for their scripts on using Ruby's GZip.

Support

• Please post in this thread for support.

Known Compatibility Issues

• Since this alters the way that information is obtained from .rvdata files, there is a high chance of breaking other scripts which rely on reading them. This can be corrected with minor intervention from the game creator.

Demo

See Attachment

Author's Notes

• This script includes re-writes for Scene_Title (load_database) and Game_Map (setup), as those are the basic necessities to allow this script to work with a new project. The script as it stands right now could definitely become more compatible, but I'm not quite sure how to do so. If anyone has ideas, please let me know!
  
  Also, at the moment, each file that is loaded with 'load_gzdata' is dumped to the hard drive briefly, unencrypted and uncompressed. I've been trying to find a way to do everything in memory, but I can't seem to get that to work.
  
  
• GzipWriter Documentation
  GzipReader Documentation

Restrictions

• Free for use in non-commercial and commercial games.

This looks great. Nice work Exhydra!

Nice script, Exhydra. I glanced over it for a while and was already impressed. I was a bit worried at RGSS encryption, but after looking at how Enterbrain handles RGSS2A files it should run no problem.   

Hi Exhydra,
I like some of your concepts in your script

I suggest you use Zlib::Deflate.deflate and Zlib::Inflate.inflate for compression as that will allow you to do in memory decompression.
I also suggest that you add the decompression logic to the load_data method as that is the only way to load marshalled objects from the encrypted archive.

Note that the compression ratios will most likely not be as big in practice. I did an experiment where I compressed the rxdata for 5 commercial RMXP projects and found that difference was smaller than what the test data indicated, but very much significant if you looked at the data folders alone. On the downside was that the reducing in total project size was only 3-4% which is still better than nothing.

@cozziekuns: RGSS and RGSS2 use the same encryption system.

*hugs*
 - Zeriab

Hi Exhydra,
I like some of your concepts in your script

Woo, it's Zeriab! 

I suggest you use Zlib::Deflate.deflate and Zlib::Inflate.inflate for compression as that will allow you to do in memory decompression.
I also suggest that you add the decompression logic to the load_data method as that is the only way to load marshalled objects from the encrypted archive.

Originally, I tried using Deflate and Inflate, but I kept getting errors about how I couldn't compress an Array/Hash into a String (which is what Deflate and Inflate try and do; turn the input into a string to compress). I dug around in the Ruby docs to try and find a way around what was going on, but I could never find it. I got frustrated and decided to dump the Gzipped files to the hard drive for a moment to un-compress and load them instead of doing it in memory like I wanted.

Here is my very alpha code when I was trying to load compressed files into memory. It's really sloppy, but I remember Deflate giving me errors about how it couldn't convert the Array to a String :

Code: [Select]

  def compress_data #(filename)
    buf_nrm = nil
    buf_zlb = nil
    buf_str = ""
   
    buf_nrm = load_data('Data\Actors.rvdata')
    buf_zlb = Zlib::Deflate.deflate(buf_nrm)
    save_data(buf_zlb, 'Data\Actors.gzdata')
  end

  def decompress_data(data_stream)
    buf = []
    buf = Zlib::Inflate.inflate(data_stream)
    return buf
  end


Right now I'm already using 'load_data' to take the marshalled files from the encrypted archive. Unless I'm not understanding what you're suggesting. I tried to find the code for the 'load_data' method, as putting decompression code in there would increase compatibility, but I couldn't find it.

To get around the problems I was having, I put a Marshall 'wrapper' around the GZipped file. This is pretty inefficient, but it works. So right now the script is operating kind of like this :

Code: [Select]

< Compression (Done before Encrypted Archive is created) >
Buffer    << File (Local Drive)
Deflate   >> File (Local Drive)
Save_Data >> File (Local Drive)
 -> File Structure [ Marshall > GZip > Marshall ]


< Decompression >
Buffer  << Load Data | Encrypted Archive
Buffer  >> File (Local Drive)
Inflate >> File (Local Drive)
Buffer  << Load_Data | Marshalled File
        << Delete File


Note that the compression ratios will most likely not be as big in practice. I did an experiment where I compressed the rxdata for 5 commercial RMXP projects and found that difference was smaller than what the test data indicated, but very much significant if you looked at the data folders alone. On the downside was that the reducing in total project size was only 3-4% which is still better than nothing.

Yeah, I know that typically the Data directory really isn't that large. I was just thinking that it would work nicely on some bigger projects (50+ maps) or projects that use sizable maps (not that there are many who do that), it would be more of a boon. But to the normal RPG Maker user, it probably won't squeeze a whole lot out ... but every little bit helps, I guess!





EDIT: By the way, I put your old Remove Comments project in the Project Optimization thread I've been slowly building. It works really nicely.