Alex Rivera | Logout

Optimize memory usage of a collection of Strings in Java

Asked 2009-04-07T13:16:44.247
9

I have a large number of name - value pairs (approx 100k) that I need to store in some sort of cache (say a hash map) where the value is a string with an average of about 30k bytes in size.

Now I know for a fact that a large number of the values have exactly the same string data. In order to avoid having to allocate the identical string data several times, I would like to somehow reuse a previously allocated string thus consuming less memory. In addition this needs to be reasonably fast. i.e. scanning through all the previously allocated values one-by-one is not an option.

Any recommendations on how I could solve this problem?

java string memory-management
Edit
Report

3 Answers

4

String.intern is the obvious choice as Brian says. But if you don't want to intern across all the String in memory, you can use a Set to first see if the value is present. Here's untested code. You will have to work out removing from reverse map when removing from main

  class Map2<K, V> implements Map<K, V>
  {
    Map<K, V> _map = Maps.newHashMap();
    Set<V, V> _rev = Maps.newHashMap();

    V put(K k, V v) {
      if (_rev.containsKey(v)) {
        V prev = _rev.get(v);
        return _map.put(k, prev);
      } else {
        _rev.put(v, v);
        return _map.put(k,v);
      }
   }
answered 2009-04-07T13:49:07.973
0

You could compress the strings. A 30K string should get a good compression ratio. I wrote a hack to compress large String as an exercise, but you could use a byte[] of the compressed data to store the String.

A 30K character string will use about 60KB (2 bytes per character) so even using getBytes() is likely to be an improvement.

answered 2009-04-07T20:24:05.780
0

Do you actually need Strings, or do you just need any old CharSequence? If not, then consider implementing a "compact" CharSequence such as the one I suggest in the link.

answered 2009-04-07T20:27:05.973

Your Answer