Login

Check Size of Object in memcached

Author:
deryck
Posted:
November 27, 2007
Language:
Python
Version:
.96
Tags:
memcache cache memcached
Score:
0 (after 0 ratings)

I've been working with a data set where a single object won't fit into memcached's 1 Mb slab limit. The two functions have been useful to me for debugging the size of the data structure once pickled, and if said pickled data structure is greater than 1 Mb.

These functions assume CACHE_BACKEND is memcached, obviously.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import memcache

try:
    from cStringIO import StringIO
except ImportError:
    from StringIO import StringIO

try:
    import cPickle as pickle
except ImportError:
    import pickle

def data_size(data):
    """Returns the size in bytes of the data passed in."""
    fp = StringIO()
    pickler = pickle.Pickler(fp, protocol=0)
    pickler.dump(data)
    val = fp.getvalue()
    return len(val)

def data_too_large(data):
    """Returns True if the data passed in is too large for memcached."""
    return data_size(data) >= memcache.SERVER_MAX_VALUE_LENGTH

More like this

  1. JSON instead of pickle for memcached by diverman 3 years, 3 months ago
  2. Improved Pickled Object Field by taavi223 5 years, 8 months ago
  3. Non-pickling locmem (in-process memory) cache backend by akaihola 4 years, 1 month ago
  4. Improved Pickled Object Field (Fixed for Django 1.2) by danielsokolowski 4 years, 2 months ago
  5. Pickled Object Field by obeattie 7 years, 4 months ago

Comments

Please login first before commenting.