Login

Check Size of Object in memcached

Author:
deryck
Posted:
November 27, 2007
Language:
Python
Version:
.96
Score:
0 (after 0 ratings)

I've been working with a data set where a single object won't fit into memcached's 1 Mb slab limit. The two functions have been useful to me for debugging the size of the data structure once pickled, and if said pickled data structure is greater than 1 Mb.

These functions assume CACHE_BACKEND is memcached, obviously.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import memcache

try:
    from cStringIO import StringIO
except ImportError:
    from StringIO import StringIO

try:
    import cPickle as pickle
except ImportError:
    import pickle

def data_size(data):
    """Returns the size in bytes of the data passed in."""
    fp = StringIO()
    pickler = pickle.Pickler(fp, protocol=0)
    pickler.dump(data)
    val = fp.getvalue()
    return len(val)

def data_too_large(data):
    """Returns True if the data passed in is too large for memcached."""
    return data_size(data) >= memcache.SERVER_MAX_VALUE_LENGTH

More like this

  1. Template tag - list punctuation for a list of items by shapiromatron 2 months, 2 weeks ago
  2. JSONRequestMiddleware adds a .json() method to your HttpRequests by cdcarter 2 months, 3 weeks ago
  3. Serializer factory with Django Rest Framework by julio 9 months, 2 weeks ago
  4. Image compression before saving the new model / work with JPG, PNG by Schleidens 10 months, 1 week ago
  5. Help text hyperlinks by sa2812 11 months ago

Comments

Please login first before commenting.