serialization - How to deserialize 1GB of objects into Python faster than cPickle? -

-

we've got python-based web server unpickles number of large data files on startup using cpickle. data files (pickled using highest_protocol) around 0.4 gb on disk , load memory 1.2 gb of python objects -- takes 20 seconds. we're using python 2.6 on 64-bit windows machines.

the bottleneck not disk (it takes less 0.5s read data), memory allocation , object creation (there millions of objects being created). want reduce 20s decrease startup time.

is there way deserialize more 1gb of objects python faster cpickle (like 5-10x)? because execution time bound memory allocation , object creation, presume using unpickling technique such json wouldn't here.

i know interpreted languages have way save entire memory image disk file, can load memory in 1 go, without allocation/creation each object. there way this, or achieve similar, in python?

  1. try marshal module - it's internal (used byte-compiler) , intentionally not advertised much, faster. note doesn't serialize arbitrary instances pickle, builtin types (don't remember exact constraints, see docs). note format isn't stable.

  2. if need initialize multiple processes , can tolerate 1 process loaded, there elegant solution: load objects in 1 process, , nothing in except forking processes on demand. forking fast (copy on write) , shares memory between processes. [disclaimers: untested; unlike ruby, python ref counting trigger page copies useless if have huge objects and/or access small fraction of them.]

  3. if objects contain lots of raw data numpy arrays, can memory-map them faster startup. pytables these scenarios.

  4. if you'll use small part of objects, oo database (like zope's) can you. though if need them in memory, waste lots of overhead little gain. (never used one, might nonsense).

  5. maybe other python implementations can it? don't know, thought...


Comments

Post a Comment

Popular posts from this blog

android - Spacing between the stars of a rating bar? -

-
ho can set spacing beween stars? thats ratingbar: ratingbar = (ratingbar) inflater.inflate(r.layout.ratingbar, null); ratingbar.setlayoutparams(new layoutparams(layoutparams.wrap_content, layoutparams.wrap_content)); the layout: <?xml version="1.0" encoding="utf-8"?> <ratingbar xmlns:android="http://schemas.android.com/apk/res/android" style="@style/mystyle" android:layout_alignparentright="true" android:layout_height="19dp" android:layout_width="wrap_content" android:numstars="5"> </ratingbar> the style: <style name="mystyle" parent="@android:style/widget.ratingbar"> <item name="android:progressdrawable">@drawable/android_r2_ratingstar_yellow</item> <item name="android:indeterminatedrawable">@drawable/android_r2_ratingstar_grey</item> </style> thanks, cheers you have
Read more

aspxgridview - Devexpress grid - header filter does not work if column is initially hidden -

-
i'm using devexpress grid , i'm trying 'country' column display header filter properly: <dx:gridviewdatacolumn caption="country" fieldname="countryname" showincustomizationform="true" visible="false"> <settings allowheaderfilter="true"/> </dx:gridviewdatacolumn> if 'country' column set visible='true', header filter displayed should(it shows value option list). however, want 'country' column hidden, available in customization window(like in code above). in case, when column dragged outside customization window , grid , header filter clicked, javascript error encountered: element null element.addeventlistener(eventname, func, true); is known bug? there workarounds? set aspxgridview.settings.showheaderfilterbutton property true resolve problem. the following markup works fine me (i using dxperience 10.1.7): <asp:accessdatasource id="accessda
Read more

c# - How to execute a particular part of code asynchronously in a class -

-
i creating speech recognition engine that's going respond user commands.i have created button enable , disable speech recognition per user convenience.i have used dispose() of speech engine disable speech recoginition.here code private void button1_click(object sender, routedeventargs e) { engineon = !engineon; if (engineon) { speechengine = speech.createspeechengine(); //speech class creates , returns new speech engine. speechengine.audiolevelupdated += new eventhandler<audiolevelupdatedeventargs>(speechengine_audiolevelupdated); // use system's default microphone speechengine.setinputtodefaultaudiodevice(); speechengine.loadgrammar(new dictationgrammar()); // start listening speechengine.recognizeasync(recognizemode.multiple); } else { speechclass.myengine.dispose(); } } but disposal of speech obj
Read more