Open in new window / Try shogun cloud
--- Log opened Thu Aug 04 00:00:15 2011
--- Day changed Thu Aug 04 2011
@sonney2kit is just that for the use cases I had it did not accelerate anything00:00
blackburnI became a speed maniac00:00
blackburnsonney2k: in DistanceMachine thread thing is kinda cumbersome00:01
blackburndon't you think so?00:02
@sonney2kblackburn, it is always cumbersome00:06
blackburnso no way to do it more generalized?00:06
@sonney2kthe problem is that you have to create some data structure that contains all meta-data00:06
@sonney2kblackburn, I would say that for some patterns there is00:07
blackburnyes, in java I would use hash-table with string key00:07
blackburnwhat to use there - don't know00:09
blackburnhmm I will try to describe thread for computing distances and kernels00:09
blackburnI feel upset we have no scalability haha00:09
blackburnokay time to change my lifestyle from 'nocturnal'00:15
blackburnsonney2k: nite00:15
@sonney2kblackburn, nite00:16
-!- blackburn [~blackburn@] has quit [Quit: Leaving.]00:17
-!- in3xes [~in3xes@] has joined #shogun09:35
-!- in3xes_ [~in3xes@] has joined #shogun09:47
-!- in3xes [~in3xes@] has quit [Ping timeout: 240 seconds]09:50
-!- f-x [~user@] has joined #shogun09:55
-!- in3xes__ [~in3xes@] has joined #shogun10:11
-!- in3xes_ [~in3xes@] has quit [Ping timeout: 240 seconds]10:14
-!- in3xes_ [~in3xes@] has joined #shogun10:34
-!- in3xes__ [~in3xes@] has quit [Ping timeout: 252 seconds]10:37
-!- in3xes__ [~in3xes@] has joined #shogun12:33
-!- in3xes_ [~in3xes@] has quit [Ping timeout: 276 seconds]12:37
-!- in3xes__ is now known as in3xes12:44
CIA-87shogun: Baozeng Ding master * r787f384 / src/interfaces/csharp_modular/swig_typemaps.i : add csout typemap -
CIA-87shogun: Soeren Sonnenburg master * rbeb108c / src/interfaces/csharp_modular/swig_typemaps.i :12:55
CIA-87shogun: Merge pull request #261 from sploving/master12:55
CIA-87shogun: add out typemap -
-!- in3xes_ [~in3xes@] has joined #shogun13:11
-!- in3xes [~in3xes@] has quit [Ping timeout: 240 seconds]13:14
-!- heiko [~heiko@] has joined #shogun13:41
@sonney2khi heiko13:44
heikohej sonney2k13:44
@sonney2kheiko, if you have time now we should discuss further steps.13:44
heikook, i have time :)13:44
heikowill send you some more patches while we are talking13:45
@sonney2kheiko, I think parameter versioning is next on the list right?13:45
heikobasically, yes13:45
heikomost classes have not registered their modsel parameters13:46
@sonney2kheiko, yeah but this is an independent construction site ...13:46
@sonney2kthe parameter add / etc stuff can be pretty intrusive again13:46
heikohow do you mean that?13:47
@sonney2kit affects all classes13:48
heikoyes, ok13:49
heikoalso there is this cached vector stuff13:49
heikoi did some SGVector replacements yesterday and noticed that this will be needed at a lot of points,13:49
heikoand also some CachedSparseVector etc stuff is needed13:49
heikobut ok, back to the versioning13:50
@sonney2kheiko, these things are all straight forward ...13:50
@sonney2kversioning is not so it seems13:50
@sonney2kI mean how shall we do it - have one *global* version or object specific?13:51
@sonney2kglobal has the advantage that only at one place the version needs to be changed13:51
@sonney2klocal - in all affected places13:51
@sonney2kplus we have to serialize this version too...13:52
heikoglobal sounds better to mee13:52
@sonney2kadvantage is fine grained control but huge overhead13:52
@sonney2kfor me too13:52
@sonney2kso global it is then13:52
@sonney2knevertheless we still have to save that version in the file once13:53
@sonney2kok say we have it saved somehow13:54
@sonney2kthen we only need to modify loading13:54
heikoand the loading then depends on the version13:55
@sonney2kso when loading - the idea is that we want to be able to always load older versions of the object13:55
heikowhat about if loading of a certain version simply only loads variables it knows13:56
@sonney2kso we need to attempt loading all things that are in the objects with version <= new_version13:56
heikoand the loading routine of a newser version just calls this and adds its own stuff13:56
@sonney2kheiko, I don't understand what you mean13:57
heikodo you want to write the version of each parameter to the file?13:58
@sonney2kheiko, I thought we just decided against that13:58
heikoah sorry, yes13:58
@sonney2kthat is what I meant above with more fine grained control...13:59
heikothen forget what i said above13:59
heikohow do you want to do it?13:59
@sonney2kif we don't attach the version to each parameter we have to assume one global version13:59
@sonney2kand the only thing we can do is to load everything <= GLOBABL_VERSION13:59
heikoand then write it once to a saved object13:59
heikobut this should be ok right14:00
heikoI mean, one has to update shogun if he gots objects with a more recent version14:00
-!- in3xes__ [~in3xes@] has joined #shogun14:00
@sonney2kand GLOBAL_VERSION only changes between releases of shogun - so it should be fine14:01
heikoand in every release, the loading routines then have to be updated14:01
heikoto work with versions before14:02
@sonney2kso the task would then be: a) write out GLOBAL_VERSION in saved files b) load GLOBAL_VERSION if exists (do this first!) and then c) modify load to load only objects <= GLOBAL_VERSION14:02
@sonney2kthis of course assumes that a) there are no variable type differences, b) there are only variable additions with sane initialization14:03
@sonney2kand no deletions :)14:03
@sonney2kwhich brings us to a new problem14:03
@sonney2khow do we know that the variable did not get deleted?14:03
@sonney2kI mean how do we support this?14:04
heikogood question14:04
@sonney2kwouldn't we then need the version attached to each variable?14:04
-!- in3xes_ [~in3xes@] has quit [Ping timeout: 276 seconds]14:04
heikodoes that solve the problem?14:04
@sonney2kif we had some introduced_in_version=0; removed_in_version=17;14:05
heikoI have to admit that I dont know how loading is done currently14:05
-!- in3xes_ [~in3xes@] has joined #shogun14:07
@sonney2kfor everything else we need some post processing functions14:07
@sonney2kthese could already now go in the load_serializable_post()14:08
@sonney2kso one would get the loaded_parameter_version in the parameters and can then do adjustments as needed14:09
-!- in3xes__ [~in3xes@] has quit [Ping timeout: 240 seconds]14:10
heikoi am still a bit unsure where to start14:11
heikoI mean saving the version is easy14:11
@sonney2kthere is still one more problem14:11
heikobut for loading, this has to be different for each new version, right?14:11
@sonney2kwhen do do loading and some variable is renamed / changed type - we still need the old ptrs in the object14:12
@sonney2kotherwise we cannot load them14:12
heikooh, yes14:12
heikommh, how to solve this?14:13
heikoperhaps by temporary void pointers and then casting them depending on old type14:13
@sonney2kyeah but it can be even worse14:14
@sonney2kthink of 3 versions of a file14:14
@sonney2kone has double* v; the other int* v; the other SGVector v;14:15
-!- in3xes__ [~in3xes@] has joined #shogun14:15
heikoand then?14:16
@sonney2kheiko, ok I think void* would work - we just need to load all objects - have them in some void* DynArray and then it can be used to construct the final thing14:16
heikoand for every element, we have to check version and then evtl do conversion stuff depending on changes14:18
heikodeletion could also be treated there: simply do nothing with the data14:18
-!- in3xes_ [~in3xes@] has quit [Ping timeout: 240 seconds]14:18
heikosame with type changes14:18
-!- in3xes_ [~in3xes@] has joined #shogun14:19
@sonney2kheiko, Re deletion - it might be worthwhile to not even load deleted variables14:19
@sonney2kI mean it could happen that these are *huge* arrays14:19
heikommh, true14:20
heikothen the list of names has to be checked before loading14:20
heikoand only load if name still exists14:20
-!- in3xes__ [~in3xes@] has quit [Ping timeout: 240 seconds]14:22
heikosonney2k, 2 mins afk14:24
@sonney2kthe only thing is that this is getting complicated14:24
@sonney2kit would really be much easier to always load everything that is <= GLOBAL_VERSION14:25
@sonney2kthen, when each parameter knows its loaded_version one can do whatever changes in the load_serializable_post()14:26
@sonney2kthis would include renames, deletions, type changes - everything14:27
heikobut is it so hard to check whether a parameter is still there?14:28
@sonney2kso when loading it needs to check the version in the file and the removed_in_version parameter14:29
heikoI mean just implement a method for this and then eventually sip14:29
@sonney2kif it is > removed version14:29
@sonney2kit will put it in some void* dynarray with the associated name14:29
@sonney2kheiko, how do you check it? I mean how do you distinguish this from rename or type change?14:30
heikommh also true14:30
heikobut the renamings have to be checked at some point14:30
heikoa map is needed anyways14:31
heikowith renamings /type changes14:31
heikoi mean, while iterating over the DynArray, the steps are:14:35
heiko1) check version, if smaller than current, go on14:35
heiko2) check map for name14:36
heiko(we could do a method for that, that simply is identity when name has not changed, and new name in other case)14:36
@sonney2kwait, why does one have to check for renaming?14:37
@sonney2kohh and how do we treat name changes like v -> vector -> v ?14:37
heikofor each version, there has to be a map, not?14:38
@sonney2kI mean some loop behavior or whatever clash14:38
heikoso when you have: 1.0: v, 2.0:vector, 3.0:v14:38
@sonney2kok that would work14:38
heikoand you load while you are in 4.014:38
heikoyou check version and then use map for that version14:38
heikoor even better14:38
heikomaps call themselves14:39
heikosorry, i meant one map for each version that contains the change14:39
heikothen you know change from 1.0 to 2.0 and from 2.0 to 3.014:39
heikoand do not need a new map fpr 1.0 to 3.014:40
heikosimply transitive14:40
@sonney2kthat makes a lot of sense14:40
heikoand all this could be in one method that is handed the name and the version and this method does all the conversion automatucally14:40
@sonney2kbut when you do a map anyways - could one instead of using the post function also attach a translate() function?14:40
heikoyes, perhaps better ?14:41
@sonney2kI only see one problem (as always)14:41
@sonney2kdependencies between loaded objects14:42
@sonney2kbut would still work14:42
@sonney2kone would then first load everything14:42
@sonney2kand then call all these translate functions14:42
@sonney2kfor version 0, then for 1 ... up to GLOBAL_VER14:43
@sonney2kactually for loaded_ver ... GLOBAL_VER14:44
heikoand i mean, couldnt deletion of variables also be treated that way?14:44
@sonney2keverything would work this way14:44
heikobut stuff has to be loaded first14:45
heikoeven if not used14:45
@sonney2kbut that is ok I think14:45
heikowhat about something like, only load names/type of variables first, but not data14:45
heikothen do all the mapping, and in the last step load data14:45
@sonney2kI mean we guarantee 100% efficiency for the latest version14:45
heikoyes also true14:46
@sonney2kand if one wants to transition things - one only has to do this once14:46
heikoolder versions slowly get slower :)14:46
heikoyes, then save it again14:46
heikowell, then it would be ok to always load everything, i mean there might be extreme cases  ....14:47
heikobut i am unsure whether this really will happen14:47
@sonney2kyeah but extreme cases require extreme measures :)14:47
@bettyboosmart sonney2k14:47
@sonney2ke.g. one could then compile 10 different shogun versions14:47
heiko hehe ;)14:47
@sonney2kand transition from one version to another ...14:47
@sonney2kso I suggest you try to first write down the map and global version nr14:50
heikoat first, will only to an identity map to get things working14:51
heikostill, my picture of all this is very abstract - not really concrete where to do things14:51
@sonney2kwell or if sth is not found in the map then it is assumed unchanged14:51
heikobut ok, will reed some code now14:51
heikoah ok14:52
heikonot found on the key side14:52
heikoand if not found on value side, it was deleted14:52
heikodo we actually have a map data structure?14:52
@sonney2kunfortunately not14:55
@sonney2kI mean in the end a sorted list would be sufficent14:56
@sonney2ksome struct sorted by variable name14:57
@sonney2kheiko, ahh and if no hash map exists then one assumes nothing changed14:57
@sonney2kheiko, and to dump this hash map it would be best if it was derived from SGObject14:58
heikobut its no hash map, or?14:58
@sonney2kheiko, but we only need  to look up 'variable_name' for version X14:59
@sonney2kand then we get the struct with whatever...14:59
-!- in3xes__ [~in3xes@] has joined #shogun15:03
-!- in3xes_ [~in3xes@] has quit [Ping timeout: 240 seconds]15:07
-!- in3xes_ [~in3xes@] has joined #shogun15:15
heikosonney2k, do you think it is safe to make computer_sparse_feature_vector a const method?15:15
heikocompiles at least15:15
-!- in3xes__ [~in3xes@] has quit [Ping timeout: 276 seconds]15:19
heikosonney2k, because i just realilsed that i assumed a feature matrix for the copy_subset method (needed for model selection), but I think it is more wise to call get_feature_vector instead. However this needs the method to be const because copy_subset is const15:19
heikooh no15:21
heikothe cache, well then it is not possible :)15:21
-!- sploving1 [~sploving@] has joined #shogun15:47
-!- f-x [~user@] has quit [Ping timeout: 260 seconds]15:55
-!- in3xes__ [~in3xes@] has joined #shogun16:06
-!- in3xes_ [~in3xes@] has quit [Ping timeout: 240 seconds]16:10
-!- blackburn [~blackburn@] has joined #shogun16:14
blackburnheiko: guten abend ;)16:16
blackburndo you need any efficient CSet?16:16
@sonney2ksploving1, how is it going with the c# typemaps?16:22
@sonney2kblackburn, he now needs a map ohh and we need an LRU cache :)16:22
blackburnhey where might you use it?16:23
@sonney2kfor kernel caching and for caching the on-demand computed feature vectors (from the preprocessors)16:23
@sonney2kI know thought that currently basically everyone uses a feature matrix - in memory :)16:24
sploving1sonney2k, not work now. len = ((sizeof($input)) / (sizeof($input[0]))); does not work16:25
sploving1but it has no realvector error now16:25
@sonney2ksploving1,and if you remove the sizeof stuff? don't you get the realvector error again?16:27
@sonney2kI mean you didn't change the in typemap or?16:27
sploving1the csout typemap can remove the realvector error16:28
sploving1but now it does not pring any correct for the get_labels16:28
@sonney2ksploving1, regarding the len thing above - can't you use a macro with the defined type such that you can cast appropriately and use sizeof(SG_TYPE) or so?16:30
sploving1sonney2k, for double[] in c#, it is converted to be  double* in the wrap.cxx file, so it is not easy to just use the macro16:33
@sonney2kand when you use CSHARPTYPE, CTYPE?16:34
sploving1has something about the array size, but I am still trying to understand it16:34
-!- f-x [~user@] has joined #shogun16:35
sploving1CSHAPTYPE is c# type, and CTYPE is c type16:36
@sonney2ksploving1, ok - just to understand. you have to marshal the array to some raw thing - how do you unmarshal it?16:37
heikoblackburn, guten Abend :) but still sonny here16:38
sploving1I think marshal includes two: passting array from .net to C, and passing array from C to .Ne16:38
sploving1they have inattributes and outattributes16:39
sploving1inattrbutes is used to convertes C# to C, and outattributtes converts C to C#, this is what i understand16:40
@sonney2ksploving1, so when you do MarshalAs(UnmanagedType.LPArray)16:45
@sonney2kit will create a pointer to what?16:46
sploving1sonney2k, yeap.16:47
sploving1double[] -> double*16:47
@sonney2ksploving1, shouldn't you use then MarshalAs(UnmanagedType.LPArray, ArraySubType = UnmanagedType.R8) ?17:02
@sonney2ksploving1, at least says 8 byte float17:03
@sonney2ksploving1, btw would your typemap now work if you hard code a length of say 3 and just use this?17:05
sploving1array = SG_MALLOC(SGTYPE, len); to be array = SG_MALLOC(SGTYPE, 3);?17:06
sploving1I have a try17:08
-!- sploving1 [~sploving@] has left #shogun []17:08
-!- sploving1 [~sploving@] has joined #shogun17:15
sploving1sonney2k, no.17:15
sploving1double[] y = new double[3] {5, 2, 3};17:15
sploving1the result is: -3.19979336298487E-4417:16
-!- sploving1 [~sploving@] has left #shogun []17:18
f-xsonney2k: how should i use SG_MALLOC if the constructor should be called after initialization?17:18
@sonney2kf-x no way - just don't do such things17:19
-!- heiko [~heiko@] has quit [Ping timeout: 258 seconds]17:28
f-xsonney2k: so could you help me out on this one thing17:33
@sonney2kf-x, which?17:34
f-xto make the ring of examples suitable for all types17:34
f-xi thought it would be ok to call the constructor of type T when the ring is initialized17:35
f-xi.e., there will be 1024 pre-initialized T-objects17:35
f-xwhere T=float64_t*, or a sparse vector entry etc17:35
f-xor maybe in my case even a VwExample17:35
f-xsonney2k: is there a way to do a generic initialization without new[]?17:36
f-xok, wait.. i'll try something and get back to you if there's a problem17:38
f-xsonney2k: yes - the code would be simpler if i could call the constructor during initialization.. is using new[] not allowed now?17:51
-!- in3xes_ [~in3xes@] has joined #shogun18:02
-!- in3xes_ [~in3xes@] has quit [Client Quit]18:04
-!- in3xes__ [~in3xes@] has quit [Ping timeout: 276 seconds]18:07
-!- f-x [~user@] has quit [Ping timeout: 260 seconds]19:02
-!- f-x [~user@] has joined #shogun19:39
@sonney2kf-x, well you can use CALLOC to have them all 020:14
blackburnsonney2k: can I introduce method not available in interfaces but only in libshogun? how?20:15
@sonney2kyou mean %ignore?20:16
@sonney2kin .i files?20:16
blackburnhmm seems yes20:16
blackburnsonney2k: so you don't mind realloc for simplefeatures, right?20:18
@sonney2kblackburn, no - that was one of the reasons for the SG_MALLOC transition20:18
f-xsonney2k: How do I allocate memory for the labels inside?20:19
f-xsay it's class a { ptr* b }; does b get allocated to any value when I calloc an obj of type a?20:20
@sonney2kyes. NULL20:22
f-xin my case for vw it's like 'class example { label* b; }', and the ring stores example objects.. i would like a new label to be created when the example is created20:26
f-xso far creating this within the constructor of example has been the simplest way i could see20:27
f-xsonney2k: what's the rule for using new/SG_MALLOC now? for single object, use new and for multiple use SG_MALLOC?20:35
@sonney2kf-x, the rule is just don't use new[]20:40
@sonney2kf-x, allocating objects in constructor is anyway not a great idea...20:40
-!- f-x [~user@] has quit [Ping timeout: 260 seconds]20:45
* serialhex *LOVES* *nix!!!!20:53
-!- f-x [~user@] has joined #shogun20:56
-!- heiko [~heiko@] has joined #shogun21:07
-!- heiko [~heiko@] has quit [Ping timeout: 258 seconds]21:26
CIA-87shogun: Heiko Strathmann master * r51aed9e / (16 files): made get_dim_feature_space method const -
CIA-87shogun: Heiko Strathmann master * r4d85fba / (2 files): made get_dim_feature_space method const -
CIA-87shogun: Heiko Strathmann master * r332c8e4 / src/shogun/lib/DataType.h : (hopefully) removed some warnings regarding initialization -
CIA-87shogun: Heiko Strathmann master * r765bfc3 / src/shogun/lib/DataType.h : removed another compiler warning -
CIA-87shogun: Soeren Sonnenburg master * rbf0c2af / (17 files in 2 dirs):23:08
CIA-87shogun: Merge pull request #262 from karlnapf/master23:08
CIA-87shogun: made get_dim_feature_space method const -
@sonney2kf-x, your patch doesn't apply23:24
f-xsonney2k: what do you mean?23:24
@sonney2kf-x, it needs to be rebased agains master23:24
f-xok.. i see you already applied heiko's DataType.h patch, probably conflicts23:25
f-xi'll send another request23:25
@sonney2ktake your time - I am too tired now will look at it tomorrow23:26
f-xok, sent anyway23:28
-!- f-x [~user@] has quit [Remote host closed the connection]23:40
--- Log closed Fri Aug 05 00:00:55 2011