Open in new window / Try shogun cloud
--- Log opened Sun Apr 13 00:00:03 2014
-!- khalednasr [~k.nasr92@41.69.160.123] has quit [Quit: Leaving]00:10
-!- HeikoS [~heiko@90.201.62.238] has joined #shogun00:20
-!- mode/#shogun [+o HeikoS] by ChanServ00:20
-!- lisitsyn [~qdrgsm@109-226-124-25.clients.tlt.100megabit.ru] has quit [Quit: Leaving.]00:30
-!- lisitsyn [~qdrgsm@109-226-124-25.clients.tlt.100megabit.ru] has joined #shogun00:32
-!- diver-in-sky_ [uid28350@gateway/web/irccloud.com/x-mddxzutqvfaondva] has quit [Quit: Connection closed for inactivity]00:35
-!- lambday [67157d51@gateway/web/freenode/ip.103.21.125.81] has joined #shogun00:40
lambdayHeikoS: hi00:40
lambdaythere?00:40
@HeikoSlambday: hey00:40
@HeikoSyes00:40
@HeikoShow are things?00:40
@HeikoSin fact was just about to write you an email00:40
lambdayHeikoS: a bit slow00:40
lambday:D00:40
lambdayHeikoS: this research took an exciting turn!00:41
lambdayHeikoS: will it make into jmlr?00:41
@HeikoSlambday: on the long run that might happen00:42
@HeikoSlambday: for now, we have some other things in mind. did you see the abstract?00:42
lambdayHeikoS: yeah00:42
lambdayI didn't know what to comment :(00:42
@HeikoSlambday: ah yeah read the email00:43
@HeikoSlambday: you dont have to for now00:43
@HeikoSyeah JMLR is the target goal00:43
@HeikoSwhich would be amazing right?00:43
lambdayyeah man!!00:43
lambdaydidn't even think that!00:43
@HeikoShaha :)00:44
lambdayso all the experiments should be redone and we'll use our shogun code00:44
@HeikoSyeah exactly00:44
lambdayramstein song ;)00:44
@HeikoSthats what my email will be about00:44
@HeikoSsome things have to be changed00:44
lambdayokay... I was confused a bit regarding how to fit things exactly...00:44
lambdaywill have to discuss after reading your mail00:45
@HeikoSwhat do you mean?00:45
lisitsynnight chat!00:45
lisitsyn:D00:45
lambdaylisitsyn: nightbirds... :D00:45
lambday4:15AM here00:45
@HeikoSlambday: for pre-gsoc, we should focus on getting this testing stuff done, but feature selection will be very similar in some ways00:45
lisitsyn% - 2hrs00:45
@HeikoSlambday: ah man, you dont sleep ;)00:45
lambdayHeikoS: I wake up at 1 :D00:45
@HeikoSI see :)00:46
lambdayHeikoS: alright... I am on it00:46
@HeikoSlambday:  Ill just write mail. 1 sec00:46
lambdayHeikoS: I was trying to draw a class diagram in this refactored scenario... not quite complete though00:46
@HeikoSor in fact, 10 mins if you have :)00:46
lambdayyeah00:46
lambdayyeah I am here00:46
@HeikoSlambday: ok sent01:04
lambdayHeikoS: checking01:04
lambdayHeikoS: alright... I get the points01:09
lambdaythere might be something that I am not currently seeing at this moment.. will discuss when I get there01:10
lambdayand for streaming stuffs.. you meant that we should always rely on online mean/var algorithms, right?01:11
lambdayso no storing but keep on computing01:11
lambdaysum of kernel values without diag seems fine... two methods is better than flag01:12
@HeikoSyeah01:13
@HeikoSlambday: and then change quadratic time mmd thing01:13
@HeikoSto just call sum01:13
@HeikoSshould be quite a bit faster when a custom kernel is used01:13
@HeikoSfor example for the permutation test01:13
lambdayhmm yeah01:14
lambdayHeikoS: I am a bit confused about what to do with H_A variances...01:15
lambdayI mean, for example, we would have methods that computes \sigma^2_{k, A} in eq (2) of Dino's writeup01:16
lambdayright?01:16
lambdaybut when do we actually use that?01:17
@HeikoSfor kernel selection01:17
lambdayokay01:17
@HeikoSlambday: dont worry about that for now01:17
lambdayalright01:17
@HeikoSfor the testing, we just need the variance under the null01:17
lambdayso p-value threshold they deal with var under null01:18
lambdayyeah01:18
@HeikoSyes exactly01:18
@HeikoSso need to correct a few things,  now, but keep the old ways in this deprecated methods01:18
lambdayokay01:19
lambdayand what do we check our new codes against? should we make a matlab implementation or so?01:19
@HeikoSlambday: I might have actually been wrong with one thing01:19
lambdayHeikoS: what?01:19
@HeikoSthe within block direct restimation contains a matrix product01:19
lambdayyeah01:20
@HeikoSbut maybe its possible to do that wihtout storing the matrix somehow, dont know, maybe think about it01:20
lambdayummm... but its just the kernel matrix for the current block, right?01:21
lambdayI mean, how huge that would be?01:21
lambdayand for the sum.. we need to specify blocks... because the kernel matrix is formed with merged samples of p_and_q01:25
lambdayfor one part we don't need diag elements.. for some parts we need01:25
lambdayanyway I'll think about it01:26
@HeikoSlambday: think big01:31
@HeikoSlambday: blocks might be larger than 10000 which is maximum matrix size one can store01:32
@HeikoSlambday: also efficiency! if you dont have to store things, dont store them if its not slower01:32
@HeikoSand yes, feel free to adjust the interface of this sum operation to your needs01:33
@HeikoSlambday: maybe it should even be average operation, have to think about a clean interface for this and via not using too many if then else statements in the middle01:33
@HeikoSlambday: I will leave soon, let me know if you ahve more quetsions01:34
@HeikoSlambday: and yes, another python draft of things to uniut test against and for us to proofread (easier in python than in c++) might be very helpful01:34
lambdayHeikoS: alright...01:36
lambdayHeikoS: I'll think about this a bit.. will mail you regarding further queries01:36
@HeikoSyeah pls ask many questions!01:36
lambdayfor permutation test we got to sacrifice some power.. gotta store stuffs else it would be really slow01:38
@HeikoSlambday: what do you mean?01:38
@HeikoSlambday: ah yes, of course01:38
@HeikoSlambday: so the thing about the permutation test is:01:38
@HeikoSif possible: store, and dont call ::kernel in the loop, but rather let CCustomKernel loop over matrix01:39
@HeikoSso try to precompute matrix in any case01:39
@HeikoSlike its now, it replaces the kernel by a custom kernel for the bootstrapping, you probably saw that01:39
lambdayyeah01:39
@HeikoSlambday: but if its not possible to allocate the memory for that01:39
@HeikoSit should still be possible to do the test01:40
@HeikoSso have a boolean setter do de-activate that01:40
lambdayalright01:40
@HeikoSand in fact, it would be cool if the precoimputing method would catch out of mem errors, print a warning that it will switch, and then do on-the-fly computing01:40
@HeikoSand then if one sets this precomputign flag to false, the warning never appears01:41
lambdayyeah that would be cool!01:41
@HeikoSwith that, the test should work for 100000 and more points (we should test that)01:41
@HeikoSyou can do unit test with 3 bootstrap iterations for such cases ;)01:41
lambday:D01:41
lambdayalright... so I am thinking that starting with quadratic time mmd and its related dependencies (like the kernel sum/mean computation etc) would be good01:43
lambdayHeikoS: then we'd deal with streaming cases01:44
lambdayHeikoS: btw with the refactored design with many methods pulled out from LinearTimeMMD to StreamingMMD, I think we'd lose those preallocated vector comprised methods... but if those can be done in parallel then that's a sacrifice worth of01:45
@HeikoSlambday:  which methods do you mean?01:47
@HeikoSfor streaming, no need to store anything right?01:47
@HeikoSjust the block-size01:47
lambdayHeikoS: yeah.. wait let me show01:47
@HeikoSwhere data has to be stored01:47
@HeikoSand kernel should probably not be stored (one test only computes statistics once, one does twice, one has this matrix product where we have to see)01:47
lambdayHeikoS: yeah... but its not related to that01:48
@HeikoSlambday: multicore parallelisation might be interesting, but not for now btw01:48
lambdayokay01:49
@HeikoSlambday: first goal is to make things stable and efficient01:49
@HeikoSthen we can speed up01:49
@HeikoSlambday: keep in mind precomputing kernel matrices is already parallelised01:49
lambdayyeah what I was saying was related to multiple kernel stuff... in the method compute_statistic_and_Q01:50
@HeikoSlambday: ah yeah01:50
lambdayso it has these preallocated vectors...01:50
@HeikoS(that might be good to do in parallel in fact)01:50
@HeikoSah thats fine01:50
lambdayyeah01:50
@HeikoSnot many kernels so storing multiple there is pok01:50
@HeikoSand yeah, this in fact SHOULD be done in parallel if easily possible01:51
lambdayyeah...01:51
@HeikoSlambday: but not too important, getting things to work is the first step01:51
@HeikoSand have thorough unit tests01:51
@HeikoSso that parallelisation can be tested easily, which makes it easier to develop it01:52
lambdayokay... so you agree with starting with quadratic time mmd ?01:52
@HeikoSyes!01:52
lambdayalthough it may be useful to first think it all before changing - I'll do some planning tomorrow01:53
@HeikoSlambday: yeah01:53
@HeikoSlambday: planning is good01:53
@HeikoSbut maybe just to the point where you know how to do things in a way that its not woo much work to change to multicore01:54
@HeikoSspending time with pthreads would be too much ;)01:54
lambdayhehe01:54
lambdayoh and for all these tests, we'll have two settings each, right? full and incomplete01:54
@HeikoSyep01:54
lambdayso enum?01:54
@HeikoSthe incomplete ones are not important for now though01:55
@HeikoSyeah enum01:55
lambdayand additional variance_under_alternative() methods for all01:56
@HeikoSyeah, this can sometimes probably be computed on the fly with other thingfs01:58
@HeikoSsee linear time mmd01:58
lambdayyeah01:58
lambdayI meant whether we need a wrapper like compute_variance() that internally calls compute_statistic_and_variance()01:59
@HeikoSlambday: I think its always a good idea to modularise02:05
@HeikoSlambday: ok going to sleep now, see you later!02:05
lambdayHeikoS: okay02:05
lambdayHeikoS: yeah me too!02:05
lambdaysee you :)02:06
lambdaygn02:06
@HeikoSlambday: you should ;)02:06
lambday:D02:06
@HeikoSbye, good night!02:06
lambdaybye02:06
-!- lambday [67157d51@gateway/web/freenode/ip.103.21.125.81] has quit []02:06
@HeikoSlisitsyn: still there?02:15
-!- HeikoS [~heiko@90.201.62.238] has quit [Ping timeout: 245 seconds]02:21
-!- lisitsyn [~qdrgsm@109-226-124-25.clients.tlt.100megabit.ru] has quit [Ping timeout: 276 seconds]02:46
-!- lisitsyn [~qdrgsm@109-226-124-25.clients.tlt.100megabit.ru] has joined #shogun03:03
shogun-buildbotbuild #673 of nightly_all is complete: Failure [failed compile test]  Build details are at http://buildbot.shogun-toolbox.org/builders/nightly_all/builds/67303:18
-!- khalednasr [~k.nasr92@41.69.137.37] has joined #shogun04:06
-!- khalednasr [~k.nasr92@41.69.137.37] has quit [Client Quit]04:11
-!- kislay|exams [~Abhijeet@117.239.94.112] has quit [Read error: Connection reset by peer]05:04
-!- Netsplit *.net <-> *.split quits: lisitsyn, sonne|work, pfm_, shogun-buildbot, @sonney2k, vvu|Log, soumyaC, @wiking05:33
-!- Netsplit *.net <-> *.split quits: chakri, [BNC]dhruv13j, @besser82, zxtx, @ChanServ, naywhayare05:33
-!- Netsplit over, joins: vvu|Log05:39
-!- sonney2k [~shogun@7nn.de] has joined #shogun07:16
-!- besser82 [quassel@gateway/shell/bnc4free/session] has joined #shogun07:16
-!- shogun-buildbot [~shogun-bu@7nn.de] has joined #shogun07:16
-!- zxtx [~zxtx@sunnythinking.org] has joined #shogun07:16
-!- naywhayare [~ryan@spoon.lugatgt.org] has joined #shogun07:16
-!- ServerMode/#shogun [+o sonney2k] by sendak.freenode.net07:16
-!- lisitsyn [~qdrgsm@109-226-124-25.clients.tlt.100megabit.ru] has joined #shogun07:18
-!- [BNC]dhruv13j [dhruv13j@gateway/shell/bnc4free/x-zhvbyfoelcvhkhls] has joined #shogun07:18
-!- chakri [srichakrad@gateway/shell/bnc4free/x-cdbhudoxjppmjuli] has joined #shogun07:18
-!- wiking [~wiking@huwico/staff/wiking] has joined #shogun07:18
-!- ServerMode/#shogun [+o wiking] by sendak.freenode.net07:18
-!- sonne|work [~sonnenbu@91-64-78-194-dynip.superkabel.de] has joined #shogun07:18
-!- pfm_ [~hurr@hamilton.maths.tcd.ie] has joined #shogun07:18
-!- besser82 [quassel@gateway/shell/bnc4free/session] has quit [Changing host]07:19
-!- besser82 [quassel@gateway/shell/bnc4free/x-sjwrsczdlqfancsq] has joined #shogun07:19
-!- aditidatta [~aditidatt@117.194.118.32] has joined #shogun08:50
-!- aditidatta [~aditidatt@117.194.118.32] has quit [Client Quit]08:52
-!- ChanServ [ChanServ@services.] has joined #shogun09:18
-!- ServerMode/#shogun [+o ChanServ] by sendak.freenode.net09:18
-!- besser82 is now known as Guest4103809:18
-!- Guest41038 is now known as besser8209:54
-!- besser82 [quassel@gateway/shell/bnc4free/x-sjwrsczdlqfancsq] has quit [Quit: freedom, friends, features, first ---> fedoraproject.org]09:54
-!- besser82 [quassel@fedora/besser82] has joined #shogun09:56
-!- mode/#shogun [+o besser82] by ChanServ09:56
-!- iglesiasg [~iglesiasg@524AE0A7.cm-4-3d.dynamic.ziggo.nl] has joined #shogun10:10
-!- mode/#shogun [+o iglesiasg] by ChanServ10:11
-!- sunil1337 [0e8bd057@gateway/web/freenode/ip.14.139.208.87] has joined #shogun11:07
-!- Netsplit *.net <-> *.split quits: @ChanServ11:18
-!- Netsplit over, joins: @ChanServ11:19
-!- PirosB3 [~pirosb3@host238-17-dynamic.245-95-r.retail.telecomitalia.it] has joined #shogun11:29
-!- iglesiasg [~iglesiasg@524AE0A7.cm-4-3d.dynamic.ziggo.nl] has quit [Quit: Leaving]11:43
-!- sunil1337 [0e8bd057@gateway/web/freenode/ip.14.139.208.87] has left #shogun []12:03
-!- lambday [67157d51@gateway/web/freenode/ip.103.21.125.81] has joined #shogun14:02
-!- pl8787 [9fe22b60@gateway/web/freenode/ip.159.226.43.96] has joined #shogun15:21
-!- pl8787 [9fe22b60@gateway/web/freenode/ip.159.226.43.96] has quit [Quit: Page closed]15:27
-!- pl8787 [9fe22b60@gateway/web/freenode/ip.159.226.43.96] has joined #shogun15:38
-!- khalednasr [~k.nasr92@41.69.186.4] has joined #shogun16:22
@wikinglisitsyn: ping16:36
lisitsynwiking: pong16:36
@wikinglisitsyn: 1) what happend with kaggle IO?16:36
lisitsynwiking: not ready yet16:37
lisitsynbut I added downloaders16:37
lisitsynwiking: will work on that soon16:37
@wiking2) https://scan.coverity.com16:38
@wikingfirst login with github16:38
lisitsynhmm16:38
@wikinghttps://scan.coverity.com/projects/1763?tab=Overview16:39
@wikingdo u see there something like request more access?16:39
lisitsynwiking: no16:39
lisitsynjust not authorized16:40
@wikingshit16:40
@wikingok i guess pathen it's because it hasn't been fully configured...16:41
lisitsynwiking: did you add shogun already?16:41
@wikingyeah i'm working on that16:42
@wikingbut configuring with our crazy travis config16:42
@wikingis not the easiest task16:42
@wikinglisitsyn: but check out in the meanwhile the whole coverity stuff16:42
@wikinglooks like a tool that we would benefit a lot from16:43
lisitsynwiking: yeah16:43
@wikingbut then again we should see the result first...16:43
lisitsyntrue16:43
lisitsynI have no idea yet16:43
@wikingi'll try to first do then the analysis locally16:44
@wikingdoesn't have to be done via travis necessarily16:44
@wikingmaybe we could even integrate a new buildbot task16:45
@wikingthat runs every week once on the develop branch16:45
@wikingas otherwise we would need to maintain a new branch16:46
@wikinglisitsyn: what do you reckon? i feel like maybe actually integrating this to buildbot as a weekly once thing would be better than to maintain a new branch16:47
@wikinglisitsyn: https://scan.coverity.com/faq#frequency16:47
lisitsynwiking: yeah sounds better for me too16:48
lisitsynwell all the thing is now like an overkill16:48
@wikingwhat do you mean?16:49
lisitsynI mean we don't really need to do static analysis for each build and so on16:49
@wikingyeah we can change that as well16:49
@wikinglisitsyn: to a twice a week thing16:49
-!- PirosB3 [~pirosb3@host238-17-dynamic.245-95-r.retail.telecomitalia.it] has quit [Quit: PirosB3]17:14
-!- PirosB3 [~pirosb3@host238-17-dynamic.245-95-r.retail.telecomitalia.it] has joined #shogun17:20
-!- iglesiasg [~iglesiasg@524AE0A7.cm-4-3d.dynamic.ziggo.nl] has joined #shogun17:51
-!- mode/#shogun [+o iglesiasg] by ChanServ17:52
-!- PirosB3 [~pirosb3@host238-17-dynamic.245-95-r.retail.telecomitalia.it] has quit [Quit: PirosB3]17:58
-!- lambday [67157d51@gateway/web/freenode/ip.103.21.125.81] has quit [Ping timeout: 240 seconds]18:34
-!- khalednasr [~k.nasr92@41.69.186.4] has quit [Quit: Leaving]19:30
-!- iglesiasg [~iglesiasg@524AE0A7.cm-4-3d.dynamic.ziggo.nl] has quit [Quit: Leaving]20:15
@wikinglisitsyn: https://scan.coverity.com/projects/176320:37
@wikinglisitsyn: do u see the request button now?20:37
lisitsynYou are not authorized to access this page.20:39
lisitsynwiking: ^20:39
@wikingdamn wtf20:39
-!- shogun-notifier- [~irker@7nn.de] has joined #shogun21:51
shogun-notifier-shogun: abinashpanda :develop * 6b8f833 / / (3 files): https://github.com/shogun-toolbox/shogun/commit/6b8f833e5253c1c7be340fae5464575c8d02063921:51
shogun-notifier-shogun: CUAIFile class added to parse and write UAI file format21:51
shogun-notifier-shogun: tklein23 :develop * 0ade087 / / (3 files): https://github.com/shogun-toolbox/shogun/commit/0ade087af368e4d14772b8db4fd1248e1a3956c621:51
shogun-notifier-shogun: Merge pull request #2019 from abinashpanda/develop21:51
shogun-notifier-shogun:21:51
shogun-notifier-shogun: Created CUAIFile class (Issue #1913) to save and load file in UAI file format.21:51
-!- thoralf [~thoralf@91-65-136-198-dynip.superkabel.de] has joined #shogun23:03
thoralfGood morning.23:03
lisitsynthoralf: yeah that's a nice winter day :D23:14
thoralflisitsyn: Still cold in russia? ;)23:21
lisitsynthoralf: nah I mean if it is morning it could be a winter too ;)23:21
thoralfI know.  I just needed an appropriate answer to you. :)23:23
thoralfIt worked. :)23:23
lisitsynthoralf: apart from that it is around 0C23:24
thoralfReally?23:25
thoralfWe had 20 degree.23:25
thoralf16 degree according to the forecast.23:25
thoralf8 degree tonight.23:25
lisitsynthoralf: it is on its way from -30 to +3023:26
lisitsyngradually23:26
lisitsynhalf way done23:26
thoralfThat's really bad.23:26
thoralfCome to Berlin. ;)23:27
lisitsynthoralf: why bad? it is pretty cool actually23:27
thoralfIt's between 10-20 most of the time.23:27
lisitsynsometimes it is cold23:27
lisitsynsometimes it is hot :D23:27
lisitsynwhat sucks is that roads are ruined every spring23:27
thoralfWe can choose between cold and warm rain. ;)23:28
lisitsynthoralf: and you speak like I don't want to come to berlin ;)23:28
lisitsyncan't see the opportunity ;)23:28
thoralflol23:28
thoralfDid you see this one? ;) https://github.com/shogun-toolbox/shogun/issues/213323:46
thoralflisitsyn: Every time I touch shogun this happens.23:53
lisitsynthoralf: actually sparse vector is assumed to be sorted23:53
thoralflisitsyn: Well, if it's assumed to be sorted, why computing dot product this way?23:59
thoralfhttps://github.com/shogun-toolbox/shogun/blob/develop/src/shogun/lib/SGSparseVector.cpp#L35523:59
thoralfYou can do it in O(n), but this one is in O(n^2).23:59
lisitsynunreadable :D23:59
--- Log closed Mon Apr 14 00:00:04 2014