Open in new window / Try shogun cloud
--- Log opened Thu Nov 24 00:00:29 2016
@wikingosx does not have openmp01:12
@wikingi mean the official clang in it01:12
@wikingand i wonder when are we ever gonna get openmp support in the official osx clang01:19
@sukeyPull Request #3562 "LinalgRefactor - Matrix elementwise product"  synchronized by OXPHOS -
@sukeyPull Request #3563 "LinalgRefactor - Remove old linalg::range_fill and linalg::max"  opened by OXPHOS -
-!- praisethemoon [~praisethe@] has joined #shogun08:50
-!- praisethemoon [~praisethe@] has quit [Changing host]09:06
-!- praisethemoon [~praisethe@unaffiliated/praisethemoon] has joined #shogun09:06
-!- praisethemoon [~praisethe@unaffiliated/praisethemoon] has quit [Quit: Leaving]10:31
-!- praisethemoon [~praisethe@] has joined #shogun10:58
-!- praisethemoon [~praisethe@] has quit [Changing host]10:58
-!- praisethemoon [~praisethe@unaffiliated/praisethemoon] has joined #shogun10:58
-!- HeikoS [] has joined #shogun11:52
-!- mode/#shogun [+o HeikoS] by ChanServ11:52
CaBawiking: still nothing with xcode 8 either?12:28
@wikingHeikoS, ping12:30
@HeikoSwiking: pong12:30
@wikingjust a sec12:30
@HeikoSCaBa: this weekend I will check the parpameter tree!12:30
@HeikoSwiking: I want to have another chat with you on video soon12:30
@HeikoShave some ideas about how to push the stuff we discussed with gunnar12:30
@wikingping me anytime12:31
@HeikoSwiking: I mean basically12:32
@HeikoSthis idea of pushing the framework one level higher12:32
@HeikoSin the sense that it simply offers an environment to *easily* write scientific code12:32
@HeikoSand offers all the help: math, data managment, modularity, interfaces, testing, build12:33
CaBawiking: is there still any correspondence between the apple llvm / clang releases and the upstream original version numbers?12:33
@HeikoSand then all the algos we have are simply plugins for that12:33
CaBaso confusing...12:33
CaBaHeikoS: oh, great, thanks!12:33
@HeikoSthat is cool12:33
@HeikoSwiking:  it would also simplify the build etc12:33
@HeikoSwiking: but I want to discuss how to approach this12:33
@HeikoSwiking: because I have the feeling that it sounds like "rewrite" ;)12:33
CaBa.oO(he said the r-word. now there's postapocalyptical silence...)12:47
@wikingso there are couple of amazing stuff12:48
@wikingget_kernel_matrix in kernel12:49
@wikingit seems we are lucky12:49
@wikingthat it's correct in linux12:49
@HeikoSthis thing12:49
@wikingeverywhere else that is incorrect12:49
@wikingi'm currently writing a unit test12:49
@wikingthat make sure that it's correct12:49
@wikingand then change the implementation to use openmp12:49
@HeikoSwiking: do you have a bit time now for a chat?12:49
@HeikoSwiking: yep that sounds good12:49
@HeikoSwiking: like in 5-10 mins for say 15 mins12:50
@HeikoSjust finishing a review12:50
@wikingskype hangout whatspapp?12:50
@wikingwhatsapp has voice12:50
@wikingso we can do that12:50
@wikingbut yeah i'm here12:50
@wikingso just ping me12:50
@HeikoSvideo maybe12:52
@HeikoSill ping in 512:52
@HeikoSwiking: ok now?12:55
@wikingjust gimme a link where12:56
@wikingneed a hangout link12:56
@wikinginvite me to an event12:56
-!- HeikoS [] has quit [Ping timeout: 245 seconds]13:30
-!- HeikoS [] has joined #shogun13:32
-!- mode/#shogun [+o HeikoS] by ChanServ13:32
-!- HeikoS [] has quit [Ping timeout: 256 seconds]14:41
CaBawiking: ping16:15
-!- HeikoS [] has joined #shogun16:31
-!- mode/#shogun [+o HeikoS] by ChanServ16:31
@sukeyPull Request #3563 "LinalgRefactor - Remove old linalg::range_fill and linalg::max"  merged by karlnapf -
@sukeyNew Commit "Merge pull request #3563 from OXPHOS/linalg_rm_rangefill_and_max16:35
@sukeyLinalgRefactor - Remove old linalg::range_fill and linalg::max" to shogun-toolbox/shogun by karlnapf:
-!- HeikoS [] has quit [Ping timeout: 260 seconds]17:07
-!- HeikoS [] has joined #shogun17:22
-!- mode/#shogun [+o HeikoS] by ChanServ17:22
@sukeyPull Request #3538 "temporary workaround for #3460"  synchronized by karlnapf -
@sukeyPull Request #3538 "fix for #3460" -
@sukeyPull Request #3538 "fix for #3460"  synchronized by karlnapf -
-!- HeikoS [] has quit [Ping timeout: 268 seconds]17:29
@sukeyPull Request #3534 "LinalgRefactor - Cholesky - CPU only"  synchronized by OXPHOS -
CaBawiking: HeikoS mentioned changes regarding builds on os x recently. both my development and production systems are mac os systems (i7-2620M is my desktop, Xeon-X5675 the production system)17:38
@sukeyPull Request #3533 "LinalgRefactor - Remove old linalg::scale"  synchronized by OXPHOS -
CaBawiking: the latter is currently much slower (!) on my small test dataset...17:39
CaBawiking: could there be any explanation for this within shogun?17:40
-!- praisethemoon [~praisethe@unaffiliated/praisethemoon] has quit [Ping timeout: 250 seconds]18:07
-!- HeikoS [~heiko@] has joined #shogun19:03
-!- mode/#shogun [+o HeikoS] by ChanServ19:03
@sukeyNew branch feature/kernel_exp_family_base_measure created on shogun-toolbox/shogun19:04
@sukeyNew Commit "temp commit" to shogun-toolbox/shogun by karlnapf:
CaBaHeikoS: ping19:23
CaBa17:37:11 < CaBa> wiking: HeikoS mentioned changes regarding builds on os x recently. both my development and production systems are mac os systems (i7-2620M is my desktop, Xeon-X5675 the production system)19:23
CaBa17:37:41 < CaBa> wiking: the latter is currently much slower (!) on my small test dataset...19:23
CaBa17:38:52 < CaBa> wiking: could there be any explanation for this within shogun?19:23
CaBaHeikoS: this does not by any change ring a bell or so? just checking.19:23
-!- HeikoS [~heiko@] has quit [Ping timeout: 265 seconds]19:29
@wikingCaBa, mmm20:42
@wikingyou use gcc for build no?20:42
CaBawiking: gcc-6 for both shogun and my application in the meantime. same versions on both systems (including stdlib). i had different versions of protobuf and hdf5 installed, but bumped that in sync also in the meantime. same effect.21:11
CaBawiking: it also doesn't depend where i build shogun and my program. if i move the lib + my bin to the other system, the performance changes21:11
CaBawiking: oh i think i know what this might be related to...21:27
@wikingthat is weird though21:29
@wikingxeon is slower?21:29
@wikingshooulnd't that be faster? :D21:29
CaBawiking: i figured it out...21:33
@wikingwhat was it?21:33
CaBawiking: the xeon box has 24 cores... after running it through a profiler, i figured he was mostly busy with locks21:33
@wikingis it us21:33
CaBawiking: too many threads21:33
@wikingwhich locks?21:34
CaBawiking: set_num_threads(4) -> now i have the same performance as on my laptop21:34
@wikingah lol21:34
@wikingbut fuck21:34
@wikingCaBa, i have some fixes21:34
@wikingi hate that you have to compile in order to change num threads21:34
@wiking(in case of c++21:34
@wikingso i have now a new env var21:34
CaBaoh, runtime21:34
@wikingwill push to develop as soon as i have a fix for kernel.cpp21:34
@wikingmmm btw21:36
@wikingwhen yo urun21:36
CaBawiking: training / testing never uses more than 4fold cpu power for me... not even on my actual large datasets. maybe i should be turning that down to 4 in general...21:36
@wikingdon't you have several tests failing?21:36
CaBawiking: never run that manually. is that part of the default build target?21:36
@wikingnono what i mean21:36
@wikingif you do this21:36
@wikingcmake -DENABLE_TESTING ....21:37
@wikingand then once compiled21:37
@wikingrun ctest -R unit21:37
@wikingdoes all the test passes for you?21:37
@wikingbtw what are you using for the depedency libraries21:37
CaBai'll check, one moment21:37
@wikingok so your eigen is already 3.3.021:37
CaBa/usr/local/Cellar/eigen/3.3.0 (484 files, 6.4M) *21:37
@wikingok it's not that big a deal21:38
@wikingPCA will fail for you21:38
@wikingthe unit test21:38
@wikingbut it's just the test's fault21:38
@wikingas it does not consider that if EV is really an eigen vector21:38
CaBa-DENABLE_TESTING is not the default?21:38
@wikingthan -1.0*ev is an eigen vector as well21:38
@wikingi dont think so21:38
CaBaok, so i have to rerun21:39
@wikingyeah sorry21:39
CaBanp, 24 cores :P21:39
@wikingld is 1 core21:39
@wikingand it'll take fucking 5 minutes21:39
@wikingto link the unit test21:39
@wikingi'm currently working on a fix for it21:39
@wikingbut unfortunately i'm not sure if i can get the gcc part working as well21:39
@wikingthe clang part is tested21:39
CaBai'm on 18204b2, is that enough?21:41
@wikinglemme check21:41
@wikingmmm i think i dont even have that locally21:41
CaBakk ;)21:42
CaBait's just 1 commit ago i just realized21:42
CaBanot much changed the last 2 days21:42
CaBa[ 99%] Built target shogun-unit-test21:43
@wikingthat'll be 5 minute21:43
CaBa[ 99%] Built target python_doxy2swig21:44
CaBathe evil 99%... almost like a good old microsoft progress bar ;)21:44
@wikingcheck if build/tests/unit/shogun-unit-test exists21:45
@wikingif it is ok21:45
@wikingthen in a separate shell21:45
@wikingcd build && ctest -R unit21:45
CaBaThe following tests FAILED: 84 - unit-PeriodicKernelTest (Failed)21:48
@wikingyep yep21:48
@wikingyou'lll have a summary on the end21:48
CaBathat's kinda the end21:49
CaBa99% tests passed, 1 tests failed out of 27121:50
CaBaTotal Test time (real) =  10.20 sec21:50
CaBayou mean that?21:50
@wikingon the end21:50
@wikingctest should give you the failed ones21:50
@wikinga list of it21:50
CaBa99% tests passed, 1 tests failed out of 27121:51
CaBaTotal Test time (real) =  10.20 sec21:51
CaBaThe following tests FAILED: 84 - unit-PeriodicKernelTest (Failed)21:51
CaBaErrors while running CTest21:51
CaBathat are the last lines.21:51
@wikingcan you run it one more time?21:52
@wikingbtw you use gcc to compile right?21:52
CaBayes, still ;)21:52
CaBai re-ran already21:52
CaBasame output21:52
@wikingsame error?21:52
@wikingcan you do this then21:53
CaBathere is a more detailed log thouth21:53
@wikingctest -R unit-PeriodicKernelTest -V21:53
@wikingand pastebin it21:53
@wikinguname -a?21:54
@wikingfor me it's21:54
@wikingDarwin rdr.local 16.1.0 Darwin Kernel Version 16.1.0: Thu Oct 13 21:26:57 PDT 2016; root:xnu-3789.21.3~60/RELEASE_X86_64 x86_6421:54
CaBaDarwin 15.6.0 Darwin Kernel Version 15.6.0: Thu Sep  1 15:01:16 PDT 2016; root:xnu-3248.60.11~2/RELEASE_X86_64 x86_6421:54
CaBano sierra yet21:54
@wikingok i'll need your help21:54
@wikingas soon as i think i'm ready with the fixes21:54
@wikingwhich would be another21:54
@wiking12 hours21:55
@wikingor so21:55
CaBawhat's the matter? what's different for you>?21:55
@wikingfor me many other tests are failing21:55
@wikingcan you try it on your laptop?21:55
CaBalaptop is on el capitan as well21:56
CaBai keep them as sync as possible config wise ;)21:56
CaBathis is my cmake config21:57
CaBajust in case this disables things that you are talking about21:57
@wikingi'll have to check with gcc622:02
@wikingas i've never tried to compile it22:02
@wikinganyhow the reason we couldnt upgrade22:02
@wikingshogun to 5.0.0 in brew22:03
@wikingbecause of some weird shit with22:03
@wikingwith some of tests22:03
-!- HeikoS [] has joined #shogun22:56
-!- mode/#shogun [+o HeikoS] by ChanServ22:56
@sukeyPull Request #3538 "fix for #3460"  merged by karlnapf -
@sukeyNew Commit "Merge pull request #3538 from karlnapf/bugfix/R_33_modular23:06
@sukeyfix for #3460" to shogun-toolbox/shogun by karlnapf:
@sukeyIssue #3460 "Rmodular cannot be built " closed by karlnapf -
@wikingHeikoS, ping23:06
@HeikoSwiking: pong23:06
@wikingi'm just testing our PRNG23:06
@HeikoSwhich one?23:07
@HeikoSthe Random23:07
@wikingand comparing it to std::23:07
@HeikoSah yes23:07
@wikingdunno what to do with this shit23:07
@wikingbecause that prng is good for23:07
@wikingbut we never use fill array23:07
@HeikoSah we loop23:07
@wikingfor (int i = 0;....23:07
@wikingbut my biggest problem23:08
@wikingthat lets say i remove23:08
@wikingthe thing and start using c++11123:08
@wikingthat is not threadsafe23:08
@wikingso we need to box it with a mutex23:08
@wikingwhich is even shittier23:08
@wikingso i'm wondering wtf with this whole random23:08
@HeikoSso mutexing it, is it really that bad?23:09
@wikingwell fuck23:09
@wikingi mean not that bad23:09
@wikingbut why would we do it23:09
@wikingi mean i'm not so sure if we should create wrappers at all23:09
@wikingc++11 has a very nice interface23:09
@wikingfor randome23:09
@HeikoSif it is part if the language itself23:10
@HeikoSI think we can use it23:10
@HeikoSit is just23:10
@HeikoSif we want to change once a cool lib comes out,23:10
@wikingmmm yeah23:10
@HeikoSthough to be fair that is mostly for external libs23:10
@HeikoSnot for c++ stuff23:11
@HeikoSI would also use the STL more heavily these days :)23:11
@wikingheheh yeah23:11
@HeikoSand get rid of DynArray23:11
@wikinganyhow i still wanna compare23:11
@HeikoSyeah sure23:11
@wikinghahaha DynArray23:11
@wikingso lemme try to write a unit tests23:11
@HeikoSIll write some dev docs now23:11
@HeikoShave a bit of time23:11
@wikingto std_normal_distro23:11
@HeikoSI merged the R fix btw23:12
@HeikoSmanual speaks clear langauge23:12
@HeikoSno R includes from external C23:12
@HeikoSI  mean the whole sg_print_functions.cpp business is a bit outdated anyways23:12
@HeikoSbut no priority23:12
@wikingthis whole shit with23:13
@wikingshould go into a toilet23:13
@wikingespecially the random part23:13
@wikingwe should put all that to deprecated23:13
@wikingstart adding acutally optimal implementation23:13
@wikingof log, exp etc23:13
@wikingsince we have actually a wrapper for it in CMath23:13
@HeikoSmake it issues23:18
@HeikoSand put them to a milestone23:18
@sukeyNew Commit "Merge pull request #3533 from OXPHOS/linalg_scale_remove23:18
@sukeyLinalgRefactor - Remove old linalg::scale" to shogun-toolbox/shogun by karlnapf:
@sukeyPull Request #3533 "LinalgRefactor - Remove old linalg::scale"  merged by karlnapf -
@HeikoSwiking: I am serious23:18
@wikingi mean nono23:19
@wikingi know23:19
@wikingthis is fucking scary23:19
@wikinglook at this23:19
@wiking./examples/undocumented/libshogun/basic_random 100000023:19
@wiking35.8699 ms23:19
@wiking~/a.out 1000000 std23:19
@wiking140.559 ms23:19
@wikingwe are not tHAT bad23:20
@wikinga.out - c++1123:20
@HeikoSdo these things pass the same tests for randomness?23:20
@wikingwhat do you mean pass the same tests?23:20
@wikingi mean they are both std normal distrib23:20
@wikingdo we have a test for this? or should i actually test it myself? :)23:20
@wikinglike on a qq plot23:21
@wikingi mean i would presume c++11 has a true std normal distrib implemented :D23:21
@HeikoSc++11 will be correct23:22
@HeikoSbut is ours?23:22
@HeikoSthere are more sohpisticated tests than qq plot23:22
@wikingok so can u just give me one23:22
@wikingi mean thi sis the random23:22
@wikingthat we use like everywhere23:22
@wikingjsut saying23:22
@wikingso if its not correct we are fucked anyways23:22
@wikingTEST(Random, random_std_normal_quantiles)23:23
@wikingwe have this test23:23
@wikingactually implemented by u23:23
@wikingso that should assure that we are really generating a std normal distro23:24
@HeikoSno it is not enough23:24
@HeikoSjust to do quick early alarms23:24
@HeikoSif something is terribly wrong23:24
@HeikoSbut if we have code written by ourselves that generates random numbers23:25
@wikingyeah but ok23:25
@HeikoSwe need to test for more subtle artifacts23:25
@wikingwell it's not our own code23:25
@wikingit's DSFTM23:25
@HeikoSah ok23:25
@HeikoSthen ok23:25
@wikingand then on top we apply23:25
@HeikoSthen the unit test is only for making sure the sign and scale is right23:25
@wikingi thnk w ehave a box muller transform implemented23:26
@sukeyNew Commit "Correct link to" to shogun-toolbox/shogun by karlnapf:
@wikingi have to check which implemntation we hae23:27
@wikingbut basically we used the uniform distrib23:27
@wikinggenerated by DSFMT23:27
@HeikoSI need the index.html23:28
@wikingyou needit23:28
@HeikoSthat seems like I should change it?23:28
@HeikoSso that this is enough23:28
@wikingah you need a redirect for that23:28
@wikingin the website23:28
@HeikoSshall I add?23:28
@wikinggo ahead23:28
@wikingi mean23:28
@wikingi wouldn't do a redirect23:28
@wikingjust here23:28
@HeikoSso change links instead?23:29
@HeikoSto include the index.html?23:29
@wikinghere check the filename23:29
@wikingif it's ending on latest or current23:30
@wikingthen need to append index.html23:30
@wikingelse do whatever is there23:30
@wikingok finally the first time i'm positively surprised about shogun in a long time23:31
@HeikoSthe rng stuff?23:32
@wikingbut i'm still wondering23:34
@wikingwhy dont we have like an array filler23:34
@wikingfor random23:34
@wikingi'll make every random related stuff in CMath deprecated23:34
@wiking [[deprecated]](C++14)23:35
@wikingc++14 feature23:35
@HeikoSwould be good to have a deprecated marker in shogun in fact23:36
@HeikoSbut not sure how much that helps23:36
@HeikoSbetter fix things23:36
@wikingyou know how many palces23:36
@wikingwe have CMath::random23:36
@wikingso i'lljust now put a deprecated shit23:36
@wikingand then change slowly23:36
@wikingwonder where should i add the deprecated macro23:36
@wikingit goes into common.h23:37
@wikingi'm wondering whether we should depend on this
@HeikoSI am not sure about deprecated23:42
@HeikoSwhat does it help us marking these things?23:42
@HeikoSwiking: ^23:43
@HeikoSsaw that guy?23:43
@wikingyes for the 2nd time23:43
@wikingignore it23:43
@HeikoSI broke the cookbooks with my redirect23:44
@HeikoScan you quickly press "examples" on website and tell me what causes that?23:45
@HeikoSmaybe I revert and add back the index.html23:45
@HeikoSseems easier23:45
@wikingdidn't you test locally?23:46
@HeikoSI did23:47
@wikingthe css seems to be fucked up23:47
@wikingits 40423:47
@wikingand that shouldn't be23:47
@HeikoSI revert this one and leave index.html23:48
@HeikoSno nerves for this now23:48
@HeikoSlocally I get
@HeikoSso cannot test23:48
@wikingjust fix it correctly23:48
@wiking    <link rel="stylesheet" href="_static/basic.css" type="text/css" />23:48
@wikingyou have this23:48
@wikingit's not absolute but relative23:48
@wikingif you get ->
@wikingsame for current23:48
@wikingand that's all23:49
@wikingpretty simple23:49
@HeikoSnot sure I understand what you are saying, but thats because i actually dont know stuff about this :)23:49
@HeikoSso lets see23:49
@wikingbut fuck23:49
@wiking i mean23:49
@wiking <link rel="stylesheet" href="_static/basic.css" type="text/css" />23:49
@wikingthi sshit referes23:49
@wikingto something that is relative path23:49
@wikingnow if you have23:49
@wikingand then append _static/basic.css23:50
@wikingthen you get23:50
@wikingwhich is wrong obviously23:50
@wikinghence the shitty looking stuff now23:50
@wikingso if your url is23:50
@wiking.endswith('latest') or endswith('current')23:50
@wikingyou should append a /23:50
@wikingto the url itself23:51
@wikingnot to the file reading part23:51
@wikingbut i mean23:51
@wikingthe hillarious part is23:51
@wikingthat this has been solved23:51
@wikinglike 15 years ago23:51
@wikingin apache23:51
@wikingnow fucking python shit fucking flask23:51
@wikingneeded of course23:51
@wikingto reinvent the fucking wheel23:51
@wikingand here we are23:51
@wikingyou have to deal with correct fucking URLs23:52
@wikingi wouldn't actually do a redirect23:54
@wikingfor /examples23:54
@wikingbecause it's fucking stupid23:54
@wikingjust add a new route on top of23:54
@HeikoSno idea23:54
@HeikoSI reverted the commit for now, easier to separate out what fucked up this way23:55
@wikingbut then dont change things in future? :)23:55
@HeikoSI want it different23:55
@HeikoSbut I have like one hour now that I wanted to write some docs23:55
@HeikoSand not deal with http links ;)23:55
@wikingyeah but fuck23:55
@wikingthen why do you do it23:55
@HeikoSwrong decision23:57
--- Log closed Fri Nov 25 00:00:30 2016