#pypy IRC log for Monday, 2005-09-12

dialtone (n=dialtone@merlinux.de) left irc: Read error: 110 (Connection timed out)00:59
pedronis (n=Samuele_@c-398b70d5.022-54-67626719.cust.bredbandsbolaget.se) left irc: "Chatzilla 0.9.68a [Firefox 1.0.4/20050511]"01:12
Action: stakkars thinks nine months fits the idea of a baby, nicely01:38
fijal (i=root@zodiac.mimuw.edu.pl) left irc: Remote closed the connection02:33
stakkars (i=kxwsqz@i577B4676.versanet.de) left irc: Read error: 110 (Connection timed out)03:42
hpk (n=hpk@merlinux.de) got netsplit.03:54
thingie24 (n=asdf@valhalla.ccp.cc) got netsplit.03:54
xorAxAx (n=xorAxax@moinmoin/coreteam/alexander) got netsplit.03:54
mwh (n=user@82-33-185-193.cable.ubr01.azte.blueyonder.co.uk) got netsplit.03:54
pyb0t (i=pybot@nimrod.terra-link.net) got netsplit.03:54
Rhamphoryncus (n=rhamph@unaffiliated/rhamphoryncus) got netsplit.03:54
ironfroggy (n=calvin@68.83.122.167) got netsplit.03:54
sabi (i=njriley@shrug.csl.uiuc.edu) left irc: Read error: 104 (Connection reset by peer)03:54
typo_ (n=enki@port-212-202-201-68.dynamic.qsc.de) left irc: Read error: 104 (Connection reset by peer)03:54
typo_ (n=enki@port-212-202-201-68.dynamic.qsc.de) joined #pypy.03:54
sabi (i=njriley@shrug.csl.uiuc.edu) joined #pypy.03:55
pyb0t (i=pybot@nimrod.terra-link.net) returned to #pypy.03:55
Rhamphoryncus (n=rhamph@unaffiliated/rhamphoryncus) returned to #pypy.03:55
hpk (n=hpk@merlinux.de) returned to #pypy.03:55
mwh (n=user@82-33-185-193.cable.ubr01.azte.blueyonder.co.uk) returned to #pypy.03:55
thingie24 (n=asdf@valhalla.ccp.cc) returned to #pypy.03:55
ironfroggy (n=calvin@68.83.122.167) returned to #pypy.03:55
xorAxAx (n=xorAxax@moinmoin/coreteam/alexander) returned to #pypy.03:55
-NickServ (NickServ@services.)- This nickname is owned by someone else03:55
-NickServ (NickServ@services.)- If this is your nickname, type /msg NickServ IDENTIFY <password>03:55
politik (n=d@c-24-147-183-28.hsd1.ma.comcast.net) left irc: Read error: 104 (Connection reset by peer)03:55
typo_ (n=enki@port-212-202-201-68.dynamic.qsc.de) left irc: "x"04:05
idnar (i=mithrand@unaffiliated/idnar) left irc: Nick collision from services.06:13
idnar_ (i=mithrand@unaffiliated/idnar) joined #pypy.06:14
idnar_ (i=mithrand@unaffiliated/idnar) left irc: Remote closed the connection06:58
idnar (n=mithrand@unaffiliated/idnar) joined #pypy.06:59
adim (n=adim@logilab.net2.nerim.net) joined #pypy.07:14
idnar (n=mithrand@unaffiliated/idnar) left irc: Remote closed the connection07:17
idnar (n=mithrand@unaffiliated/idnar) joined #pypy.07:18
dialtone (n=dialtone@merlinux.de) joined #pypy.07:27
idnar (n=mithrand@unaffiliated/idnar) left irc: Remote closed the connection07:38
adim (n=adim@logilab.net2.nerim.net) left #pypy.07:41
dialtone (n=dialtone@merlinux.de) left irc: Read error: 110 (Connection timed out)07:52
fijal (i=root@zodiac.mimuw.edu.pl) joined #pypy.08:05
fijal (i=root@zodiac.mimuw.edu.pl) left irc: Remote closed the connection08:08
fijal (i=root@zodiac.mimuw.edu.pl) joined #pypy.08:08
dialtone (n=dialtone@merlinux.de) joined #pypy.08:12
arre (i=ac@ratthing-b40e.strakt.com) joined #pypy.08:33
stakkars (i=rmtdttv@i577B4751.versanet.de) joined #pypy.08:47
arigo (n=arigo@134.99.112.244) joined #pypy.09:15
arigogood, ports to the server have been opened09:55
stakkarsgood morning!10:03
arigomorning!10:04
mwhmorning!10:04
stakkarsnow I thought I had a complete low level range implementation (and it works just fine),10:05
stakkarsbut then I figured out that we do differently from CPython, concerning ranges with > maxint elements.10:05
stakkarsI'm wondering how to handle that, ignoring is probably not ok, at least I should check for it.10:06
arigowhat is the difference exactly?10:06
stakkarsalso, the CPython version doesn't store stop, but the length, which would have been my approach, too.10:06
stakkarswell, xrange length is an unsigned, they allow the full range.10:07
stakkarsif rrange did this too, I could use it to implement applevel xrange,but this way it is not powerful enough.10:07
stakkars(not sure if xrange is correct in this sense, too, checking...)10:08
arigoit doesn't make much sense to use rrange to implement app-level xrange, as far as I see10:09
stakkarsdifference? you can have xrange(-maxint, maxint)10:09
arigoin CPython? no...10:09
stakkarsyes..10:09
arigo>>> xrange(-sys.maxint, sys.maxint)10:10
arigoTraceback (most recent call last):10:10
arigo  File "<stdin>", line 1, in ?10:10
arigoOverflowError: xrange() result has too many items10:10
stakkarsok,almost. Anything that doesn'tlet unsigned long wrap.10:11
arigo? no10:11
arigoxrange(-1, sys.maxint) -> OverflowError10:11
arigosorry, I'm lost -- I really see a 'long len' in rangeobject.c10:12
stakkarshum, that was what confused me. Of course they use a long, and I thought they would use the full range.10:12
stakkarsbut they don't do it, cause the result is assigned to a signed long10:13
stakkarsnot sure what the injtent was.10:13
stakkarsand yes. xrange.len is a signed. Dumb ofme. But why do they the unsigned crap.10:14
arigoin rangeobject.c?  to compute the length, e.g. to support10:14
arigoxrange(-sys.maxint, sys.maxint, 2)10:14
stakkarsok, I see.So I have no point, our stuff *is* compatible.10:15
stakkarsalthough the impl stoing len has some attraction.10:15
arigokeep in mind that these rranges are meant to vanish away10:16
arigothe current impl has the advantage that it looks very much like a C for loop: there is a single index field that gets incremented10:17
stakkarsvanishing means they get inlined, unless they are exposed.10:17
arigoyes, the whole 'for x in range()' graphs and calls becomes like a normal C for loop10:18
stakkarsyes, I noticed. It implies the up/down iterators. a generic up/down would do better with an extra counter.10:18
arigothe motivation was that doing divisions and multiplications is not common in C for loops10:18
stakkarsso you hope to support C optimizations better. IC10:19
stakkarsanyway, I think I fail to check for range overflow. Need to add that somewhere10:20
arigono, that probably doesn't belong to RPython10:21
stakkarsor maybe not? unnecesary slowdown to have ot computelength al the time?10:21
arigodrawing the line is always some kind of funny exercice, but let's not forget that at Paris already we should start thinking seriously about JIT10:22
stakkarssure. not forgetting that we need to speed up this baby a little, to run a JIT on top of it :-)10:24
fijalI've got a question: suppose I want to add some python-checking routines (some static analyzis like pylint), shall I put it as different translator?10:27
arigowell, it depends what more precisely you are trying to do10:28
stakkarsif you think of an RPython checker, I'd put it as different backend.10:29
stakkarsfor generic Python, we don't have the tools.10:29
fijalNo, I think about app-level checker to warn user about several caveats that he have in his code10:29
fijaltaken from flow-graph analyzis mainly10:30
stakkarsyou cannot flow-graph analyze aplevel code.10:30
arigothis is work :-)  you need to enhance the flow object space to handle any app-level construction10:30
arigocurrently it will not be happy on some stuff like generators, def or class statements, etc.10:30
arigothen you can probably try running the annotator on it, and see where it explodes10:31
stakkarsand add extra options to remove current assumptions like globals == const, etc.10:31
arigobut this is likely to be more confusing than useful at the moment10:31
arigostakkars: indeed10:31
arigoit is the kind of analysis that we'll want to do later, but our toolchain is not oriented towards this right now10:31
fijalso what's the target of analyzis?10:32
fijalare You going to make such things in the future?10:32
arigothe current target is RPython code, i.e. "Python code with a finite amount dynamism"10:33
Action: stakkars would like to point you at the PyPy docs10:33
arigohow exactly we will support analysis of app-level code is yet open10:34
stakkarsarigo: you reminded me of JIT. This means I should not worry too much about static optimization.10:36
arigoyes, at least for features that we don't use in PyPy anyway10:37
stakkarsHow far is this true? DO you think we can gain speed using JIT technology, if we are unable to do it statically?10:37
arigolike catching OverflowError upon calling xrange()...10:37
stakkarsok, that's only interesting for people writing RPython code (which will probably come)10:38
arigothe JIT means that we will need something that supports not just RPython but the full Python, and we definitely don't want to rewrite everything in r*.py, so we need to focus on how to use the existing stdobjspace instead for that10:38
arigoyes, RPython is nice to have, but should be kept basic enough (that's its raison d'etre)10:39
stakkarsit's even a tad that we had to make the compiler RPython. A good JIT would save most of these I guess.10:41
arigoyes10:41
arigobut the plan is to use the translator to produce the JIT, so we need the translator...10:42
mwhi take it the jit will work in a similar way to pypy10:43
stakkars(and we probably thought to work on it much earlier, but things happen)10:43
mwhsuspending execution & so on10:43
stakkarswhich translator did you mean, actually?10:43
arigostakkars: well the whole objspace+annotation+rpython10:43
stakkarsyeah, ok10:44
arigomwh: in a similar way to Psyco, which is itself similar to PyPy's flow objspace10:44
mwharigo: what a coincidence!10:44
stakkarsso we actually *have* to re-think flow objspace (and maybe write a different one)10:44
arigoindeed10:44
arigoI wrote some introductory stuff yesterday night10:45
stakkarsI thought you rewrote the whole dict world! When do you sleep, actually?10:45
arigo:-)10:45
arigowith hindsight, the flow objspace's role can be made clearer10:45
arigocurrently, we import our program in CPython and then run the flow objspace on it, using introspection10:46
arigobut another point of view is possible10:46
arigowe could import the program to analyse in PyPy, in a std or trivial object space10:47
stakkars:-X10:47
arigoand then switch between this objspace and the flow objspace to switch between actual and abstract execution10:47
arigothis has the advantage of less cross-level confusion10:47
stakkarshaving it both real and abstract. Operating in parallel,or exchanging the object space in a way?10:48
arigodepends10:48
arigoat first, exchanging the object space10:48
arigo(which is not unrelated to the flow object space that can currently switch to "concrete mode", btw)10:48
arigofor Psyco, I guess it would operate with both spaces in parallel10:49
arigoso that the flow space can ask the concrete space for more information, e.g. the type of an object10:49
stakkarsfor full Python, it sounds a little that we have to extend std objspace to grow an "abstract mode" :-)10:50
arigoyes, but not necessarily by programming this features into the std objspace10:51
arigobut with a proxy objspace on top of it10:51
stakkars(not serious, but flow space lacks a lot of features)  -- yes10:51
stakkarsand the proxy would operate them in parallel, and you can ask it to give you specific info (my picture)10:52
arigoyes10:52
xorAxAxhas anybody looked at  http://shedskin.sourceforge.net/? /me takes a look10:54
stakkarshum, your remarks mademe explicitly not look into it :-]10:54
xorAxAxwhy?10:55
xorAxAxbecause he didnt contact the pypy team? :-)10:55
xorAxAxhmm, its mit-licensed10:56
xorAxAx"Thanks a lot to Google, for sponsoring this project via their SoC program. This has really been a major motivation to me, and enabled me to work on the project longer than would have been possible otherwise.  "10:56
xorAxAxah!10:56
stakkarsno I misunderstood your thought. Will read it.10:56
arigono, we know about his work10:57
xorAxAxarigo: thats good10:58
xorAxAx30% unit tests, wow10:59
arigowell, what I drafted yesterday also gives a reason that we don't follow this project very closely11:00
stakkars(tried to read shedskin's doco, but can't find any)   --  is your draft available, somewhere? Not in doc, sofar11:01
arigohttp://codespeak.net/pypy/dist/pypy/doc/draft-dynamic-language-translation.txt11:02
stakkars:) ok I should svn up even more often :)11:03
arigohum, the .html version doesn't show up on codespeak.  I wonder if the update-on-checkin scripts are activated.11:04
stakkarsare they cron-driver, or triggered by svn?11:05
stakkarsdriven11:05
arigoshould be triggered by svn11:05
stakkarsyes,your text is there but no html11:06
xorAxAxkinda monolithic, the ss.py (211kb). reminds me at pyc, the greek compiler :)11:09
arigoouch11:12
stakkarsfor some reason, the draft fails on codespeak. Not on my machine.11:15
stakkarsah, ok, I don't have permission.11:16
mwharigo: that's not the most comprehensible document i've ever read11:20
mwhsome long sentences :)11:21
pedronis (i=pedronis@ratthing-b246.strakt.com) joined #pypy.11:21
Rhamphoryncus (n=rhamph@unaffiliated/rhamphoryncus) left irc: "Support ISO 8601! http://www.cl.cam.ac.uk/~mgk25/iso-time.html"11:25
pedronishi11:26
stakkarshi11:29
magent1 (n=dbernhar@tiramisu.imag.fr) joined #pypy.11:30
magent1 (n=dbernhar@tiramisu.imag.fr) left #pypy.11:30
aleale (n=andersle@clogs.dfki.uni-sb.de) joined #pypy.11:35
arigo (n=arigo@134.99.112.244) left irc: Read error: 110 (Connection timed out)11:36
stakkarswell, and lots of note here and there, content ok, some things would be clearer if shorter. I understand it because I know a little how the author thinks.11:37
stakkarsthe idea is just great11:37
mwhoh yes11:40
mwhnot meant to be an extreme criticism11:40
stakkarsneither here, it is just in fact not too easy to read.11:41
arigo (n=arigo@134.99.112.244) joined #pypy.12:04
arigowhere are the IRC logs now?12:21
stakkarsas before, plus on tismerysoft.de/pypy/irc-logs12:21
stakkarsI'm logging pypy-sync there as well. Anything else, maybe? pypy-tb?12:22
arigothanks12:22
arigopypy-tb, maybe12:23
stakkarstook a little more time than planned, because I had to circumvent blockade of port 6667 at my provider.12:23
arigomissed the discussion on the document -- hum, yes I shouldn't write too late at night12:24
stakkarsit's fine, but a bit "strubbelig" :-)12:24
arigowell I feel it's too lengthy already, not sure if it should be split up to a separate introduction document maybe12:25
stakkarsthe intro goes a bit much into details of pypy, I'd split this apart a little12:26
stakkarsI've just skimmed the 5737 lines of ss.py12:27
stakkarsseems to be what you want if you think C++12:28
stakkarswhich is exactly what we don't want :-)12:28
arigo:-)12:32
stakkarscan be a static translation tool for static source code as it looks, and appears to be absolutely unflexible.12:33
stakkarslow abstraction, hardly commented set of functions, well 'nuf said. I still can't figure out if something would help us.12:34
pedronisI tried on richards.py12:42
pedronisI tried it12:42
stakkarsdid it grok it?12:43
Nick change: arigo -> arigo|lunch12:44
stakkars( tried the ss.py, or are you talking of the new dict implementation?)12:44
pedronisthe ss.py12:45
xorAxAxand?12:45
stakkarsI wondered about the mentors: Jeremy Hylton, Brett Cannon12:45
xorAxAxit looks like google might use it for some of their works12:46
pedronisit was not happy about is is not and /=12:46
pedronischanged those12:46
pedronisthen it was unhappy about subclassing from object12:46
pedronisremoved that too12:46
stakkarsremoved that from richards?12:47
pedronisyes12:47
pedronisthen it was unhappy about assigment in a class suite12:47
pedroniswhich would mean a class attribute which is understandable12:47
pedronisbut it crashed not nicely12:47
pedronisafter all these changes12:47
stakkarsand after richard was empty, it compiled,mission accomplished :-)12:47
pedronisit runs a bit12:47
pedronisbut emit warnings about argument numbers12:48
pedronisfor calling unbound __init__12:48
pedronisand then crashes on a KeyError12:48
xorAxAxhow about pystones?12:48
pedronissomething related to entry_point12:48
pedronisand at that point I gave up12:48
stakkarsI would have expected more influence from the mentors on concept and layout12:49
pedronisit does some interesting analysises12:49
xorAxAxstakkars: :-)12:49
pedronisbut is far from robust12:49
pedronisso 0.0.1 is indeed the right numbering12:49
stakkarsyes, at some point there is interesting stuff. It isjust such a huge mixture of genc and analyse and whatsoever :(12:50
stakkarsand no tables, everything hacked right in (sorry, should be more polite)12:51
pedroniswell, the author said that has not been tryed beyond the set of around~200 lines programs in the unit tests12:51
stakkarsit would be interesting to know if it would run a bit of RPython code. rpystone?12:52
pedronismaybe rpystone is easier for it12:52
pedronisbut run the unit tests before trying12:53
stakkarsextremely simple code with no classish or dynamic features at all12:53
pedronisbecause I had12:53
pedronisto tweak ss itself12:53
stakkarsand they give problems, too?12:53
pedronisto have compilation working12:53
pedronisfor my setup12:53
pedronisand notice that you need boehm, it uses it12:54
stakkarsah. No I guess this is enough waste of time, already, unless you think we should learn from it.12:54
stakkarsboehm is no problem, I'm using it on windows with PyPy all the time, just because it has a very fast allocator.12:54
pedronisI think there's no support for compiling on window12:56
pedronisbut you can probably stop it after the code is generated12:56
stakkarsthere is windows mentioned, but I think it uses cygwin, there.12:56
pedronisit writes a Makefile for g++12:56
_hannes (i=ore@i577B4751.versanet.de) joined #pypy.12:56
stakkarsI'm actually not interested in something that writes C++, but I'd like to see if it has advantages in type inference?12:58
stakkarsIOW reading the theory would help more than the source12:59
pedronisif you want the theory I think the best thing is still David Grove thesis13:00
pedronisschedskin is based on refinement of that approaches13:00
stakkarsAh. I don't see this mentioned, but Plevyak  (which I didn't read either, maybe it is based on Grove)13:01
stakkarsdo you know what Tiejun & Wang is about and why it is considered incomprehensible?13:02
pedronisare you readin is optimising python?13:03
stakkarspedronis: btw. thanks for the missing bit with rrange, although I should have know. is complete now.13:04
__alex (n=alex@141.22.64.4) joined #pypy.13:04
stakkarsI was reading http://wiki.python.org/moin/SummerOfCode13:04
stakkarsEfficiently Analysing Data Polymorphism and Deducing Generics in Shedskin13:04
pedronisPlevyak is cited in Grove13:10
pedronisI see, Tiejun is more recent13:13
stakkarsA dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy   8-)13:13
pedronishttp://www.cs.jhu.edu/~scott/pll/constraints.html13:14
-NickServ (NickServ@services.)- This nickname is owned by someone else13:25
-NickServ (NickServ@services.)- If this is your nickname, type /msg NickServ IDENTIFY <password>13:25
pybot joined #pypy.13:25
-NickServ (NickServ@services.)- This nickname is owned by someone else13:26
-NickServ (NickServ@services.)- If this is your nickname, type /msg NickServ IDENTIFY <password>13:26
pybot joined #pypy.13:26
stakkarsadded #pypy-tb to eggdrop logging13:26
ericvrp (n=chatzill@ericvrp.demon.nl) joined #pypy.13:26
stakkarswah, horrible inline braces and symbols all over the text :-) a least around page 1813:32
xorAxAxjava? :-)13:33
stakkarssets, lattices, lemmata, theorems and corrolaries in a paper about '''Polymorphic Constraint-Based Type Inference for Objects''', written in math style.13:35
stakkarsno, java is worse :)13:36
fijal (i=root@zodiac.mimuw.edu.pl) left irc: Remote closed the connection13:42
xorAxAxah, those huge braces13:45
stakkarspedronis: is PyPy translatable, again? I am compiling, and I wonder about structure names which seem to be multiple PAGES long? :-)14:19
stakkarsI guess it will break, soon14:19
pedronisit's the Repr passed along to dict methods that generate that names14:20
pedronisI think14:20
stakkarsI guess VC will spit names with 10000 chars at me :)14:20
stakkarsyes, it is dictionary type names14:22
stakkarsmight use an md5 digest, instead :)14:23
pedronisthe fact is that repr of the Repr is used and this one contains the lltype involved14:23
stakkarsI hope they are unique quite early,since most probably soe truncation will occour.14:25
pedronisyes, but it simply a bad approach to have the lltype repr in there14:26
pedronisI know the code involved14:26
pedronisI just need to think exactly what to change14:26
stakkarsI don't care, as log the compiler doesn't as well.14:26
pedronisno, but it's annoying in the graphs too14:27
pedronisbecause you get huge nodes to contain the names14:27
stakkarsreally funny to look at. There is notmuch missing, and you will read your whole program in the names.14:27
pedronisthe fault is the W_Root lltype with all the methods14:29
stakkarsrelated to the change of having no need for position_key?14:30
pedronisno14:31
pedronisrelated to hlinvoke14:32
pedronistaking a repr14:32
stakkarsreally no joke,asa veryeasy hack: whynoch change the repr to its hexdigests, if it is longer than, say, 100 chars?14:34
stakkarssorry about my keyboard. as a very easy hack, one could use the md5 digest for long names.14:34
pedronisthe name show up in the graphs14:35
pedronisand are useful for debugging14:35
stakkarsnot useful if you need to scroll several pages to read the name. I thought to stop afer some 100 and add the digest, then.14:36
pedronisno, they to messy anyway and too long already at that point14:38
pedronisI'm looking at fixing this14:38
stakkarspooooooooof14:40
stakkarstesting_1.c(17680) : fatal error C1064: Compilerlimit: Ɯberlauf des internen Puffers durch Token14:40
stakkarsit's probably possible to implement a trojan into VC++ by using long tokens :)14:41
stakkarsbtw., it would probably be a nice feature instead of encoding reprs in loong names, to have a way to browse the repr structure in PyGames. Or do we have that?14:43
Nick change: tic|away -> tic14:47
stakkarsthe line where VC crashes has 22322 columns. I think we should not be angry with it.14:49
arigo|lunchstakkars: no, we don't have that.  It would be useful indeed14:52
arigo|lunchnote that using a hash for long names is useless; the names don't have to be unique at this point14:52
arigo|lunchso we could just truncate or whatever gives the most readable output14:53
arigo|lunchgenc will ensure the function names become unique in the .c file14:53
stakkarsthe names for the C source have to be unique, methinks. Just less kilobytes 14:53
arigo|lunchit's genc's job to make the names unique14:54
stakkarsyes, I forgot.   (don't forget to unlunch)14:54
arigo|lunchbut it's true that the NameManager could make sure it doesn't return too long names14:54
Nick change: arigo|lunch -> arigo14:54
arigoit's probably a good step towards making sure we generate officially valid C code...14:55
stakkarsit's not too bad to have names that describe the contents. But I'd be hapy with something "I_m_the_repr_for_nice_dicts_of_strings" or somesuch.14:56
stakkarsbut sure, this doesn't help, sice we have a lot of diversification.14:57
stakkarswe are also not using names for substructures, but always tell the contents. Would probably help.14:59
Action: stakkars is just renaming quickly in the C source, using Python, just to see if we can compile15:01
pedronisI'm about to checkin something that makes the name shorter15:03
arigopedronis: then you just beat me :-)15:05
stakkarsand me15:06
arigoah no, what I'd about to check in truncates the names to 50 chars plus prefix plus numeric suffix15:06
pedronisno, I changed the __repr__ of Repr15:06
pedronisdifferent levels15:06
arigook, it's all worthwhile I guess15:06
stakkarswould work, but doing it structurally more level-wise would be even better15:06
arigook, it's all worthwhile I guess15:14
arigosorry, wrong window15:14
mwharigo: damn that up-return finger macro!15:15
arigo:-)15:15
xorAxAxyeah, the irc client should ask "do you really want to press cursor-up?"15:20
stakkars50 chars is quite rigid. but thanks :)15:22
arigooooops I think I broke tests by inserting this [:50]15:40
xorAxAxit looks like a weird smiley15:40
arigo[:-O]15:40
stakkars10I'm looking for an even weirder smiley, to01 comment on my latest test result 8-[]15:47
stakkarscomparing a pypy built on Sept. 04, using boehm and full optimization, versus today's version.15:48
stakkarsrichard's, 1 run, before = 37373 ms   after = 54277 ms15:49
stakkarswe are doing *something* into the wrong direction :-))15:49
pedronisnotice that now you get thread enabled with richards too15:50
pedronisI mean with boehm too15:50
stakkarsargll, yes, this is probably veery expensive, as it is done every opcode!15:51
pedroniswhen they are one, they add a 50% overhead15:51
pedronisare on15:51
stakkarsok, *then* we have a clear advantage, reinterpreting the result.15:51
pedronisno, it's not done on every opcode15:52
pedronisbut yes we still get a huge overhead15:52
stakkarsI think it was at some time.15:52
pedronisthat at some point we need to understand15:52
pedronisyou probably want to disable them to compare15:53
stakkarsis there a simple waytoswitch threads off in the generated source? I don't want tore-generate, exactly because things seem to be broken, again.15:53
arigostakkars: no, and probably no more, respectively15:53
arigoI'm about to check in a simpler malloc-removal algorithm, but I'll try translate_pypy before15:54
stakkarsI have no problem with boehm, just with threads.15:54
arigoI know, I really meant "no you can't switch it off easily in the .c source"15:55
stakkarsand it would be nice to use what boehm does,to getcomparable results.15:55
arigostakkars: I don't get that point15:56
pedronisI think it refers not asking for memory from the OS in small chunks15:56
stakkarsATM, using boehm accelerates pypy by a factor of 2-4, just by the fact that it optimizes malloc in a way.15:57
arigoah, then we should try dropping PyMalloc in here15:57
stakkarsto get comparable results, I'd like to have the same strategy by default, whether using boehm or not.15:57
stakkarsor we compare apples with peaches15:57
stakkarsit would be nice if boehm had a switch that switches it off, but keeps its malloc strategy15:58
stakkarsPyMalloc is probably vry different.15:59
xorAxAxstakkars: have you tried this malloc chunk size global var?15:59
stakkarsno, forgot about that16:00
stakkarsxorAxAx: is this available on more than this platform?16:00
xorAxAxstakkars: it is malloc-specific. the one i talked about was windows-specific16:02
stakkarsI'd like to have something common for all platforms, allocating in a way that the OS influence is neglectible16:02
xorAxAxthere are malloc replacements (if you want to use ref counting)16:03
stakkarsbut if we use PyMalloc, how do I replace things for boehm.16:03
xorAxAxwhat do you mean by replace?16:03
xorAxAxyou mean the calls?16:03
stakkarsboehm already does something to make the OS overhead neglectible. But I don't want to stack malloc strategies.16:04
stakkarsor even mixing them is a bad idea.16:04
xorAxAxthe overhead consists of allocating heap chunks16:04
xorAxAxand it is high if you do that often16:04
xorAxAxboehm doesnt do it often16:04
xorAxAxnormal malloc does on windows because the chunks are small16:04
xorAxAxso you need an allocator with large chunks which might be optimised for small objects if you dont trust the OS malloc16:05
stakkarssure, that's not the problem. The problem is that with boehm, I buy one strategy, and with PyMalloc another one. Both have the side effect of reducing OS overhead, but the resultd are still not comparable.16:06
xorAxAxwhy do you want comparable results in this case?16:09
pedroniswell, you could try to use GC_MALLOC_ATOMIC and GC_FREE16:09
pedroniswith reference counting16:10
stakkarsthesemethods are borrowedfrom boehm, then?16:11
pedronisor GC_MALLOCA_UNCOLLETABLE16:11
stakkarssounds good.16:11
pedronisthe problem is that there is not public interface meaning both ATOMIC+UNCOLLETABLE16:11
stakkarsATM I have the problem how to switch threads off. There is no entry in translate_pypy --help16:12
pedronisno16:12
xorAxAx--damn-dont-use-modules:thread :-)16:12
stakkarsI like that, please add :)16:13
stakkars- sigh -16:13
stakkarsdo we use the new translate_pypy,already, or do I patch the old one?16:16
arigostakkars: target-specific options are messy anyway16:17
arigofor now you just change a line in targetpypystandalone16:17
pedronisyes, one of the things that the refactoring should fix16:17
stakkarswe don't want the option, and you don't care if I disable it?16:17
stakkarsmakes sense ATM since we know it works and that it eats lots of time for a reason to investigate not now16:18
arigoyes (though you can also not check it in)16:18
stakkars(I really hate the latter because at some time I *will* check it in if I'm not careful)   aye aye16:19
arigostakkars: then copy it to targetpypystandalonewithoutthreads.py :-)16:19
stakkarsROTFLMAO16:20
arigonote that the rdict change was an un-optimization for string-keyed dicts16:20
arigobecause they now carry two extra flags in each entry, just like every dict16:21
stakkarsyes,but I guess not an expensiveone. I believed the faster dict access in richards would outperform this16:21
stakkarsthis is of course expensive and woul be even more if we use refcounting. That's why I always use boehm:object access is for free.16:22
pedronisarigo: what kind of simpler malloc removal are you working on?16:23
arigohttp://codespeak.net/svn/user/arigo/hack/pypy-hack/remove_malloc2.py16:24
arigo(which I am moving to backendoptimization.py)16:24
stakkarsok, compiling with targetpypystandalonewithoutthreads.py, boehm, takes 40 minutes16:25
arigofor a graph, it divides the vars in families, where two vars are in the same family if a value can go from one to the other;16:25
arigothe family records all usages of its variables16:25
arigothen we just use this information to see which families correspond to malloc/getfield/setfield only16:26
arigoit's < 200 lines of code, as opposed to 500 for the previous attempt16:26
arigobut it must be re-run iteratively, as inlining structures may expose more opportunity for inlining16:26
pedronisI see, yes is similar to what I had in mind16:29
pedronisthe lifetime part can be reused to properly detect16:29
pedroniswhat exception a graph can raise16:29
arigoI guess so16:30
pedronisof course it needs to be comibined with helpers inlining to get much effect16:31
arigoyes16:31
arigoI ran the tests, it only applied once16:32
pedronisnot too strange16:32
stakkarsbreaks structures into pieces and tries to allocate on stack?16:34
arigojust breaks structures into variables16:34
stakkarsso if I have a list, the length becomes a local variable, and the data is still a malloced array?16:35
arigohum, I'm confused16:36
arigothere are more snippets where it should succeed16:36
pedronisprobably some shallow problem?16:37
stakkarsif that array can be made local, that's it.16:37
arigopedronis: indeed, test_backendoptimized doesn't run all backend optimizations16:38
pedronisit list them esplicitly16:40
arigothat's two more tests where it has an effect :-)16:40
arigostakkars: this list optimization is still not quite achievable because most list operations call a helper16:40
arigothat's why inlining is important: to help this malloc removal process16:41
stakkarsyes, it would also imply inlining of those I think16:41
arigono, you need to inline the helpers before you can detect that you can remove the mallocs16:42
arigounless we go for some more advanced algorithms (i.e. > 200 lines :-)16:42
stakkarsthis is what I thought. Did I write differently?16:42
pedroniswell, I think it is worth to see how fat the simple approaches bring us16:42
pedronisbefore trying more complicated things16:43
arigostakkars: what did you mean?16:43
stakkarswhyt you said. you first inline the helpers, then the listis no longer exposed, then you figure that you can melt the structure away, and so on.16:44
arigook16:44
pedronisyes, that's the idea16:44
pedronisand iterators are probably the most interesting thing16:45
pedronisbefore lists16:45
stakkarsand you do that until you get to some very huge limits, and then you try todo the opposite :-)16:45
pedronisI don't think we have in mind to indiscriminately inline all helpers16:45
stakkarsyou think iterators (heh, I'm joking) would turn themselves into the nice, efficient loops arigo mentioned, without extra magic but inlining? would be great.16:46
pedronisweel, we need removal of unused malloc results16:48
pedronisbecause as Carl noticed the inlining still leaves back the exception construction16:49
arigoshould melt away already with the remove_simple_malloc()16:50
stakkarsthe exception handler for the malloc? Maybe this relationship could be recorded when we create the malloc16:50
stakkarsah, the exception handler around no code would just go away.16:50
arigowow, the results of the inlining are impressive16:51
arigothese mallocs will go away, no problem -- we're almost there :-)16:51
stakkars$-D $-D $-D 16:51
pedronisarigo: you are looking at Carl tests16:52
arigoyes16:52
arigoif we add t.backend_optimizations() in test_inline_exceptions16:52
arigothen f() melts away completely16:52
arigoI mean g(), even16:53
arigoit leaves a couple of empty blocks behind, and only two int_eq() operations16:53
Action: stakkars is jumping on his chair like a child at Xmas evening16:54
sabiheh16:54
arigo.-)16:54
arigohum, test_inline_var_exception() still needs some work16:55
arigonot sure that we care at the moment, though16:55
pedronisnotice that tests16:55
pedronisoops16:55
pedronisnotice that that tests uncovered16:55
pedronisan annotation problem16:56
arigo:-)16:56
pedronisis really a bit atypical16:56
arigook16:56
pedronisthe change in the ordering of issubtype16:56
pedroniswas because of it16:56
pedronisI also think that we don't do the correct thing16:56
pedroniswith issubtype and isinstance16:57
pedronisabout considering old style classes16:57
pedronisnew style16:57
pedronisfrom the annotator point of view16:57
pedronisalthough the test is not affected because it deals with instances16:57
pedronisand not classes16:57
arigoI see16:58
arigofor now I'd like to see if we can inline some helpers automatically16:58
arigoand reach this for loop simplification16:58
arigobasically, we should look at all call sites and decide if it should be inlined16:58
arigowe need to figure out a not too unreasonable condition16:59
pedronisyes16:59
pedronisit would be good to have a count of ops16:59
pedronisin the graphs16:59
arigowell it's easy to count16:59
pedronisthe other thing if whether it loops16:59
arigoand it would be messy to maintain, given how we change the graphs in-place16:59
arigointuitively, not inlining functions that loop simplifies things, is it a good criteria?17:00
xorAxAxi think it solely depends on the block size17:01
xorAxAxis loop setup expensive in pypy?17:01
pedronisno but the problem with loops17:02
pedronisis the amount of code size17:02
xorAxAxsmall loops exist :)17:02
pedronisand cache misses17:02
pedronisvs non gaining much17:02
pedronisbecause the call is much faster than the looping17:02
arigoI see17:02
xorAxAxdepends on the no. of passes17:02
pedronisbut it's all a trade-offs17:03
xorAxAxi would look at the loop setup code and see if it trashes cache lines or is longer than a call17:03
arigowe could count some wild estimates of how many operations are likely to be executed in the called functions17:03
pedronisyes17:04
arigoby doing things like "here's a branch, let's say it's a 50%-50% chance to follow each path"17:04
arigoa bit of linear algebra if there are loops...17:04
arigothen we add a constant for the call overhead17:04
arigodivide by the total number of operations for the size overhead17:05
arigosomething arbitrary like this17:05
stakkarsnew results on the richards front: still quite messy. no threads, boehm, all options identical to before one week.17:05
stakkarsbefore: 36822 ms   after: 46717 ms17:05
arigoI'm still wondering how much removing of the string dict special case hurts17:06
pedroniswell, considering that involves in all ops dispatch17:06
stakkarswe are probably using this in the interpreter much more heavily than I expected.17:06
pedronisis involved17:06
arigostill, the difference is big; maybe there is a lot to gain by tuning this carefully (later)17:07
pedronisdid we change something else?17:08
arigoI'm thinking about specializations (later again) for cases where we know two special values for the keys (for pristine and deleted)17:08
arigogood question17:09
arigowe should compare around the change that removed strdicts17:09
stakkarswhat do we use string keys for. in opcodes???17:09
arigostakkars: to look up special method names, all the time17:09
pedronisfor the dict_w of types17:09
pedronisthat is accessed all the time17:09
pedronisin lookups17:09
arigoarf17:11
stakkarstypes of applevel objects17:11
arigolook at translator.backendoptimization.collect_called_functions() attentively17:12
pedronisstakkars: yes types of applevel objects, int str etc17:13
stakkarsI wasn't aware that we could deduce those to be string only dicts, already.17:14
arigobut we only supported string only dicts, not so long ago17:16
pedronisarigo: it returns nothing17:16
arigopedronis: indeed17:16
stakkarsarigo: yes, it's a nice function. Could be a bit more useful if it would use its visit :-)17:16
arigoso there's a small problem17:16
arigothe next() of iterators do a "raise StopIteration"17:16
arigowhich is equivalent to "raise StopIteration()"17:16
arigoi.e. they contain a direct_call17:16
arigoso right now they can't be inlined17:16
arigo? sorry, I am talking nonsense17:17
Action: arigo confused by a failing test17:17
pedronisit seems that carl added some exception support17:19
pedronisbut did not remove the assert17:20
stakkarsanyway, collect_called_functions is a complete no-op17:20
arigoI think the assert really means "there is no direct_call in the graph to inline"17:21
arigoI'll have to try to see if this is now supported17:21
arigothere is no test, so I'm not sure17:21
pedronisI see17:23
pedronisprobably17:23
pedronisnot17:23
pedronisbecause you need recursive17:23
pedronisinfo about17:23
pedroniswhat the function can raise17:23
Action: pedronis needs to logout, my window manager got slightly insane17:25
pedronis (i=pedronis@ratthing-b246.strakt.com) left irc: "Chatzilla 0.9.67 [Firefox 1.0.2/20050325]"17:25
stakkarswell, starting with functions that don't call any other functions, inlining always just those,you have a fixed set of exceptions that propagates up.17:25
arigostakkars: yes, that's part of the non-sense I realized17:26
arigoeven if a small helper calls another small helper, we can inline in two steps17:26
pedronis (i=pedronis@ratthing-b246.strakt.com) joined #pypy.17:26
stakkarswhat's non-sense?17:27
arigowhat I said above the line "? sorry, I am talking nonsense"17:30
stakkarsah, and I thought the approach to start inlining with the bottom level funcs (leafs) was non-sense.17:31
pedronisyes, the issue is inlining something small that calls something big17:31
arigoright17:31
arigo...while in addition catching an exception explicitely on the outermost call17:31
pedronisbecause the bottom up approach doesn't work in that case17:32
arigo(otherwise, it works already)17:32
stakkarswhile I doubt this case is so importante,since the codewillspend most time in the calledbig one.17:32
arigoyes, though remember it's also about enabling further optimizations17:33
arigobut indeed, I doubt this case is common17:33
stakkarscorrection: if thjis leads to inlining of locals, then you are absolutely right17:33
pedronisit's more relevant as a dynamic opt probably17:33
pedronisbecause the small one could contain a fast-path for some case17:33
pedronisdiferring to the big one otherwise17:33
stakkarswhat are you trying: inline the smallee into its caller, or the bigee into the smallee?17:35
pedronisthe smaller17:35
stakkarsinto its caller, which then needs to cope with the biggies unknown exceptions17:36
pedronisyes, but it's not a priority I think17:39
pedronisI'm still wondering why we got slower btw17:40
stakkarswell, maybe it is, if I think of the three-times indirection of multimethod calls, which are often some tiny wrappers around something bigger, which actually has some fast path. But you always walk through this shashlik.17:41
stakkarsI gues it would even pay off if we just do worst-case assumptions of exceptions.17:41
arigostakkars: in this case, most all calls are "virtual", so no inlining possible17:42
arre (i=ac@ratthing-b40e.strakt.com) left irc: Remote closed the connection17:42
stakkarswhen I single step through marshal_w calls, I see helper functions called which call something from their minilocals, which is actually a constant.17:44
pedronisyes we have a bit of that17:45
stakkarsI agree that it ends upin a virtual call,but the preparation for that can be flattened.17:45
stakkarsat least we could handle these minilocals entries like struct  members and dispose of the dicts17:47
pedroniseh?17:47
pedronisthe constant are captured by the flow graph17:48
arigothe minilocals is not a dict any more than the usual globals17:48
stakkarsfine, didn't know that. sorry, I'm missing a small enough C file tofigure out what exactly is gone and what stays.17:49
pedronisthe change to the dict impls was done at 1774017:49
Nick change: tic -> tic|away17:50
stakkars(or bad if the fruit hangs higher and higher)17:50
pedronisso the question is what was the speed before that17:51
stakkarsdo you need a test?17:52
stakkarsarigo: sure,it as adumb assumption after I know what happens toglobals, whether mini or not :-)17:53
pedronisstakkars: yes, you could try to compile the rev before17:53
stakkarsok,please givemean existing rev, this time17:54
stakkars17470?17:55
pedronis1743917:55
pedronisthe change was at 1744017:55
pedronisI typed a seven too much17:55
ironfroggy (n=calvin@68.83.122.167) left #pypy ("Konversation terminated!").17:56
stakkarsok, reverting to that.17:56
pedronisto 1743917:56
stakkarsthere was one change to the build that might affect translation: the /Op option to the compiler. But I doubt it is relevant17:59
arigodoes it only affect float math?17:59
stakkarsthe option is documented as doing just that,yes. But who knows18:00
stakkarsthe 17439 compilation now runs without that. If in doubt, I can do it for the current version as well.18:00
pedronisrichards doesn't use floats I think18:01
stakkarsno, and it wouldn't even count, if you look into the C code what it's busy with :)18:02
stakkarsreal work is still little compared to overhead18:03
Action: stakkars will stop the day after this, frustrated, produced almost nothing but hot air, buhuu18:04
stakkarsbtw. I see that valid and everused are mapped to char in dict entries. DO we deduce such automatically,or is theresome hint?18:29
pedroniswhen we define the struct18:30
pedronisrdict.py18:30
pedronisself.DICTENTRY =18:31
HelmutG (i=helmut@subdivi.de) joined #pypy.18:33
HelmutGHi. Do you think having Debian packages would be some sort of contribution or is this generally unneeded?18:34
arigostakkars: Bool maps to a C 'char' in genc.py18:34
arigoHelmutG: hi!  no, I think it's not useful at the moment -- thanks for the proposal18:34
HelmutGarigo: Can you estimate a point when it might be useful?18:35
arigohard to say18:35
arigoit would be when PyPy itself starts to become useful, which it definitely isn't at the moment18:36
HelmutGarigo: Would 6 to 12 months be such a range?18:36
arigoI'd assume so18:36
HelmutGThat's what I wanted to know, thanks.18:36
arigo:-)18:36
HelmutGTell me if you need a maintainer.18:36
arigook, we've got you in the logs, when we need to look back :-)18:37
stakkarsnew benchmarks, soon18:42
pedronisgood18:43
arigoI wrote a function to compute the statistical execution cost of a function18:43
arigobut it uses Numeric18:43
arigoto inverse a matrix18:43
arigois there some code available around there to do that in pure Python?18:43
pedronisdoesn't pygame requires Numberic too18:44
arigono, optionally18:44
arigoI wouldn't like to add Numeric as a dependency at this point18:45
stakkarspypy-c-17439.exe 40707 ms18:45
pedronisso we are progressively getting slower18:46
stakkarspypy-c-current.exe 45946 ms18:47
stakkars(my measure differ,depending on ho much processes are there, but I always run both in the same env)18:47
pedronisbut it seems that overall the dict changes did not improve things18:48
stakkarspython 2.3.3  850 ms  but for 10 iterations18:49
xorAxAxfactor 540?18:49
arigothen we need again specialization for str keys18:49
stakkarsI fearso, yes18:49
arigowell, wait a minute18:49
arigoit's also possible that there is simply an extra call indirection somewhere18:50
pedronisyes, I was thinking about that18:50
arigoall these numbers are a bit meaningless before we can remove the most horrible inefficiencies18:50
stakkarsyes, I observed jumps by 20 percent for no good reason at all, which might just be due to code ordering18:51
pedronisI was the str dict impl calling hash and equality18:51
stakkarswe must be loosing heavily at some point which we don't know18:52
stakkarsdoing pystone, now18:52
stakkarsstunning results,all over the place !!!18:55
stakkarsversion 17439    592.966  pystones18:55
stakkarsversion current   615.233 pystones18:55
stakkarspython 2.3.3  37986.218:55
pedronisarigo: the string implementation of comparing hashes the before doing eq18:56
pedroniss/of/was18:56
pedronisalthough the string we are mostly interested in should be interned18:56
stakkarswe must be incredibly worse at instance attribute access.18:56
arigoI see18:57
stakkarswhich we thought would become better with the new dict18:57
pedronisbut we don't intern strings in code18:58
pedronisso we are probably worse18:58
stakkarsbut we use cached strig hashes I hope18:59
pedronisno, but the new dict impl18:59
pedronisbefore trying equality18:59
pedronisjust tries identity18:59
stakkarswhich always fails19:00
pedroniswell, not for the string in corresponding to special methods in types19:01
pedronisbut probably yes19:01
stakkars540, compared to about 70 with pystone19:01
pedronisfor the app-level strings in instance dicts19:01
stakkarswhich richards is exercizing exhaustively19:01
dialtone (n=dialtone@merlinux.de) left irc: Read error: 104 (Connection reset by peer)19:04
hpk (n=hpk@merlinux.de) left irc: Read error: 104 (Connection reset by peer)19:04
hpk (n=hpk@merlinux.de) joined #pypy.19:05
stakkarswe really really need interning19:05
dialtone (n=dialtone@merlinux.de) joined #pypy.19:05
pedronisour hold interp level dict implementation was caching the hash19:06
pedronisin the entries19:06
pedronisand comparing it19:06
stakkarswhich is like a surrogate for interning. And we don't cache hash in the string impl?19:07
pedroniswell, now everything happens one level down19:07
stakkarslevel down is ok, but why slow down. :)19:08
pedroniswell, CPython caches the hash too19:10
stakkarsyes, in the dict table and in the strings19:11
stakkarsbeating a Python dict's speed is really hard. AFAIK it'smuch faster than all the C++ mapping libraries19:12
xorAxAxwhats the algorithm behind?19:13
stakkarsdictobject.c, dict_look explains it.19:15
pedronisstakkars: I think that we should cache the hash too in rdict for non primitive types, although it will make the code more involved19:15
stakkarsusing a good algorithm to cycle through all entries, using size optimized to have an 1.5 change to find the entry on the first hit, caching hashes, well I can't write a book, here19:16
stakkarspedronis: absolutely. acutally for all pointer types?19:17
pedronisbasically the repr of the key should tell whether to or not19:17
pedronisfor ints is meaningful19:17
pedronisis not19:17
stakkarsand table layout becomes dependant of what it will contain19:18
pedronisyes19:18
pedronisand you need more flags19:18
stakkarssure, not the types which are almost their own hash19:18
pedronisin the repr19:18
stakkarsmore flags than CPython?19:18
stakkarsah, in the repr19:18
pedronisyes to control what operations lookup should do19:18
stakkarsat some point duplicate lookups might be more readable than one hyper generic one19:19
pedronislikely yes; of course we have the problem with valid and everused and hash and the overall used space19:20
stakkarswhere valid and everused are necoded differently in CPython,because they just have pointer?19:20
pedronisyes19:21
stakkarsas an observation: The efficiency that we lost by unoptimizing partially is outweighing the win that we wanted to gain by turning the slot entries into a table.19:22
stakkarsI said it cannot stay this way,and I was wrong unless we push further.19:22
pedroniswell, not using the cached hash is a big it on lookup19:23
pedronisand the entries were a major cost on setitem and a bit the indirection19:23
stakkarsmaybe we turn it just on everywhere and remove it for simple types,later?19:24
pedronisyou mean the cached hash19:24
pedronis?19:24
stakkarsyup19:24
pedronisyes, you could try with that19:24
pedronisand see how much it helps19:24
pedronisI'm about to go home, will join later from there19:25
pedronissee you19:25
pedronis (i=pedronis@ratthing-b246.strakt.com) left irc: "Chatzilla 0.9.67 [Firefox 1.0.2/20050325]"19:25
stakkarsbye19:26
arigobye19:40
arigo (n=arigo@134.99.112.244) left irc: Remote closed the connection19:40
token_ (n=qwertyui@200.193.161.60) joined #pypy.19:43
aleale (n=andersle@clogs.dfki.uni-sb.de) left irc: Read error: 110 (Connection timed out)19:46
pedronis (n=Samuele_@213.112.139.57) joined #pypy.20:00
idnar (i=mithrand@unaffiliated/idnar) joined #pypy.20:16
idnar (i=mithrand@unaffiliated/idnar) left irc: Client Quit20:19
ericvrppedronis, arigo: the LLVM formula for inline costs can be found in llvm/lib/Transforms/IPO/InlineSimple.cpp20:21
ericvrp (n=chatzill@ericvrp.demon.nl) left irc: "Chatzilla 0.9.68.5.1 [Firefox 1.0.6/20050727]"20:51
__alex (n=alex@141.22.64.4) left irc: Remote closed the connection21:10
ludal (n=ludal@lab75-1-81-57-254-81.fbx.proxad.net) joined #pypy.21:17
stakkarspedronis:had a small bug,building once more. we will know more in 40 min21:48
stakkars(will need to augment test_rdict, resizing is not properly tested)21:49
HelmutG (i=helmut@subdivi.de) left #pypy.21:53
stakkarsI'm pretty sure this will show quite a difference. On instance dicts,calling the string compare via callback and through the objspace abstraction is probably very expensive, which the hash cache minimizes21:53
ludalhi stakkars21:55
stakkarshi21:55
ludalcan you translate_pypy with latest svn ?21:56
stakkarsyes21:56
ludalok so it must be my changes then ;)21:56
stakkarsrev. 17512 is what I compiled last.21:56
Rhamphoryncus (n=rhamph@unaffiliated/rhamphoryncus) joined #pypy.21:57
ludalthat's what I'm trying now, although it's with targetcompiler.py21:57
stakkarsthat's different.I use targetpypystandalonewithoutthread :-)21:58
ludalbtw did you try to profile the translation process ?21:58
stakkars(my special version where I hacked threads away)21:58
stakkarsyes.21:58
stakkarsyou cannot do it with profile.py, that thing seems to have deeper bugs.21:59
ludalwell, the bugs I saw came from the changes I introduced and reworked by pedronis21:59
ludalusing hotspot ?21:59
stakkarsI got it with hotspot, yes.21:59
stakkarsprofile.py is not happy with PyPy at all. It crashes in the rtyper phase.22:00
xorAxAxisnt it python-only?22:00
stakkarsyes. But it seems to have some assumption about which frames should be where when, which PyPy does not fit :)22:01
xorAxAxand what kind of crash do you see?22:02
stakkarswrong frame, it stops with a message. just try it22:02
stakkarsit works when I run PyPy upon Python 2.3, but not on 2.422:03
xorAxAxok22:03
stakkarsand hotspot is much faster. drawback: it takes years to read its huge stats file in.22:03
ludalis there a starting option to run pypy and/or translation with profiling ?22:05
pedronisludal: I'm trying targetcompiler22:06
pedronisbut the problem I'm seeing seems related22:06
pedronisto the changes to iterators22:06
pedronisand the fact that the union22:06
pedroniswas not updated22:06
pedronisfor them22:06
ludalyes22:06
ludalI think I fixed that with my local changes because I don't see the bug any more22:07
pedronisdo you see another one?22:07
ludalI just got stuck on another dict[block]=block22:07
ludalfor now no22:07
pedronisin theory we can support22:07
pedronisdict on instances22:07
pedronisI'm with instances as keys22:08
pedronisusing idenities22:08
pedronisis just that things are not wired in the rtyper for that22:08
ludalthe fixes I made are not entirely related but I thought it would be a good idea to implement contains on some types (Float, Integer, Bool) without using pair(x,y).union22:08
ludalyes it actually crashes in the rtyper22:09
ludalwhenever I see dict of blocks I can change it easily since blocks have a .bid field which uniquely identifies them22:09
pedronisyes, but we could simply enable supporting them22:10
ludalsure, and you'll probably finish it before I get through with the compiler ;)22:10
pedronisno, it's a bit involved not too much but a bit22:11
pedronisbecause hash support for instances need to be enabled per class22:12
ludalhash support as with __hash__ or more complex than that ?22:12
pedronisbasically when we see a SomeInstance has key of a dict with to put the classdef on a table22:13
pedronison the bookkeeper22:14
ludalbtw It may be just an impression, but the translation process with .contains changes seem to use less memory (I'm at 190Mb still raising so it may just be an impression)22:14
pedronisdid you write tests :) ?22:14
ludalwell there is already test_contains for this one22:14
ludaland they pass22:14
ludalas for SomeBool changes I made earlier I couldn't find any test checking the semantics of equality, so I just followed the trend ;)22:15
ludalthe thing is there is this assert s_value.contains[self.bindings[arg]] somewhere in annrpython.py 22:16
pedronisyes22:16
pedronisis quite useful22:16
ludaland I just don't know what it's trying to verify22:16
ludalthe thing is after the changes I triggered it that was the reason I changed SomeBool.__eq__22:17
pedronisthat new bindings are generalisation of the old ones22:17
pedronisfor knowndatatype is bit trickier to make that happy22:17
ludalwhen you rebind an existing variable ?22:17
pedronisyes22:18
pedronisit checks the basic invariant22:18
pedronisof the whole annotation process22:18
ludalwell for testing the changes I just compiled pypy and it worked :-/22:18
ludalok22:18
ludalI got this "TyperError: can only iterate over tuples of length 1 for now"22:20
ludalcan I just convert my tuple to a list ?22:20
pedronisyes22:21
pedronisI'm writing a test about dicts with instance keys22:21
pedroniswill try to make it pass22:22
ludalok22:22
ludalpedronis I have this local fix for iterators:22:25
ludal class __extend__(pairtype(SomeIterator, SomeIterator)):22:25
ludal 22:25
ludal     def union((iter1, iter2)):22:25
ludal-        return SomeIterator(unionof(iter1.s_container, iter2.s_container))22:25
ludal+        cont = unionof(iter1.s_container, iter2.s_container)22:25
ludal+        assert iter1.variant == iter2.variant22:25
ludal+        return SomeIterator(cont,*iter1.variant)22:25
pedronismakes sense22:25
ludalI'm not sure I got all the subtelties of variant22:26
ludalit seems to be used only for dicts?22:26
ludaland although its a tuple I could only see one or zero item in it...22:26
_hannes (i=ore@i577B4751.versanet.de) left irc: "utz utz utz"22:51
pedronisludal: I'm running the tests but it seems the basic support for instance keys is there22:51
ludalthis could ease the rest of the translation :)22:52
stakkarspedronis: new timings!22:53
stakkarsfirstof all asmallcorrection:22:53
ludaldid you comit btw cause I just got another one of those dict{Block:Block} ?22:53
pedronisI'm running the tests22:53
stakkarswe are not that bad as it sounded. My answer to xorAxAx was wrong: I was always comparing average time.22:53
pedroniswell some tests22:53
stakkarsso the factor was about 56,not 10 times worse.22:54
xorAxAxstakkars: ah, ok22:54
stakkarsthen, here what we got by the simple hash cache in the dict table:22:54
stakkarspypy-c-17439: 40929 ms    pypy-c-17512: 46105 ms   pypy-current: 33937 ms22:55
stakkarsall average for one run22:55
stakkarsthis shows how very expensive the string comparison call into the space is.22:56
stakkarspython 2.3: 856 ms22:57
stakkarswhich is a very good factorbelow 40, even better than pystone !!22:58
pedronisconsindering that 0.7 had a factor of 300 or worse (with threads)22:59
token_ (n=qwertyui@200.193.161.60) left #pypy ("Fui embora").23:00
stakkarsI have boehm, no threads, -t-lowmem (which is probably not relevant)23:00
pedronisyes, 450 was threads and refcounting23:01
stakkarsstill, it is probably not the refcounting, but the direct malloc that makes everything else so relative.23:02
pedronisbut we are going in the right direction23:02
stakkarsyes,not backwards is quite a relief23:02
stakkarsand now the new dicts really paif off, although we carry the overhead.23:03
stakkarspay off23:03
stakkarsnow a lot can be gained by special casing a few hash functions. Would be great if we knew23:04
pedronisludal: checked in23:05
stakkarsthat an istance dict uses string keys only. DO we know that?23:05
pedronisit's a w_dict23:05
pedronisso it has W_Root keys23:05
pedronisnot strings23:05
stakkarsmaybe better not now but at a later point23:05
ludalok23:05
stakkarsyes, but it will contain wrapped string keys, only.23:06
stakkarscan't wefind that out? probably not on an application program. But we might try that first.23:06
pedroniswell, in CPython  you can inst.__dict__[1] = 223:07
stakkarssure I can. But I can assume that the common case is a string key, and tweak dict hash to find that variant first23:07
Rhamphoryncus (n=rhamph@unaffiliated/rhamphoryncus) left irc: "Support ISO 8601! http://www.cl.cam.ac.uk/~mgk25/iso-time.html"23:34
pedronisgoing to try to sleep23:55
pedronisnight23:55
pedronis (n=Samuele_@213.112.139.57) left irc: "Chatzilla 0.9.68a [Firefox 1.0.4/20050511]"23:55
--- Tue Sep 13 200500:00

Generated by irclog2html.py 2.9.2 by Marius Gedminas - find it at mg.pov.lt!