| dialtone (n=dialtone@merlinux.de) left irc: Read error: 110 (Connection timed out) | 00:59 | |
| pedronis (n=Samuele_@c-398b70d5.022-54-67626719.cust.bredbandsbolaget.se) left irc: "Chatzilla 0.9.68a [Firefox 1.0.4/20050511]" | 01:12 | |
| Action: stakkars thinks nine months fits the idea of a baby, nicely | 01:38 | |
| fijal (i=root@zodiac.mimuw.edu.pl) left irc: Remote closed the connection | 02:33 | |
| stakkars (i=kxwsqz@i577B4676.versanet.de) left irc: Read error: 110 (Connection timed out) | 03:42 | |
| hpk (n=hpk@merlinux.de) got netsplit. | 03:54 | |
| thingie24 (n=asdf@valhalla.ccp.cc) got netsplit. | 03:54 | |
| xorAxAx (n=xorAxax@moinmoin/coreteam/alexander) got netsplit. | 03:54 | |
| mwh (n=user@82-33-185-193.cable.ubr01.azte.blueyonder.co.uk) got netsplit. | 03:54 | |
| pyb0t (i=pybot@nimrod.terra-link.net) got netsplit. | 03:54 | |
| Rhamphoryncus (n=rhamph@unaffiliated/rhamphoryncus) got netsplit. | 03:54 | |
| ironfroggy (n=calvin@68.83.122.167) got netsplit. | 03:54 | |
| sabi (i=njriley@shrug.csl.uiuc.edu) left irc: Read error: 104 (Connection reset by peer) | 03:54 | |
| typo_ (n=enki@port-212-202-201-68.dynamic.qsc.de) left irc: Read error: 104 (Connection reset by peer) | 03:54 | |
| typo_ (n=enki@port-212-202-201-68.dynamic.qsc.de) joined #pypy. | 03:54 | |
| sabi (i=njriley@shrug.csl.uiuc.edu) joined #pypy. | 03:55 | |
| pyb0t (i=pybot@nimrod.terra-link.net) returned to #pypy. | 03:55 | |
| Rhamphoryncus (n=rhamph@unaffiliated/rhamphoryncus) returned to #pypy. | 03:55 | |
| hpk (n=hpk@merlinux.de) returned to #pypy. | 03:55 | |
| mwh (n=user@82-33-185-193.cable.ubr01.azte.blueyonder.co.uk) returned to #pypy. | 03:55 | |
| thingie24 (n=asdf@valhalla.ccp.cc) returned to #pypy. | 03:55 | |
| ironfroggy (n=calvin@68.83.122.167) returned to #pypy. | 03:55 | |
| xorAxAx (n=xorAxax@moinmoin/coreteam/alexander) returned to #pypy. | 03:55 | |
| -NickServ (NickServ@services.)- This nickname is owned by someone else | 03:55 | |
| -NickServ (NickServ@services.)- If this is your nickname, type /msg NickServ IDENTIFY <password> | 03:55 | |
| politik (n=d@c-24-147-183-28.hsd1.ma.comcast.net) left irc: Read error: 104 (Connection reset by peer) | 03:55 | |
| typo_ (n=enki@port-212-202-201-68.dynamic.qsc.de) left irc: "x" | 04:05 | |
| idnar (i=mithrand@unaffiliated/idnar) left irc: Nick collision from services. | 06:13 | |
| idnar_ (i=mithrand@unaffiliated/idnar) joined #pypy. | 06:14 | |
| idnar_ (i=mithrand@unaffiliated/idnar) left irc: Remote closed the connection | 06:58 | |
| idnar (n=mithrand@unaffiliated/idnar) joined #pypy. | 06:59 | |
| adim (n=adim@logilab.net2.nerim.net) joined #pypy. | 07:14 | |
| idnar (n=mithrand@unaffiliated/idnar) left irc: Remote closed the connection | 07:17 | |
| idnar (n=mithrand@unaffiliated/idnar) joined #pypy. | 07:18 | |
| dialtone (n=dialtone@merlinux.de) joined #pypy. | 07:27 | |
| idnar (n=mithrand@unaffiliated/idnar) left irc: Remote closed the connection | 07:38 | |
| adim (n=adim@logilab.net2.nerim.net) left #pypy. | 07:41 | |
| dialtone (n=dialtone@merlinux.de) left irc: Read error: 110 (Connection timed out) | 07:52 | |
| fijal (i=root@zodiac.mimuw.edu.pl) joined #pypy. | 08:05 | |
| fijal (i=root@zodiac.mimuw.edu.pl) left irc: Remote closed the connection | 08:08 | |
| fijal (i=root@zodiac.mimuw.edu.pl) joined #pypy. | 08:08 | |
| dialtone (n=dialtone@merlinux.de) joined #pypy. | 08:12 | |
| arre (i=ac@ratthing-b40e.strakt.com) joined #pypy. | 08:33 | |
| stakkars (i=rmtdttv@i577B4751.versanet.de) joined #pypy. | 08:47 | |
| arigo (n=arigo@134.99.112.244) joined #pypy. | 09:15 | |
| arigo | good, ports to the server have been opened | 09:55 |
|---|---|---|
| stakkars | good morning! | 10:03 |
| arigo | morning! | 10:04 |
| mwh | morning! | 10:04 |
| stakkars | now I thought I had a complete low level range implementation (and it works just fine), | 10:05 |
| stakkars | but then I figured out that we do differently from CPython, concerning ranges with > maxint elements. | 10:05 |
| stakkars | I'm wondering how to handle that, ignoring is probably not ok, at least I should check for it. | 10:06 |
| arigo | what is the difference exactly? | 10:06 |
| stakkars | also, the CPython version doesn't store stop, but the length, which would have been my approach, too. | 10:06 |
| stakkars | well, xrange length is an unsigned, they allow the full range. | 10:07 |
| stakkars | if rrange did this too, I could use it to implement applevel xrange,but this way it is not powerful enough. | 10:07 |
| stakkars | (not sure if xrange is correct in this sense, too, checking...) | 10:08 |
| arigo | it doesn't make much sense to use rrange to implement app-level xrange, as far as I see | 10:09 |
| stakkars | difference? you can have xrange(-maxint, maxint) | 10:09 |
| arigo | in CPython? no... | 10:09 |
| stakkars | yes.. | 10:09 |
| arigo | >>> xrange(-sys.maxint, sys.maxint) | 10:10 |
| arigo | Traceback (most recent call last): | 10:10 |
| arigo | File "<stdin>", line 1, in ? | 10:10 |
| arigo | OverflowError: xrange() result has too many items | 10:10 |
| stakkars | ok,almost. Anything that doesn'tlet unsigned long wrap. | 10:11 |
| arigo | ? no | 10:11 |
| arigo | xrange(-1, sys.maxint) -> OverflowError | 10:11 |
| arigo | sorry, I'm lost -- I really see a 'long len' in rangeobject.c | 10:12 |
| stakkars | hum, that was what confused me. Of course they use a long, and I thought they would use the full range. | 10:12 |
| stakkars | but they don't do it, cause the result is assigned to a signed long | 10:13 |
| stakkars | not sure what the injtent was. | 10:13 |
| stakkars | and yes. xrange.len is a signed. Dumb ofme. But why do they the unsigned crap. | 10:14 |
| arigo | in rangeobject.c? to compute the length, e.g. to support | 10:14 |
| arigo | xrange(-sys.maxint, sys.maxint, 2) | 10:14 |
| stakkars | ok, I see.So I have no point, our stuff *is* compatible. | 10:15 |
| stakkars | although the impl stoing len has some attraction. | 10:15 |
| arigo | keep in mind that these rranges are meant to vanish away | 10:16 |
| arigo | the current impl has the advantage that it looks very much like a C for loop: there is a single index field that gets incremented | 10:17 |
| stakkars | vanishing means they get inlined, unless they are exposed. | 10:17 |
| arigo | yes, the whole 'for x in range()' graphs and calls becomes like a normal C for loop | 10:18 |
| stakkars | yes, I noticed. It implies the up/down iterators. a generic up/down would do better with an extra counter. | 10:18 |
| arigo | the motivation was that doing divisions and multiplications is not common in C for loops | 10:18 |
| stakkars | so you hope to support C optimizations better. IC | 10:19 |
| stakkars | anyway, I think I fail to check for range overflow. Need to add that somewhere | 10:20 |
| arigo | no, that probably doesn't belong to RPython | 10:21 |
| stakkars | or maybe not? unnecesary slowdown to have ot computelength al the time? | 10:21 |
| arigo | drawing the line is always some kind of funny exercice, but let's not forget that at Paris already we should start thinking seriously about JIT | 10:22 |
| stakkars | sure. not forgetting that we need to speed up this baby a little, to run a JIT on top of it :-) | 10:24 |
| fijal | I've got a question: suppose I want to add some python-checking routines (some static analyzis like pylint), shall I put it as different translator? | 10:27 |
| arigo | well, it depends what more precisely you are trying to do | 10:28 |
| stakkars | if you think of an RPython checker, I'd put it as different backend. | 10:29 |
| stakkars | for generic Python, we don't have the tools. | 10:29 |
| fijal | No, I think about app-level checker to warn user about several caveats that he have in his code | 10:29 |
| fijal | taken from flow-graph analyzis mainly | 10:30 |
| stakkars | you cannot flow-graph analyze aplevel code. | 10:30 |
| arigo | this is work :-) you need to enhance the flow object space to handle any app-level construction | 10:30 |
| arigo | currently it will not be happy on some stuff like generators, def or class statements, etc. | 10:30 |
| arigo | then you can probably try running the annotator on it, and see where it explodes | 10:31 |
| stakkars | and add extra options to remove current assumptions like globals == const, etc. | 10:31 |
| arigo | but this is likely to be more confusing than useful at the moment | 10:31 |
| arigo | stakkars: indeed | 10:31 |
| arigo | it is the kind of analysis that we'll want to do later, but our toolchain is not oriented towards this right now | 10:31 |
| fijal | so what's the target of analyzis? | 10:32 |
| fijal | are You going to make such things in the future? | 10:32 |
| arigo | the current target is RPython code, i.e. "Python code with a finite amount dynamism" | 10:33 |
| Action: stakkars would like to point you at the PyPy docs | 10:33 | |
| arigo | how exactly we will support analysis of app-level code is yet open | 10:34 |
| stakkars | arigo: you reminded me of JIT. This means I should not worry too much about static optimization. | 10:36 |
| arigo | yes, at least for features that we don't use in PyPy anyway | 10:37 |
| stakkars | How far is this true? DO you think we can gain speed using JIT technology, if we are unable to do it statically? | 10:37 |
| arigo | like catching OverflowError upon calling xrange()... | 10:37 |
| stakkars | ok, that's only interesting for people writing RPython code (which will probably come) | 10:38 |
| arigo | the JIT means that we will need something that supports not just RPython but the full Python, and we definitely don't want to rewrite everything in r*.py, so we need to focus on how to use the existing stdobjspace instead for that | 10:38 |
| arigo | yes, RPython is nice to have, but should be kept basic enough (that's its raison d'etre) | 10:39 |
| stakkars | it's even a tad that we had to make the compiler RPython. A good JIT would save most of these I guess. | 10:41 |
| arigo | yes | 10:41 |
| arigo | but the plan is to use the translator to produce the JIT, so we need the translator... | 10:42 |
| mwh | i take it the jit will work in a similar way to pypy | 10:43 |
| stakkars | (and we probably thought to work on it much earlier, but things happen) | 10:43 |
| mwh | suspending execution & so on | 10:43 |
| stakkars | which translator did you mean, actually? | 10:43 |
| arigo | stakkars: well the whole objspace+annotation+rpython | 10:43 |
| stakkars | yeah, ok | 10:44 |
| arigo | mwh: in a similar way to Psyco, which is itself similar to PyPy's flow objspace | 10:44 |
| mwh | arigo: what a coincidence! | 10:44 |
| stakkars | so we actually *have* to re-think flow objspace (and maybe write a different one) | 10:44 |
| arigo | indeed | 10:44 |
| arigo | I wrote some introductory stuff yesterday night | 10:45 |
| stakkars | I thought you rewrote the whole dict world! When do you sleep, actually? | 10:45 |
| arigo | :-) | 10:45 |
| arigo | with hindsight, the flow objspace's role can be made clearer | 10:45 |
| arigo | currently, we import our program in CPython and then run the flow objspace on it, using introspection | 10:46 |
| arigo | but another point of view is possible | 10:46 |
| arigo | we could import the program to analyse in PyPy, in a std or trivial object space | 10:47 |
| stakkars | :-X | 10:47 |
| arigo | and then switch between this objspace and the flow objspace to switch between actual and abstract execution | 10:47 |
| arigo | this has the advantage of less cross-level confusion | 10:47 |
| stakkars | having it both real and abstract. Operating in parallel,or exchanging the object space in a way? | 10:48 |
| arigo | depends | 10:48 |
| arigo | at first, exchanging the object space | 10:48 |
| arigo | (which is not unrelated to the flow object space that can currently switch to "concrete mode", btw) | 10:48 |
| arigo | for Psyco, I guess it would operate with both spaces in parallel | 10:49 |
| arigo | so that the flow space can ask the concrete space for more information, e.g. the type of an object | 10:49 |
| stakkars | for full Python, it sounds a little that we have to extend std objspace to grow an "abstract mode" :-) | 10:50 |
| arigo | yes, but not necessarily by programming this features into the std objspace | 10:51 |
| arigo | but with a proxy objspace on top of it | 10:51 |
| stakkars | (not serious, but flow space lacks a lot of features) -- yes | 10:51 |
| stakkars | and the proxy would operate them in parallel, and you can ask it to give you specific info (my picture) | 10:52 |
| arigo | yes | 10:52 |
| xorAxAx | has anybody looked at http://shedskin.sourceforge.net/? /me takes a look | 10:54 |
| stakkars | hum, your remarks mademe explicitly not look into it :-] | 10:54 |
| xorAxAx | why? | 10:55 |
| xorAxAx | because he didnt contact the pypy team? :-) | 10:55 |
| xorAxAx | hmm, its mit-licensed | 10:56 |
| xorAxAx | "Thanks a lot to Google, for sponsoring this project via their SoC program. This has really been a major motivation to me, and enabled me to work on the project longer than would have been possible otherwise. " | 10:56 |
| xorAxAx | ah! | 10:56 |
| stakkars | no I misunderstood your thought. Will read it. | 10:56 |
| arigo | no, we know about his work | 10:57 |
| xorAxAx | arigo: thats good | 10:58 |
| xorAxAx | 30% unit tests, wow | 10:59 |
| arigo | well, what I drafted yesterday also gives a reason that we don't follow this project very closely | 11:00 |
| stakkars | (tried to read shedskin's doco, but can't find any) -- is your draft available, somewhere? Not in doc, sofar | 11:01 |
| arigo | http://codespeak.net/pypy/dist/pypy/doc/draft-dynamic-language-translation.txt | 11:02 |
| stakkars | :) ok I should svn up even more often :) | 11:03 |
| arigo | hum, the .html version doesn't show up on codespeak. I wonder if the update-on-checkin scripts are activated. | 11:04 |
| stakkars | are they cron-driver, or triggered by svn? | 11:05 |
| stakkars | driven | 11:05 |
| arigo | should be triggered by svn | 11:05 |
| stakkars | yes,your text is there but no html | 11:06 |
| xorAxAx | kinda monolithic, the ss.py (211kb). reminds me at pyc, the greek compiler :) | 11:09 |
| arigo | ouch | 11:12 |
| stakkars | for some reason, the draft fails on codespeak. Not on my machine. | 11:15 |
| stakkars | ah, ok, I don't have permission. | 11:16 |
| mwh | arigo: that's not the most comprehensible document i've ever read | 11:20 |
| mwh | some long sentences :) | 11:21 |
| pedronis (i=pedronis@ratthing-b246.strakt.com) joined #pypy. | 11:21 | |
| Rhamphoryncus (n=rhamph@unaffiliated/rhamphoryncus) left irc: "Support ISO 8601! http://www.cl.cam.ac.uk/~mgk25/iso-time.html" | 11:25 | |
| pedronis | hi | 11:26 |
| stakkars | hi | 11:29 |
| magent1 (n=dbernhar@tiramisu.imag.fr) joined #pypy. | 11:30 | |
| magent1 (n=dbernhar@tiramisu.imag.fr) left #pypy. | 11:30 | |
| aleale (n=andersle@clogs.dfki.uni-sb.de) joined #pypy. | 11:35 | |
| arigo (n=arigo@134.99.112.244) left irc: Read error: 110 (Connection timed out) | 11:36 | |
| stakkars | well, and lots of note here and there, content ok, some things would be clearer if shorter. I understand it because I know a little how the author thinks. | 11:37 |
| stakkars | the idea is just great | 11:37 |
| mwh | oh yes | 11:40 |
| mwh | not meant to be an extreme criticism | 11:40 |
| stakkars | neither here, it is just in fact not too easy to read. | 11:41 |
| arigo (n=arigo@134.99.112.244) joined #pypy. | 12:04 | |
| arigo | where are the IRC logs now? | 12:21 |
| stakkars | as before, plus on tismerysoft.de/pypy/irc-logs | 12:21 |
| stakkars | I'm logging pypy-sync there as well. Anything else, maybe? pypy-tb? | 12:22 |
| arigo | thanks | 12:22 |
| arigo | pypy-tb, maybe | 12:23 |
| stakkars | took a little more time than planned, because I had to circumvent blockade of port 6667 at my provider. | 12:23 |
| arigo | missed the discussion on the document -- hum, yes I shouldn't write too late at night | 12:24 |
| stakkars | it's fine, but a bit "strubbelig" :-) | 12:24 |
| arigo | well I feel it's too lengthy already, not sure if it should be split up to a separate introduction document maybe | 12:25 |
| stakkars | the intro goes a bit much into details of pypy, I'd split this apart a little | 12:26 |
| stakkars | I've just skimmed the 5737 lines of ss.py | 12:27 |
| stakkars | seems to be what you want if you think C++ | 12:28 |
| stakkars | which is exactly what we don't want :-) | 12:28 |
| arigo | :-) | 12:32 |
| stakkars | can be a static translation tool for static source code as it looks, and appears to be absolutely unflexible. | 12:33 |
| stakkars | low abstraction, hardly commented set of functions, well 'nuf said. I still can't figure out if something would help us. | 12:34 |
| pedronis | I tried on richards.py | 12:42 |
| pedronis | I tried it | 12:42 |
| stakkars | did it grok it? | 12:43 |
| Nick change: arigo -> arigo|lunch | 12:44 | |
| stakkars | ( tried the ss.py, or are you talking of the new dict implementation?) | 12:44 |
| pedronis | the ss.py | 12:45 |
| xorAxAx | and? | 12:45 |
| stakkars | I wondered about the mentors: Jeremy Hylton, Brett Cannon | 12:45 |
| xorAxAx | it looks like google might use it for some of their works | 12:46 |
| pedronis | it was not happy about is is not and /= | 12:46 |
| pedronis | changed those | 12:46 |
| pedronis | then it was unhappy about subclassing from object | 12:46 |
| pedronis | removed that too | 12:46 |
| stakkars | removed that from richards? | 12:47 |
| pedronis | yes | 12:47 |
| pedronis | then it was unhappy about assigment in a class suite | 12:47 |
| pedronis | which would mean a class attribute which is understandable | 12:47 |
| pedronis | but it crashed not nicely | 12:47 |
| pedronis | after all these changes | 12:47 |
| stakkars | and after richard was empty, it compiled,mission accomplished :-) | 12:47 |
| pedronis | it runs a bit | 12:47 |
| pedronis | but emit warnings about argument numbers | 12:48 |
| pedronis | for calling unbound __init__ | 12:48 |
| pedronis | and then crashes on a KeyError | 12:48 |
| xorAxAx | how about pystones? | 12:48 |
| pedronis | something related to entry_point | 12:48 |
| pedronis | and at that point I gave up | 12:48 |
| stakkars | I would have expected more influence from the mentors on concept and layout | 12:49 |
| pedronis | it does some interesting analysises | 12:49 |
| xorAxAx | stakkars: :-) | 12:49 |
| pedronis | but is far from robust | 12:49 |
| pedronis | so 0.0.1 is indeed the right numbering | 12:49 |
| stakkars | yes, at some point there is interesting stuff. It isjust such a huge mixture of genc and analyse and whatsoever :( | 12:50 |
| stakkars | and no tables, everything hacked right in (sorry, should be more polite) | 12:51 |
| pedronis | well, the author said that has not been tryed beyond the set of around~200 lines programs in the unit tests | 12:51 |
| stakkars | it would be interesting to know if it would run a bit of RPython code. rpystone? | 12:52 |
| pedronis | maybe rpystone is easier for it | 12:52 |
| pedronis | but run the unit tests before trying | 12:53 |
| stakkars | extremely simple code with no classish or dynamic features at all | 12:53 |
| pedronis | because I had | 12:53 |
| pedronis | to tweak ss itself | 12:53 |
| stakkars | and they give problems, too? | 12:53 |
| pedronis | to have compilation working | 12:53 |
| pedronis | for my setup | 12:53 |
| pedronis | and notice that you need boehm, it uses it | 12:54 |
| stakkars | ah. No I guess this is enough waste of time, already, unless you think we should learn from it. | 12:54 |
| stakkars | boehm is no problem, I'm using it on windows with PyPy all the time, just because it has a very fast allocator. | 12:54 |
| pedronis | I think there's no support for compiling on window | 12:56 |
| pedronis | but you can probably stop it after the code is generated | 12:56 |
| stakkars | there is windows mentioned, but I think it uses cygwin, there. | 12:56 |
| pedronis | it writes a Makefile for g++ | 12:56 |
| _hannes (i=ore@i577B4751.versanet.de) joined #pypy. | 12:56 | |
| stakkars | I'm actually not interested in something that writes C++, but I'd like to see if it has advantages in type inference? | 12:58 |
| stakkars | IOW reading the theory would help more than the source | 12:59 |
| pedronis | if you want the theory I think the best thing is still David Grove thesis | 13:00 |
| pedronis | schedskin is based on refinement of that approaches | 13:00 |
| stakkars | Ah. I don't see this mentioned, but Plevyak (which I didn't read either, maybe it is based on Grove) | 13:01 |
| stakkars | do you know what Tiejun & Wang is about and why it is considered incomprehensible? | 13:02 |
| pedronis | are you readin is optimising python? | 13:03 |
| stakkars | pedronis: btw. thanks for the missing bit with rrange, although I should have know. is complete now. | 13:04 |
| __alex (n=alex@141.22.64.4) joined #pypy. | 13:04 | |
| stakkars | I was reading http://wiki.python.org/moin/SummerOfCode | 13:04 |
| stakkars | Efficiently Analysing Data Polymorphism and Deducing Generics in Shedskin | 13:04 |
| pedronis | Plevyak is cited in Grove | 13:10 |
| pedronis | I see, Tiejun is more recent | 13:13 |
| stakkars | A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy 8-) | 13:13 |
| pedronis | http://www.cs.jhu.edu/~scott/pll/constraints.html | 13:14 |
| -NickServ (NickServ@services.)- This nickname is owned by someone else | 13:25 | |
| -NickServ (NickServ@services.)- If this is your nickname, type /msg NickServ IDENTIFY <password> | 13:25 | |
| pybot joined #pypy. | 13:25 | |
| -NickServ (NickServ@services.)- This nickname is owned by someone else | 13:26 | |
| -NickServ (NickServ@services.)- If this is your nickname, type /msg NickServ IDENTIFY <password> | 13:26 | |
| pybot joined #pypy. | 13:26 | |
| stakkars | added #pypy-tb to eggdrop logging | 13:26 |
| ericvrp (n=chatzill@ericvrp.demon.nl) joined #pypy. | 13:26 | |
| stakkars | wah, horrible inline braces and symbols all over the text :-) a least around page 18 | 13:32 |
| xorAxAx | java? :-) | 13:33 |
| stakkars | sets, lattices, lemmata, theorems and corrolaries in a paper about '''Polymorphic Constraint-Based Type Inference for Objects''', written in math style. | 13:35 |
| stakkars | no, java is worse :) | 13:36 |
| fijal (i=root@zodiac.mimuw.edu.pl) left irc: Remote closed the connection | 13:42 | |
| xorAxAx | ah, those huge braces | 13:45 |
| stakkars | pedronis: is PyPy translatable, again? I am compiling, and I wonder about structure names which seem to be multiple PAGES long? :-) | 14:19 |
| stakkars | I guess it will break, soon | 14:19 |
| pedronis | it's the Repr passed along to dict methods that generate that names | 14:20 |
| pedronis | I think | 14:20 |
| stakkars | I guess VC will spit names with 10000 chars at me :) | 14:20 |
| stakkars | yes, it is dictionary type names | 14:22 |
| stakkars | might use an md5 digest, instead :) | 14:23 |
| pedronis | the fact is that repr of the Repr is used and this one contains the lltype involved | 14:23 |
| stakkars | I hope they are unique quite early,since most probably soe truncation will occour. | 14:25 |
| pedronis | yes, but it simply a bad approach to have the lltype repr in there | 14:26 |
| pedronis | I know the code involved | 14:26 |
| pedronis | I just need to think exactly what to change | 14:26 |
| stakkars | I don't care, as log the compiler doesn't as well. | 14:26 |
| pedronis | no, but it's annoying in the graphs too | 14:27 |
| pedronis | because you get huge nodes to contain the names | 14:27 |
| stakkars | really funny to look at. There is notmuch missing, and you will read your whole program in the names. | 14:27 |
| pedronis | the fault is the W_Root lltype with all the methods | 14:29 |
| stakkars | related to the change of having no need for position_key? | 14:30 |
| pedronis | no | 14:31 |
| pedronis | related to hlinvoke | 14:32 |
| pedronis | taking a repr | 14:32 |
| stakkars | really no joke,asa veryeasy hack: whynoch change the repr to its hexdigests, if it is longer than, say, 100 chars? | 14:34 |
| stakkars | sorry about my keyboard. as a very easy hack, one could use the md5 digest for long names. | 14:34 |
| pedronis | the name show up in the graphs | 14:35 |
| pedronis | and are useful for debugging | 14:35 |
| stakkars | not useful if you need to scroll several pages to read the name. I thought to stop afer some 100 and add the digest, then. | 14:36 |
| pedronis | no, they to messy anyway and too long already at that point | 14:38 |
| pedronis | I'm looking at fixing this | 14:38 |
| stakkars | pooooooooof | 14:40 |
| stakkars | testing_1.c(17680) : fatal error C1064: Compilerlimit: Überlauf des internen Puffers durch Token | 14:40 |
| stakkars | it's probably possible to implement a trojan into VC++ by using long tokens :) | 14:41 |
| stakkars | btw., it would probably be a nice feature instead of encoding reprs in loong names, to have a way to browse the repr structure in PyGames. Or do we have that? | 14:43 |
| Nick change: tic|away -> tic | 14:47 | |
| stakkars | the line where VC crashes has 22322 columns. I think we should not be angry with it. | 14:49 |
| arigo|lunch | stakkars: no, we don't have that. It would be useful indeed | 14:52 |
| arigo|lunch | note that using a hash for long names is useless; the names don't have to be unique at this point | 14:52 |
| arigo|lunch | so we could just truncate or whatever gives the most readable output | 14:53 |
| arigo|lunch | genc will ensure the function names become unique in the .c file | 14:53 |
| stakkars | the names for the C source have to be unique, methinks. Just less kilobytes | 14:53 |
| arigo|lunch | it's genc's job to make the names unique | 14:54 |
| stakkars | yes, I forgot. (don't forget to unlunch) | 14:54 |
| arigo|lunch | but it's true that the NameManager could make sure it doesn't return too long names | 14:54 |
| Nick change: arigo|lunch -> arigo | 14:54 | |
| arigo | it's probably a good step towards making sure we generate officially valid C code... | 14:55 |
| stakkars | it's not too bad to have names that describe the contents. But I'd be hapy with something "I_m_the_repr_for_nice_dicts_of_strings" or somesuch. | 14:56 |
| stakkars | but sure, this doesn't help, sice we have a lot of diversification. | 14:57 |
| stakkars | we are also not using names for substructures, but always tell the contents. Would probably help. | 14:59 |
| Action: stakkars is just renaming quickly in the C source, using Python, just to see if we can compile | 15:01 | |
| pedronis | I'm about to checkin something that makes the name shorter | 15:03 |
| arigo | pedronis: then you just beat me :-) | 15:05 |
| stakkars | and me | 15:06 |
| arigo | ah no, what I'd about to check in truncates the names to 50 chars plus prefix plus numeric suffix | 15:06 |
| pedronis | no, I changed the __repr__ of Repr | 15:06 |
| pedronis | different levels | 15:06 |
| arigo | ok, it's all worthwhile I guess | 15:06 |
| stakkars | would work, but doing it structurally more level-wise would be even better | 15:06 |
| arigo | ok, it's all worthwhile I guess | 15:14 |
| arigo | sorry, wrong window | 15:14 |
| mwh | arigo: damn that up-return finger macro! | 15:15 |
| arigo | :-) | 15:15 |
| xorAxAx | yeah, the irc client should ask "do you really want to press cursor-up?" | 15:20 |
| stakkars | 50 chars is quite rigid. but thanks :) | 15:22 |
| arigo | oooops I think I broke tests by inserting this [:50] | 15:40 |
| xorAxAx | it looks like a weird smiley | 15:40 |
| arigo | [:-O] | 15:40 |
| stakkars | 10I'm looking for an even weirder smiley, to01 comment on my latest test result 8-[] | 15:47 |
| stakkars | comparing a pypy built on Sept. 04, using boehm and full optimization, versus today's version. | 15:48 |
| stakkars | richard's, 1 run, before = 37373 ms after = 54277 ms | 15:49 |
| stakkars | we are doing *something* into the wrong direction :-)) | 15:49 |
| pedronis | notice that now you get thread enabled with richards too | 15:50 |
| pedronis | I mean with boehm too | 15:50 |
| stakkars | argll, yes, this is probably veery expensive, as it is done every opcode! | 15:51 |
| pedronis | when they are one, they add a 50% overhead | 15:51 |
| pedronis | are on | 15:51 |
| stakkars | ok, *then* we have a clear advantage, reinterpreting the result. | 15:51 |
| pedronis | no, it's not done on every opcode | 15:52 |
| pedronis | but yes we still get a huge overhead | 15:52 |
| stakkars | I think it was at some time. | 15:52 |
| pedronis | that at some point we need to understand | 15:52 |
| pedronis | you probably want to disable them to compare | 15:53 |
| stakkars | is there a simple waytoswitch threads off in the generated source? I don't want tore-generate, exactly because things seem to be broken, again. | 15:53 |
| arigo | stakkars: no, and probably no more, respectively | 15:53 |
| arigo | I'm about to check in a simpler malloc-removal algorithm, but I'll try translate_pypy before | 15:54 |
| stakkars | I have no problem with boehm, just with threads. | 15:54 |
| arigo | I know, I really meant "no you can't switch it off easily in the .c source" | 15:55 |
| stakkars | and it would be nice to use what boehm does,to getcomparable results. | 15:55 |
| arigo | stakkars: I don't get that point | 15:56 |
| pedronis | I think it refers not asking for memory from the OS in small chunks | 15:56 |
| stakkars | ATM, using boehm accelerates pypy by a factor of 2-4, just by the fact that it optimizes malloc in a way. | 15:57 |
| arigo | ah, then we should try dropping PyMalloc in here | 15:57 |
| stakkars | to get comparable results, I'd like to have the same strategy by default, whether using boehm or not. | 15:57 |
| stakkars | or we compare apples with peaches | 15:57 |
| stakkars | it would be nice if boehm had a switch that switches it off, but keeps its malloc strategy | 15:58 |
| stakkars | PyMalloc is probably vry different. | 15:59 |
| xorAxAx | stakkars: have you tried this malloc chunk size global var? | 15:59 |
| stakkars | no, forgot about that | 16:00 |
| stakkars | xorAxAx: is this available on more than this platform? | 16:00 |
| xorAxAx | stakkars: it is malloc-specific. the one i talked about was windows-specific | 16:02 |
| stakkars | I'd like to have something common for all platforms, allocating in a way that the OS influence is neglectible | 16:02 |
| xorAxAx | there are malloc replacements (if you want to use ref counting) | 16:03 |
| stakkars | but if we use PyMalloc, how do I replace things for boehm. | 16:03 |
| xorAxAx | what do you mean by replace? | 16:03 |
| xorAxAx | you mean the calls? | 16:03 |
| stakkars | boehm already does something to make the OS overhead neglectible. But I don't want to stack malloc strategies. | 16:04 |
| stakkars | or even mixing them is a bad idea. | 16:04 |
| xorAxAx | the overhead consists of allocating heap chunks | 16:04 |
| xorAxAx | and it is high if you do that often | 16:04 |
| xorAxAx | boehm doesnt do it often | 16:04 |
| xorAxAx | normal malloc does on windows because the chunks are small | 16:04 |
| xorAxAx | so you need an allocator with large chunks which might be optimised for small objects if you dont trust the OS malloc | 16:05 |
| stakkars | sure, that's not the problem. The problem is that with boehm, I buy one strategy, and with PyMalloc another one. Both have the side effect of reducing OS overhead, but the resultd are still not comparable. | 16:06 |
| xorAxAx | why do you want comparable results in this case? | 16:09 |
| pedronis | well, you could try to use GC_MALLOC_ATOMIC and GC_FREE | 16:09 |
| pedronis | with reference counting | 16:10 |
| stakkars | thesemethods are borrowedfrom boehm, then? | 16:11 |
| pedronis | or GC_MALLOCA_UNCOLLETABLE | 16:11 |
| stakkars | sounds good. | 16:11 |
| pedronis | the problem is that there is not public interface meaning both ATOMIC+UNCOLLETABLE | 16:11 |
| stakkars | ATM I have the problem how to switch threads off. There is no entry in translate_pypy --help | 16:12 |
| pedronis | no | 16:12 |
| xorAxAx | --damn-dont-use-modules:thread :-) | 16:12 |
| stakkars | I like that, please add :) | 16:13 |
| stakkars | - sigh - | 16:13 |
| stakkars | do we use the new translate_pypy,already, or do I patch the old one? | 16:16 |
| arigo | stakkars: target-specific options are messy anyway | 16:17 |
| arigo | for now you just change a line in targetpypystandalone | 16:17 |
| pedronis | yes, one of the things that the refactoring should fix | 16:17 |
| stakkars | we don't want the option, and you don't care if I disable it? | 16:17 |
| stakkars | makes sense ATM since we know it works and that it eats lots of time for a reason to investigate not now | 16:18 |
| arigo | yes (though you can also not check it in) | 16:18 |
| stakkars | (I really hate the latter because at some time I *will* check it in if I'm not careful) aye aye | 16:19 |
| arigo | stakkars: then copy it to targetpypystandalonewithoutthreads.py :-) | 16:19 |
| stakkars | ROTFLMAO | 16:20 |
| arigo | note that the rdict change was an un-optimization for string-keyed dicts | 16:20 |
| arigo | because they now carry two extra flags in each entry, just like every dict | 16:21 |
| stakkars | yes,but I guess not an expensiveone. I believed the faster dict access in richards would outperform this | 16:21 |
| stakkars | this is of course expensive and woul be even more if we use refcounting. That's why I always use boehm:object access is for free. | 16:22 |
| pedronis | arigo: what kind of simpler malloc removal are you working on? | 16:23 |
| arigo | http://codespeak.net/svn/user/arigo/hack/pypy-hack/remove_malloc2.py | 16:24 |
| arigo | (which I am moving to backendoptimization.py) | 16:24 |
| stakkars | ok, compiling with targetpypystandalonewithoutthreads.py, boehm, takes 40 minutes | 16:25 |
| arigo | for a graph, it divides the vars in families, where two vars are in the same family if a value can go from one to the other; | 16:25 |
| arigo | the family records all usages of its variables | 16:25 |
| arigo | then we just use this information to see which families correspond to malloc/getfield/setfield only | 16:26 |
| arigo | it's < 200 lines of code, as opposed to 500 for the previous attempt | 16:26 |
| arigo | but it must be re-run iteratively, as inlining structures may expose more opportunity for inlining | 16:26 |
| pedronis | I see, yes is similar to what I had in mind | 16:29 |
| pedronis | the lifetime part can be reused to properly detect | 16:29 |
| pedronis | what exception a graph can raise | 16:29 |
| arigo | I guess so | 16:30 |
| pedronis | of course it needs to be comibined with helpers inlining to get much effect | 16:31 |
| arigo | yes | 16:31 |
| arigo | I ran the tests, it only applied once | 16:32 |
| pedronis | not too strange | 16:32 |
| stakkars | breaks structures into pieces and tries to allocate on stack? | 16:34 |
| arigo | just breaks structures into variables | 16:34 |
| stakkars | so if I have a list, the length becomes a local variable, and the data is still a malloced array? | 16:35 |
| arigo | hum, I'm confused | 16:36 |
| arigo | there are more snippets where it should succeed | 16:36 |
| pedronis | probably some shallow problem? | 16:37 |
| stakkars | if that array can be made local, that's it. | 16:37 |
| arigo | pedronis: indeed, test_backendoptimized doesn't run all backend optimizations | 16:38 |
| pedronis | it list them esplicitly | 16:40 |
| arigo | that's two more tests where it has an effect :-) | 16:40 |
| arigo | stakkars: this list optimization is still not quite achievable because most list operations call a helper | 16:40 |
| arigo | that's why inlining is important: to help this malloc removal process | 16:41 |
| stakkars | yes, it would also imply inlining of those I think | 16:41 |
| arigo | no, you need to inline the helpers before you can detect that you can remove the mallocs | 16:42 |
| arigo | unless we go for some more advanced algorithms (i.e. > 200 lines :-) | 16:42 |
| stakkars | this is what I thought. Did I write differently? | 16:42 |
| pedronis | well, I think it is worth to see how fat the simple approaches bring us | 16:42 |
| pedronis | before trying more complicated things | 16:43 |
| arigo | stakkars: what did you mean? | 16:43 |
| stakkars | whyt you said. you first inline the helpers, then the listis no longer exposed, then you figure that you can melt the structure away, and so on. | 16:44 |
| arigo | ok | 16:44 |
| pedronis | yes, that's the idea | 16:44 |
| pedronis | and iterators are probably the most interesting thing | 16:45 |
| pedronis | before lists | 16:45 |
| stakkars | and you do that until you get to some very huge limits, and then you try todo the opposite :-) | 16:45 |
| pedronis | I don't think we have in mind to indiscriminately inline all helpers | 16:45 |
| stakkars | you think iterators (heh, I'm joking) would turn themselves into the nice, efficient loops arigo mentioned, without extra magic but inlining? would be great. | 16:46 |
| pedronis | weel, we need removal of unused malloc results | 16:48 |
| pedronis | because as Carl noticed the inlining still leaves back the exception construction | 16:49 |
| arigo | should melt away already with the remove_simple_malloc() | 16:50 |
| stakkars | the exception handler for the malloc? Maybe this relationship could be recorded when we create the malloc | 16:50 |
| stakkars | ah, the exception handler around no code would just go away. | 16:50 |
| arigo | wow, the results of the inlining are impressive | 16:51 |
| arigo | these mallocs will go away, no problem -- we're almost there :-) | 16:51 |
| stakkars | $-D $-D $-D | 16:51 |
| pedronis | arigo: you are looking at Carl tests | 16:52 |
| arigo | yes | 16:52 |
| arigo | if we add t.backend_optimizations() in test_inline_exceptions | 16:52 |
| arigo | then f() melts away completely | 16:52 |
| arigo | I mean g(), even | 16:53 |
| arigo | it leaves a couple of empty blocks behind, and only two int_eq() operations | 16:53 |
| Action: stakkars is jumping on his chair like a child at Xmas evening | 16:54 | |
| sabi | heh | 16:54 |
| arigo | .-) | 16:54 |
| arigo | hum, test_inline_var_exception() still needs some work | 16:55 |
| arigo | not sure that we care at the moment, though | 16:55 |
| pedronis | notice that tests | 16:55 |
| pedronis | oops | 16:55 |
| pedronis | notice that that tests uncovered | 16:55 |
| pedronis | an annotation problem | 16:56 |
| arigo | :-) | 16:56 |
| pedronis | is really a bit atypical | 16:56 |
| arigo | ok | 16:56 |
| pedronis | the change in the ordering of issubtype | 16:56 |
| pedronis | was because of it | 16:56 |
| pedronis | I also think that we don't do the correct thing | 16:56 |
| pedronis | with issubtype and isinstance | 16:57 |
| pedronis | about considering old style classes | 16:57 |
| pedronis | new style | 16:57 |
| pedronis | from the annotator point of view | 16:57 |
| pedronis | although the test is not affected because it deals with instances | 16:57 |
| pedronis | and not classes | 16:57 |
| arigo | I see | 16:58 |
| arigo | for now I'd like to see if we can inline some helpers automatically | 16:58 |
| arigo | and reach this for loop simplification | 16:58 |
| arigo | basically, we should look at all call sites and decide if it should be inlined | 16:58 |
| arigo | we need to figure out a not too unreasonable condition | 16:59 |
| pedronis | yes | 16:59 |
| pedronis | it would be good to have a count of ops | 16:59 |
| pedronis | in the graphs | 16:59 |
| arigo | well it's easy to count | 16:59 |
| pedronis | the other thing if whether it loops | 16:59 |
| arigo | and it would be messy to maintain, given how we change the graphs in-place | 16:59 |
| arigo | intuitively, not inlining functions that loop simplifies things, is it a good criteria? | 17:00 |
| xorAxAx | i think it solely depends on the block size | 17:01 |
| xorAxAx | is loop setup expensive in pypy? | 17:01 |
| pedronis | no but the problem with loops | 17:02 |
| pedronis | is the amount of code size | 17:02 |
| xorAxAx | small loops exist :) | 17:02 |
| pedronis | and cache misses | 17:02 |
| pedronis | vs non gaining much | 17:02 |
| pedronis | because the call is much faster than the looping | 17:02 |
| arigo | I see | 17:02 |
| xorAxAx | depends on the no. of passes | 17:02 |
| pedronis | but it's all a trade-offs | 17:03 |
| xorAxAx | i would look at the loop setup code and see if it trashes cache lines or is longer than a call | 17:03 |
| arigo | we could count some wild estimates of how many operations are likely to be executed in the called functions | 17:03 |
| pedronis | yes | 17:04 |
| arigo | by doing things like "here's a branch, let's say it's a 50%-50% chance to follow each path" | 17:04 |
| arigo | a bit of linear algebra if there are loops... | 17:04 |
| arigo | then we add a constant for the call overhead | 17:04 |
| arigo | divide by the total number of operations for the size overhead | 17:05 |
| arigo | something arbitrary like this | 17:05 |
| stakkars | new results on the richards front: still quite messy. no threads, boehm, all options identical to before one week. | 17:05 |
| stakkars | before: 36822 ms after: 46717 ms | 17:05 |
| arigo | I'm still wondering how much removing of the string dict special case hurts | 17:06 |
| pedronis | well, considering that involves in all ops dispatch | 17:06 |
| stakkars | we are probably using this in the interpreter much more heavily than I expected. | 17:06 |
| pedronis | is involved | 17:06 |
| arigo | still, the difference is big; maybe there is a lot to gain by tuning this carefully (later) | 17:07 |
| pedronis | did we change something else? | 17:08 |
| arigo | I'm thinking about specializations (later again) for cases where we know two special values for the keys (for pristine and deleted) | 17:08 |
| arigo | good question | 17:09 |
| arigo | we should compare around the change that removed strdicts | 17:09 |
| stakkars | what do we use string keys for. in opcodes??? | 17:09 |
| arigo | stakkars: to look up special method names, all the time | 17:09 |
| pedronis | for the dict_w of types | 17:09 |
| pedronis | that is accessed all the time | 17:09 |
| pedronis | in lookups | 17:09 |
| arigo | arf | 17:11 |
| stakkars | types of applevel objects | 17:11 |
| arigo | look at translator.backendoptimization.collect_called_functions() attentively | 17:12 |
| pedronis | stakkars: yes types of applevel objects, int str etc | 17:13 |
| stakkars | I wasn't aware that we could deduce those to be string only dicts, already. | 17:14 |
| arigo | but we only supported string only dicts, not so long ago | 17:16 |
| pedronis | arigo: it returns nothing | 17:16 |
| arigo | pedronis: indeed | 17:16 |
| stakkars | arigo: yes, it's a nice function. Could be a bit more useful if it would use its visit :-) | 17:16 |
| arigo | so there's a small problem | 17:16 |
| arigo | the next() of iterators do a "raise StopIteration" | 17:16 |
| arigo | which is equivalent to "raise StopIteration()" | 17:16 |
| arigo | i.e. they contain a direct_call | 17:16 |
| arigo | so right now they can't be inlined | 17:16 |
| arigo | ? sorry, I am talking nonsense | 17:17 |
| Action: arigo confused by a failing test | 17:17 | |
| pedronis | it seems that carl added some exception support | 17:19 |
| pedronis | but did not remove the assert | 17:20 |
| stakkars | anyway, collect_called_functions is a complete no-op | 17:20 |
| arigo | I think the assert really means "there is no direct_call in the graph to inline" | 17:21 |
| arigo | I'll have to try to see if this is now supported | 17:21 |
| arigo | there is no test, so I'm not sure | 17:21 |
| pedronis | I see | 17:23 |
| pedronis | probably | 17:23 |
| pedronis | not | 17:23 |
| pedronis | because you need recursive | 17:23 |
| pedronis | info about | 17:23 |
| pedronis | what the function can raise | 17:23 |
| Action: pedronis needs to logout, my window manager got slightly insane | 17:25 | |
| pedronis (i=pedronis@ratthing-b246.strakt.com) left irc: "Chatzilla 0.9.67 [Firefox 1.0.2/20050325]" | 17:25 | |
| stakkars | well, starting with functions that don't call any other functions, inlining always just those,you have a fixed set of exceptions that propagates up. | 17:25 |
| arigo | stakkars: yes, that's part of the non-sense I realized | 17:26 |
| arigo | even if a small helper calls another small helper, we can inline in two steps | 17:26 |
| pedronis (i=pedronis@ratthing-b246.strakt.com) joined #pypy. | 17:26 | |
| stakkars | what's non-sense? | 17:27 |
| arigo | what I said above the line "? sorry, I am talking nonsense" | 17:30 |
| stakkars | ah, and I thought the approach to start inlining with the bottom level funcs (leafs) was non-sense. | 17:31 |
| pedronis | yes, the issue is inlining something small that calls something big | 17:31 |
| arigo | right | 17:31 |
| arigo | ...while in addition catching an exception explicitely on the outermost call | 17:31 |
| pedronis | because the bottom up approach doesn't work in that case | 17:32 |
| arigo | (otherwise, it works already) | 17:32 |
| stakkars | while I doubt this case is so importante,since the codewillspend most time in the calledbig one. | 17:32 |
| arigo | yes, though remember it's also about enabling further optimizations | 17:33 |
| arigo | but indeed, I doubt this case is common | 17:33 |
| stakkars | correction: if thjis leads to inlining of locals, then you are absolutely right | 17:33 |
| pedronis | it's more relevant as a dynamic opt probably | 17:33 |
| pedronis | because the small one could contain a fast-path for some case | 17:33 |
| pedronis | diferring to the big one otherwise | 17:33 |
| stakkars | what are you trying: inline the smallee into its caller, or the bigee into the smallee? | 17:35 |
| pedronis | the smaller | 17:35 |
| stakkars | into its caller, which then needs to cope with the biggies unknown exceptions | 17:36 |
| pedronis | yes, but it's not a priority I think | 17:39 |
| pedronis | I'm still wondering why we got slower btw | 17:40 |
| stakkars | well, maybe it is, if I think of the three-times indirection of multimethod calls, which are often some tiny wrappers around something bigger, which actually has some fast path. But you always walk through this shashlik. | 17:41 |
| stakkars | I gues it would even pay off if we just do worst-case assumptions of exceptions. | 17:41 |
| arigo | stakkars: in this case, most all calls are "virtual", so no inlining possible | 17:42 |
| arre (i=ac@ratthing-b40e.strakt.com) left irc: Remote closed the connection | 17:42 | |
| stakkars | when I single step through marshal_w calls, I see helper functions called which call something from their minilocals, which is actually a constant. | 17:44 |
| pedronis | yes we have a bit of that | 17:45 |
| stakkars | I agree that it ends upin a virtual call,but the preparation for that can be flattened. | 17:45 |
| stakkars | at least we could handle these minilocals entries like struct members and dispose of the dicts | 17:47 |
| pedronis | eh? | 17:47 |
| pedronis | the constant are captured by the flow graph | 17:48 |
| arigo | the minilocals is not a dict any more than the usual globals | 17:48 |
| stakkars | fine, didn't know that. sorry, I'm missing a small enough C file tofigure out what exactly is gone and what stays. | 17:49 |
| pedronis | the change to the dict impls was done at 17740 | 17:49 |
| Nick change: tic -> tic|away | 17:50 | |
| stakkars | (or bad if the fruit hangs higher and higher) | 17:50 |
| pedronis | so the question is what was the speed before that | 17:51 |
| stakkars | do you need a test? | 17:52 |
| stakkars | arigo: sure,it as adumb assumption after I know what happens toglobals, whether mini or not :-) | 17:53 |
| pedronis | stakkars: yes, you could try to compile the rev before | 17:53 |
| stakkars | ok,please givemean existing rev, this time | 17:54 |
| stakkars | 17470? | 17:55 |
| pedronis | 17439 | 17:55 |
| pedronis | the change was at 17440 | 17:55 |
| pedronis | I typed a seven too much | 17:55 |
| ironfroggy (n=calvin@68.83.122.167) left #pypy ("Konversation terminated!"). | 17:56 | |
| stakkars | ok, reverting to that. | 17:56 |
| pedronis | to 17439 | 17:56 |
| stakkars | there was one change to the build that might affect translation: the /Op option to the compiler. But I doubt it is relevant | 17:59 |
| arigo | does it only affect float math? | 17:59 |
| stakkars | the option is documented as doing just that,yes. But who knows | 18:00 |
| stakkars | the 17439 compilation now runs without that. If in doubt, I can do it for the current version as well. | 18:00 |
| pedronis | richards doesn't use floats I think | 18:01 |
| stakkars | no, and it wouldn't even count, if you look into the C code what it's busy with :) | 18:02 |
| stakkars | real work is still little compared to overhead | 18:03 |
| Action: stakkars will stop the day after this, frustrated, produced almost nothing but hot air, buhuu | 18:04 | |
| stakkars | btw. I see that valid and everused are mapped to char in dict entries. DO we deduce such automatically,or is theresome hint? | 18:29 |
| pedronis | when we define the struct | 18:30 |
| pedronis | rdict.py | 18:30 |
| pedronis | self.DICTENTRY = | 18:31 |
| HelmutG (i=helmut@subdivi.de) joined #pypy. | 18:33 | |
| HelmutG | Hi. Do you think having Debian packages would be some sort of contribution or is this generally unneeded? | 18:34 |
| arigo | stakkars: Bool maps to a C 'char' in genc.py | 18:34 |
| arigo | HelmutG: hi! no, I think it's not useful at the moment -- thanks for the proposal | 18:34 |
| HelmutG | arigo: Can you estimate a point when it might be useful? | 18:35 |
| arigo | hard to say | 18:35 |
| arigo | it would be when PyPy itself starts to become useful, which it definitely isn't at the moment | 18:36 |
| HelmutG | arigo: Would 6 to 12 months be such a range? | 18:36 |
| arigo | I'd assume so | 18:36 |
| HelmutG | That's what I wanted to know, thanks. | 18:36 |
| arigo | :-) | 18:36 |
| HelmutG | Tell me if you need a maintainer. | 18:36 |
| arigo | ok, we've got you in the logs, when we need to look back :-) | 18:37 |
| stakkars | new benchmarks, soon | 18:42 |
| pedronis | good | 18:43 |
| arigo | I wrote a function to compute the statistical execution cost of a function | 18:43 |
| arigo | but it uses Numeric | 18:43 |
| arigo | to inverse a matrix | 18:43 |
| arigo | is there some code available around there to do that in pure Python? | 18:43 |
| pedronis | doesn't pygame requires Numberic too | 18:44 |
| arigo | no, optionally | 18:44 |
| arigo | I wouldn't like to add Numeric as a dependency at this point | 18:45 |
| stakkars | pypy-c-17439.exe 40707 ms | 18:45 |
| pedronis | so we are progressively getting slower | 18:46 |
| stakkars | pypy-c-current.exe 45946 ms | 18:47 |
| stakkars | (my measure differ,depending on ho much processes are there, but I always run both in the same env) | 18:47 |
| pedronis | but it seems that overall the dict changes did not improve things | 18:48 |
| stakkars | python 2.3.3 850 ms but for 10 iterations | 18:49 |
| xorAxAx | factor 540? | 18:49 |
| arigo | then we need again specialization for str keys | 18:49 |
| stakkars | I fearso, yes | 18:49 |
| arigo | well, wait a minute | 18:49 |
| arigo | it's also possible that there is simply an extra call indirection somewhere | 18:50 |
| pedronis | yes, I was thinking about that | 18:50 |
| arigo | all these numbers are a bit meaningless before we can remove the most horrible inefficiencies | 18:50 |
| stakkars | yes, I observed jumps by 20 percent for no good reason at all, which might just be due to code ordering | 18:51 |
| pedronis | I was the str dict impl calling hash and equality | 18:51 |
| stakkars | we must be loosing heavily at some point which we don't know | 18:52 |
| stakkars | doing pystone, now | 18:52 |
| stakkars | stunning results,all over the place !!! | 18:55 |
| stakkars | version 17439 592.966 pystones | 18:55 |
| stakkars | version current 615.233 pystones | 18:55 |
| stakkars | python 2.3.3 37986.2 | 18:55 |
| pedronis | arigo: the string implementation of comparing hashes the before doing eq | 18:56 |
| pedronis | s/of/was | 18:56 |
| pedronis | although the string we are mostly interested in should be interned | 18:56 |
| stakkars | we must be incredibly worse at instance attribute access. | 18:56 |
| arigo | I see | 18:57 |
| stakkars | which we thought would become better with the new dict | 18:57 |
| pedronis | but we don't intern strings in code | 18:58 |
| pedronis | so we are probably worse | 18:58 |
| stakkars | but we use cached strig hashes I hope | 18:59 |
| pedronis | no, but the new dict impl | 18:59 |
| pedronis | before trying equality | 18:59 |
| pedronis | just tries identity | 18:59 |
| stakkars | which always fails | 19:00 |
| pedronis | well, not for the string in corresponding to special methods in types | 19:01 |
| pedronis | but probably yes | 19:01 |
| stakkars | 540, compared to about 70 with pystone | 19:01 |
| pedronis | for the app-level strings in instance dicts | 19:01 |
| stakkars | which richards is exercizing exhaustively | 19:01 |
| dialtone (n=dialtone@merlinux.de) left irc: Read error: 104 (Connection reset by peer) | 19:04 | |
| hpk (n=hpk@merlinux.de) left irc: Read error: 104 (Connection reset by peer) | 19:04 | |
| hpk (n=hpk@merlinux.de) joined #pypy. | 19:05 | |
| stakkars | we really really need interning | 19:05 |
| dialtone (n=dialtone@merlinux.de) joined #pypy. | 19:05 | |
| pedronis | our hold interp level dict implementation was caching the hash | 19:06 |
| pedronis | in the entries | 19:06 |
| pedronis | and comparing it | 19:06 |
| stakkars | which is like a surrogate for interning. And we don't cache hash in the string impl? | 19:07 |
| pedronis | well, now everything happens one level down | 19:07 |
| stakkars | level down is ok, but why slow down. :) | 19:08 |
| pedronis | well, CPython caches the hash too | 19:10 |
| stakkars | yes, in the dict table and in the strings | 19:11 |
| stakkars | beating a Python dict's speed is really hard. AFAIK it'smuch faster than all the C++ mapping libraries | 19:12 |
| xorAxAx | whats the algorithm behind? | 19:13 |
| stakkars | dictobject.c, dict_look explains it. | 19:15 |
| pedronis | stakkars: I think that we should cache the hash too in rdict for non primitive types, although it will make the code more involved | 19:15 |
| stakkars | using a good algorithm to cycle through all entries, using size optimized to have an 1.5 change to find the entry on the first hit, caching hashes, well I can't write a book, here | 19:16 |
| stakkars | pedronis: absolutely. acutally for all pointer types? | 19:17 |
| pedronis | basically the repr of the key should tell whether to or not | 19:17 |
| pedronis | for ints is meaningful | 19:17 |
| pedronis | is not | 19:17 |
| stakkars | and table layout becomes dependant of what it will contain | 19:18 |
| pedronis | yes | 19:18 |
| pedronis | and you need more flags | 19:18 |
| stakkars | sure, not the types which are almost their own hash | 19:18 |
| pedronis | in the repr | 19:18 |
| stakkars | more flags than CPython? | 19:18 |
| stakkars | ah, in the repr | 19:18 |
| pedronis | yes to control what operations lookup should do | 19:18 |
| stakkars | at some point duplicate lookups might be more readable than one hyper generic one | 19:19 |
| pedronis | likely yes; of course we have the problem with valid and everused and hash and the overall used space | 19:20 |
| stakkars | where valid and everused are necoded differently in CPython,because they just have pointer? | 19:20 |
| pedronis | yes | 19:21 |
| stakkars | as an observation: The efficiency that we lost by unoptimizing partially is outweighing the win that we wanted to gain by turning the slot entries into a table. | 19:22 |
| stakkars | I said it cannot stay this way,and I was wrong unless we push further. | 19:22 |
| pedronis | well, not using the cached hash is a big it on lookup | 19:23 |
| pedronis | and the entries were a major cost on setitem and a bit the indirection | 19:23 |
| stakkars | maybe we turn it just on everywhere and remove it for simple types,later? | 19:24 |
| pedronis | you mean the cached hash | 19:24 |
| pedronis | ? | 19:24 |
| stakkars | yup | 19:24 |
| pedronis | yes, you could try with that | 19:24 |
| pedronis | and see how much it helps | 19:24 |
| pedronis | I'm about to go home, will join later from there | 19:25 |
| pedronis | see you | 19:25 |
| pedronis (i=pedronis@ratthing-b246.strakt.com) left irc: "Chatzilla 0.9.67 [Firefox 1.0.2/20050325]" | 19:25 | |
| stakkars | bye | 19:26 |
| arigo | bye | 19:40 |
| arigo (n=arigo@134.99.112.244) left irc: Remote closed the connection | 19:40 | |
| token_ (n=qwertyui@200.193.161.60) joined #pypy. | 19:43 | |
| aleale (n=andersle@clogs.dfki.uni-sb.de) left irc: Read error: 110 (Connection timed out) | 19:46 | |
| pedronis (n=Samuele_@213.112.139.57) joined #pypy. | 20:00 | |
| idnar (i=mithrand@unaffiliated/idnar) joined #pypy. | 20:16 | |
| idnar (i=mithrand@unaffiliated/idnar) left irc: Client Quit | 20:19 | |
| ericvrp | pedronis, arigo: the LLVM formula for inline costs can be found in llvm/lib/Transforms/IPO/InlineSimple.cpp | 20:21 |
| ericvrp (n=chatzill@ericvrp.demon.nl) left irc: "Chatzilla 0.9.68.5.1 [Firefox 1.0.6/20050727]" | 20:51 | |
| __alex (n=alex@141.22.64.4) left irc: Remote closed the connection | 21:10 | |
| ludal (n=ludal@lab75-1-81-57-254-81.fbx.proxad.net) joined #pypy. | 21:17 | |
| stakkars | pedronis:had a small bug,building once more. we will know more in 40 min | 21:48 |
| stakkars | (will need to augment test_rdict, resizing is not properly tested) | 21:49 |
| HelmutG (i=helmut@subdivi.de) left #pypy. | 21:53 | |
| stakkars | I'm pretty sure this will show quite a difference. On instance dicts,calling the string compare via callback and through the objspace abstraction is probably very expensive, which the hash cache minimizes | 21:53 |
| ludal | hi stakkars | 21:55 |
| stakkars | hi | 21:55 |
| ludal | can you translate_pypy with latest svn ? | 21:56 |
| stakkars | yes | 21:56 |
| ludal | ok so it must be my changes then ;) | 21:56 |
| stakkars | rev. 17512 is what I compiled last. | 21:56 |
| Rhamphoryncus (n=rhamph@unaffiliated/rhamphoryncus) joined #pypy. | 21:57 | |
| ludal | that's what I'm trying now, although it's with targetcompiler.py | 21:57 |
| stakkars | that's different.I use targetpypystandalonewithoutthread :-) | 21:58 |
| ludal | btw did you try to profile the translation process ? | 21:58 |
| stakkars | (my special version where I hacked threads away) | 21:58 |
| stakkars | yes. | 21:58 |
| stakkars | you cannot do it with profile.py, that thing seems to have deeper bugs. | 21:59 |
| ludal | well, the bugs I saw came from the changes I introduced and reworked by pedronis | 21:59 |
| ludal | using hotspot ? | 21:59 |
| stakkars | I got it with hotspot, yes. | 21:59 |
| stakkars | profile.py is not happy with PyPy at all. It crashes in the rtyper phase. | 22:00 |
| xorAxAx | isnt it python-only? | 22:00 |
| stakkars | yes. But it seems to have some assumption about which frames should be where when, which PyPy does not fit :) | 22:01 |
| xorAxAx | and what kind of crash do you see? | 22:02 |
| stakkars | wrong frame, it stops with a message. just try it | 22:02 |
| stakkars | it works when I run PyPy upon Python 2.3, but not on 2.4 | 22:03 |
| xorAxAx | ok | 22:03 |
| stakkars | and hotspot is much faster. drawback: it takes years to read its huge stats file in. | 22:03 |
| ludal | is there a starting option to run pypy and/or translation with profiling ? | 22:05 |
| pedronis | ludal: I'm trying targetcompiler | 22:06 |
| pedronis | but the problem I'm seeing seems related | 22:06 |
| pedronis | to the changes to iterators | 22:06 |
| pedronis | and the fact that the union | 22:06 |
| pedronis | was not updated | 22:06 |
| pedronis | for them | 22:06 |
| ludal | yes | 22:06 |
| ludal | I think I fixed that with my local changes because I don't see the bug any more | 22:07 |
| pedronis | do you see another one? | 22:07 |
| ludal | I just got stuck on another dict[block]=block | 22:07 |
| ludal | for now no | 22:07 |
| pedronis | in theory we can support | 22:07 |
| pedronis | dict on instances | 22:07 |
| pedronis | I'm with instances as keys | 22:08 |
| pedronis | using idenities | 22:08 |
| pedronis | is just that things are not wired in the rtyper for that | 22:08 |
| ludal | the fixes I made are not entirely related but I thought it would be a good idea to implement contains on some types (Float, Integer, Bool) without using pair(x,y).union | 22:08 |
| ludal | yes it actually crashes in the rtyper | 22:09 |
| ludal | whenever I see dict of blocks I can change it easily since blocks have a .bid field which uniquely identifies them | 22:09 |
| pedronis | yes, but we could simply enable supporting them | 22:10 |
| ludal | sure, and you'll probably finish it before I get through with the compiler ;) | 22:10 |
| pedronis | no, it's a bit involved not too much but a bit | 22:11 |
| pedronis | because hash support for instances need to be enabled per class | 22:12 |
| ludal | hash support as with __hash__ or more complex than that ? | 22:12 |
| pedronis | basically when we see a SomeInstance has key of a dict with to put the classdef on a table | 22:13 |
| pedronis | on the bookkeeper | 22:14 |
| ludal | btw It may be just an impression, but the translation process with .contains changes seem to use less memory (I'm at 190Mb still raising so it may just be an impression) | 22:14 |
| pedronis | did you write tests :) ? | 22:14 |
| ludal | well there is already test_contains for this one | 22:14 |
| ludal | and they pass | 22:14 |
| ludal | as for SomeBool changes I made earlier I couldn't find any test checking the semantics of equality, so I just followed the trend ;) | 22:15 |
| ludal | the thing is there is this assert s_value.contains[self.bindings[arg]] somewhere in annrpython.py | 22:16 |
| pedronis | yes | 22:16 |
| pedronis | is quite useful | 22:16 |
| ludal | and I just don't know what it's trying to verify | 22:16 |
| ludal | the thing is after the changes I triggered it that was the reason I changed SomeBool.__eq__ | 22:17 |
| pedronis | that new bindings are generalisation of the old ones | 22:17 |
| pedronis | for knowndatatype is bit trickier to make that happy | 22:17 |
| ludal | when you rebind an existing variable ? | 22:17 |
| pedronis | yes | 22:18 |
| pedronis | it checks the basic invariant | 22:18 |
| pedronis | of the whole annotation process | 22:18 |
| ludal | well for testing the changes I just compiled pypy and it worked :-/ | 22:18 |
| ludal | ok | 22:18 |
| ludal | I got this "TyperError: can only iterate over tuples of length 1 for now" | 22:20 |
| ludal | can I just convert my tuple to a list ? | 22:20 |
| pedronis | yes | 22:21 |
| pedronis | I'm writing a test about dicts with instance keys | 22:21 |
| pedronis | will try to make it pass | 22:22 |
| ludal | ok | 22:22 |
| ludal | pedronis I have this local fix for iterators: | 22:25 |
| ludal | class __extend__(pairtype(SomeIterator, SomeIterator)): | 22:25 |
| ludal | 22:25 | |
| ludal | def union((iter1, iter2)): | 22:25 |
| ludal | - return SomeIterator(unionof(iter1.s_container, iter2.s_container)) | 22:25 |
| ludal | + cont = unionof(iter1.s_container, iter2.s_container) | 22:25 |
| ludal | + assert iter1.variant == iter2.variant | 22:25 |
| ludal | + return SomeIterator(cont,*iter1.variant) | 22:25 |
| pedronis | makes sense | 22:25 |
| ludal | I'm not sure I got all the subtelties of variant | 22:26 |
| ludal | it seems to be used only for dicts? | 22:26 |
| ludal | and although its a tuple I could only see one or zero item in it... | 22:26 |
| _hannes (i=ore@i577B4751.versanet.de) left irc: "utz utz utz" | 22:51 | |
| pedronis | ludal: I'm running the tests but it seems the basic support for instance keys is there | 22:51 |
| ludal | this could ease the rest of the translation :) | 22:52 |
| stakkars | pedronis: new timings! | 22:53 |
| stakkars | firstof all asmallcorrection: | 22:53 |
| ludal | did you comit btw cause I just got another one of those dict{Block:Block} ? | 22:53 |
| pedronis | I'm running the tests | 22:53 |
| stakkars | we are not that bad as it sounded. My answer to xorAxAx was wrong: I was always comparing average time. | 22:53 |
| pedronis | well some tests | 22:53 |
| stakkars | so the factor was about 56,not 10 times worse. | 22:54 |
| xorAxAx | stakkars: ah, ok | 22:54 |
| stakkars | then, here what we got by the simple hash cache in the dict table: | 22:54 |
| stakkars | pypy-c-17439: 40929 ms pypy-c-17512: 46105 ms pypy-current: 33937 ms | 22:55 |
| stakkars | all average for one run | 22:55 |
| stakkars | this shows how very expensive the string comparison call into the space is. | 22:56 |
| stakkars | python 2.3: 856 ms | 22:57 |
| stakkars | which is a very good factorbelow 40, even better than pystone !! | 22:58 |
| pedronis | consindering that 0.7 had a factor of 300 or worse (with threads) | 22:59 |
| token_ (n=qwertyui@200.193.161.60) left #pypy ("Fui embora"). | 23:00 | |
| stakkars | I have boehm, no threads, -t-lowmem (which is probably not relevant) | 23:00 |
| pedronis | yes, 450 was threads and refcounting | 23:01 |
| stakkars | still, it is probably not the refcounting, but the direct malloc that makes everything else so relative. | 23:02 |
| pedronis | but we are going in the right direction | 23:02 |
| stakkars | yes,not backwards is quite a relief | 23:02 |
| stakkars | and now the new dicts really paif off, although we carry the overhead. | 23:03 |
| stakkars | pay off | 23:03 |
| stakkars | now a lot can be gained by special casing a few hash functions. Would be great if we knew | 23:04 |
| pedronis | ludal: checked in | 23:05 |
| stakkars | that an istance dict uses string keys only. DO we know that? | 23:05 |
| pedronis | it's a w_dict | 23:05 |
| pedronis | so it has W_Root keys | 23:05 |
| pedronis | not strings | 23:05 |
| stakkars | maybe better not now but at a later point | 23:05 |
| ludal | ok | 23:05 |
| stakkars | yes, but it will contain wrapped string keys, only. | 23:06 |
| stakkars | can't wefind that out? probably not on an application program. But we might try that first. | 23:06 |
| pedronis | well, in CPython you can inst.__dict__[1] = 2 | 23:07 |
| stakkars | sure I can. But I can assume that the common case is a string key, and tweak dict hash to find that variant first | 23:07 |
| Rhamphoryncus (n=rhamph@unaffiliated/rhamphoryncus) left irc: "Support ISO 8601! http://www.cl.cam.ac.uk/~mgk25/iso-time.html" | 23:34 | |
| pedronis | going to try to sleep | 23:55 |
| pedronis | night | 23:55 |
| pedronis (n=Samuele_@213.112.139.57) left irc: "Chatzilla 0.9.68a [Firefox 1.0.4/20050511]" | 23:55 | |
| --- Tue Sep 13 2005 | 00:00 | |
Generated by irclog2html.py 2.9.2 by Marius Gedminas - find it at mg.pov.lt!