[00:10] _hannes (htttgz@82.140.29.121) left irc: "utz utz utz"
[00:17] esteban (esteban@DWM-55-230.go.retevision.es) left #pypy.
----- silence for 38 minutes ----- [00:55] pedronis (~Samuele_P@c-1e8b70d5.022-54-67626719.cust.bredbandsbolaget.se) left irc: "Chatzilla 0.9.66 [Mozilla rv:1.7.5/20041107]"
----- silence for 47 minutes ----- [01:42] stakkars (updrjx@82.140.29.121) joined #pypy.
[01:43] stakkars (updrjx@82.140.29.121) left #pypy.
[01:53] stakkars (~tismer@82.140.29.121) joined #pypy.
[01:57] CTCP PING: 1109116362 from stakkars (~tismer@82.140.29.121)
----- silence for 24 minutes ----- [02:21] dialtone (~dialtone@host116-56.pool80117.interbusiness.it) left irc: Read error: 113 (No route to host)
----- silence for 17 minutes ----- [02:38] [stakkars!tismer@82.140.29.121] huh
[02:40] CTCP TIME: from stakkars (~tismer@82.140.29.121)
[02:41] CTCP VERSION: from stakkars (~tismer@82.140.29.121)
[02:43] CTCP PING: 1109119159 from stakkars (~tismer@82.140.29.121)
[02:44] Action: stakkars wonders how to ask the bot
----- silence for 35 minutes ----- [03:19] stakkars (tismer@82.140.29.121) left #pypy.
[03:30] stakkars (ayobfazg@82.140.29.121) joined #pypy.
[03:31] Continuity (~kittish@host81-156-158-172.range81-156.btcentralplus.com) left irc: Read error: 60 (Operation timed out)
----- silence for 3 hr and 9 minutes ----- [06:40] rxe (~rxe@adsl-63-206-191-33.dsl.sktn01.pacbell.net) joined #pypy.
----- silence for 23 minutes ----- [07:03] idnar (mithrandi@idnar.user) left irc: Nick collision from services.
[07:03] idnar_ (mithy@idnar.user) joined #pypy.
----- silence for 46 minutes ----- [07:49] Nick change: idnar_ -> idnar
----- silence for 1 hr and 55 minutes ----- [09:44] arre (ac@ratthing-b3fa.strakt.com) joined #pypy.
----- silence for 27 minutes ----- [10:11] adim (~adim@logilab.net2.nerim.net) joined #pypy.
----- silence for 16 minutes ----- [10:27] sanxiyn (~tinuviel@218.145.51.39) joined #pypy.
[10:29] dlk (~dlk@walter.ita.chalmers.se) joined #pypy.
[10:31] <sanxiyn> dlk: hi
[10:40] _dlk (~dlk@dhcp-246-12.nomad.chalmers.se) joined #pypy.
[10:41] dlk (~dlk@walter.ita.chalmers.se) left irc: Nick collision from services.
[10:42] Nick change: _dlk -> dlk
[10:42] dlk (dlk@dhcp-246-12.nomad.chalmers.se) left #pypy.
[10:42] dlk (~dlk@dhcp-246-12.nomad.chalmers.se) joined #pypy.
----- silence for 29 minutes ----- [11:11] sanxiyn (~tinuviel@218.145.51.39) left irc: "Àü À8¸ °©´ôÙ."
----- silence for 1 hr ----- [12:11] dlk (~dlk@dhcp-246-12.nomad.chalmers.se) left irc: "User pushed the X - because it's Xtra, baby"
[12:20] stakkars (ayobfazg@82.140.29.121) left irc: Read error: 60 (Operation timed out)
[12:28] ludal (~ludal@logilab.net2.nerim.net) joined #pypy.
[12:40] stakkars (~tismer@82.144.60.19) joined #pypy.
[12:54] pedronis (pedronis@ratthing-b246.strakt.com) joined #pypy.
[12:58] <stakkars> hi!
[12:59] <pedronis> hi
[13:00] <stakkars> I think I know a bit more now. I compiled several times with MSVC which is really a mess
[13:00] <stakkars> there is a chance that tcc is just fine. I just don't have time to test it now.
[13:00] <stakkars> The mess is here:
[13:01] <stakkars> the init func starts with this body:
[13:01] <stakkars> #define SETUP_MODULE(modname) \
[13:01] <stakkars> PyObject *m = Py_InitModule(#modname, no_methods); \
[13:01] <stakkars> PyModule_AddStringConstant(m, "__sourcefile__", __FILE__); \
[13:01] <stakkars> this_module_globals = PyModule_GetDict(m); \
[13:01] <stakkars> and that's bad because it starts with a declaration. You can'tput a breakpoint there.
[13:02] <stakkars> we have an overdose of macros in the code.
[13:02] <stakkars> what I found out so far is, that Python does a PyObject_GetAttr with a null argument.
[13:03] <stakkars> This is called by some f___new__ which is called by some fastf___new__
[13:03] <stakkars> (swap the last,please)
[13:17] Continuity (kittish@host81-156-158-172.range81-156.btcentralplus.com) joined #pypy.
[13:25] Nick change: pedronis -> pedronis_lunch
----- silence for 41 minutes ----- [14:06] <stakkars> when using debug, we get a different error.
[14:06] rxe (~rxe@adsl-63-206-191-33.dsl.sktn01.pacbell.net) left irc: Read error: 104 (Connection reset by peer)
[14:06] <stakkars> (gdb) bt
[14:06] <stakkars> #0 0x4000a757 in _dl_relocate_object () from /lib/ld-linux.so.2
[14:06] <stakkars> #1 0x4019e1d3 in getutmpx () from /lib/libc.so.6
[14:06] <stakkars> #2 0x4000bf26 in _dl_catch_error () from /lib/ld-linux.so.2
[14:06] <stakkars> #3 0x4019e456 in _dl_open () from /lib/libc.so.6
[14:06] <stakkars> #4 0x4006f008 in ?? () from /lib/libdl.so.2
[14:07] <stakkars> #5 0x40016c00 in ?? () from /lib/ld-linux.so.2
[14:07] <stakkars> #6 0xbfffebc0 in ?? ()
[14:07] <stakkars> #7 0xbfffead8 in ?? ()
[14:07] <stakkars> #8 0x4000bf26 in _dl_catch_error () from /lib/ld-linux.so.2
[14:07] <stakkars> #9 0x4000bf26 in _dl_catch_error () from /lib/ld-linux.so.2
[14:07] <stakkars> #10 0x4006f4c6 in dlerror () from /lib/libdl.so.2
[14:07] <stakkars> #11 0x4006f054 in dlopen () from /lib/libdl.so.2
[14:07] <stakkars> #12 0x080dcfc6 in _PyImport_GetDynLoadFunc ()
[14:07] <stakkars> so that seems in fact to be a limit of something.
[14:07] <stakkars> The real error seems to be portable, I get it with tcc and Windows, too.
[14:14] _hannes (czvoxc@82.140.29.121) joined #pypy.
[14:15] Nick change: pedronis_lunch -> pedronis
[14:15] <_hannes> morning guys#
[14:15] <_hannes> everything okay with the bot?
[14:16] <pedronis> stakkars: yes I got that error insinde the what seems the dynamic linking code too
[14:17] <stakkars> I'm doing a release build on windows with optimization disabled, now. That shouldcompile faster and let me debug.
----- silence for 27 minutes ----- [14:44] yuuh (awyuoj@i528C1EFA.versanet.de) joined #pypy.
[14:45] Nick change: yuuh -> _hannes_
[14:48] <stakkars> pedronis: I stopped this mess now. After I now get my MS project compiled rather quickly, I still cannot set break points, because line numbers are limited to 2**16. ARRRGG
[14:49] <pedronis> stakkars: yes
[14:49] <pedronis> right now I'm working on new code
[14:49] <pedronis> to fill up the caches properly
[14:49] <pedronis> after the changes we did
[14:49] <pedronis> to multimethods
[14:49] <pedronis> and the last branch
[14:50] <stakkars> you think there is a wrong initialisation order?
[14:50] <stakkars> or you say you just don't think of the c codenow?
[14:50] <stakkars> as a last attempt before lunch, I will hard-code a break point into it.
[14:51] <pedronis> it is not related
[14:51] <pedronis> although maybe it could be
[14:51] <stakkars> ok.
[14:51] <pedronis> we need anyway proper code
[14:51] <pedronis> to fill caches
[14:51] <pedronis> things are much more lazy now
[14:51] <stakkars> Anyway, I will try to stop this whole mess. Generating this kind of codedoes not make sense.
[14:51] <pedronis> yup
[14:51] <pedronis> I agree
[14:52] <pedronis> I the cache filling code
[14:52] <pedronis> because I want to see
[14:52] <pedronis> whether things have improved
[14:52] <pedronis> for the annotator
[14:52] <pedronis> but it really need laziness removed
[14:52] <pedronis> otherwise it dies pretty soon
[14:53] <stakkars> fine. (and I remember how much I dislike unneeded laziness)
[14:53] <pedronis> well, removing laziness for MMs and the builtin modules is easy
[14:53] <pedronis> it's just a matter of asking the full dict of type and the modules
[14:54] <stakkars> I'm still hoping that at some point we really have a simple, linear setup routine for everything. It must exist, it is just a matter of order.
[15:00] _hannes (czvoxc@82.140.29.121) left irc: Read error: 110 (Connection timed out)
[15:12] <stakkars> pedronis: ok, I could debug it a little.
[15:13] <stakkars> pedronis: we die in the evaluation of the frozen python code.
[15:13] <pedronis> yes
[15:14] <stakkars> there is a PyObject_GetAttr done on a type object...(wait)
[15:15] <stakkars> gcls_r_uint is the type.
[15:15] <stakkars> the name argument given is a NULL, unfortunately.
[15:16] <stakkars> happens in fastf___new__
[15:18] <pedronis> I see
[15:18] <pedronis> of course
[15:18] <stakkars> maybe something is wrong with the parsing.
[15:18] <pedronis> no
[15:18] <pedronis> we create instances of our own defined type
[15:19] <stakkars> before initializing it?? :-)
[15:19] <pedronis> in the old model everything was in globals
[15:19] <pedronis> to do that
[15:19] <pedronis> we need to run our own functions
[15:19] <pedronis> but while executing the python code
[15:19] <pedronis> we still haven't filled in the
[15:19] <pedronis> global constants
[15:19] <pedronis> that our c code uses
[15:20] <stakkars> sure. I thought this is what the python code is supposed todo?
[15:20] <pedronis> but the filling global constants
[15:20] <pedronis> is done afterward
[15:20] <pedronis> which is too late
[15:20] <stakkars> :-/
[15:20] <pedronis> in some cases
[15:20] <pedronis> does this make sense?
[15:20] <stakkars> yup
[15:21] <pedronis> at least that's my understanding of the new code
[15:21] <stakkars> so we need to initialize something more before running the marshalled code?
[15:22] <pedronis> no
[15:22] <stakkars> ah, setup_globalobjects is run later.
[15:22] <pedronis> yes
[15:22] <pedronis> we setup the functions
[15:22] <pedronis> but the functions
[15:22] <pedronis> need the constants in some cases
[15:22] <pedronis> we really need to pass in a function
[15:23] <pedronis> to our python code
[15:23] <pedronis> to setup the constant
[15:23] <pedronis> that means
[15:23] <stakkars> so we cannot switch order. yes
[15:23] <pedronis> no
[15:23] <pedronis> but the order I think is still ok
[15:23] <stakkars> the python code needs to do the right things at the right time.
[15:23] <pedronis> yes
[15:23] <pedronis> which it does
[15:23] <pedronis> because is emitted
[15:23] <pedronis> in the order
[15:24] <pedronis> things are needed
[15:24] <pedronis> like the old init c code
[15:24] <pedronis> but what setup_globalsobject
[15:24] <pedronis> does
[15:24] <pedronis> should be done
[15:24] <pedronis> interspersed
[15:24] <pedronis> with constructing object using python
[15:24] <pedronis> that means
[15:24] <stakkars> yes. so the setup_glob must be merged into the python
[15:25] <pedronis> instead of having the python code do
[15:25] <pedronis> just assignments
[15:25] <pedronis> g... = ...
[15:25] <pedronis> it needs a function
[15:25] <pedronis> setup_global
[15:25] <pedronis> to call
[15:25] <pedronis> that does one at a time the work
[15:25] <pedronis> that setup_globalobjects
[15:25] <pedronis> is doing too late right now
[15:26] <stakkars> ahh, ahh, the globals of the python initializer are not the globals the C code njeeds!
[15:26] <stakkars> I see, the objects must be put into their C slot as well.
[15:27] <pedronis> yes
[15:27] <pedronis> we need a dict or something instead of table
[15:27] <pedronis> with name -> adress
[15:27] <pedronis> and then it should be possible
[15:27] <pedronis> to write such a setup_global
[15:27] <pedronis> to pass to the cpython init code
[15:29] <pedronis> which then should look like
[15:29] <pedronis> <gname> = ...
[15:30] <pedronis> setup_global("<gname>",<gname>)
[15:30] <pedronis> because the python init code
[15:30] <pedronis> also reuses some of its globals
[15:30] <pedronis> so we need both to setup the python global and the global constant
[15:31] <pedronis> for c
[15:32] <pedronis> stakkars: do you want to try this out?
[15:33] <pedronis> it's interesting because the problem
[15:33] <pedronis> is not triggered
[15:33] <pedronis> but any of our smaller
[15:33] <pedronis> tests
[15:38] <stakkars> still guebeling...
[15:38] <stakkars> gruebel
[15:40] <stakkars> in a sense, it could work without extra addresses. We know the order of the globals appearing...
[15:41] <stakkars> they could be arranged in exactly that order, in a globals array.
[15:42] <stakkars> huh, it would even be predictable where every object will sit in the globals dict. We could use that directly (no I won't) :-)
[15:44] <stakkars> ok, I will do that!
----- silence for 1 hr and 5 minutes ----- [16:49] hpk (~hpk@mail.trillke.de) joined #pypy.
----- silence for 1 hr and 11 minutes ----- [18:00] dlk (~dlk@walter.ita.chalmers.se) joined #pypy.
[18:04] Nick change: _hannes_ -> _hannes
[18:06] idnar (mithy@idnar.user) left irc: "kbye"
[18:13] <pedronis> stakkars: how things are going
[18:14] _hannes (awyuoj@i528C1EFA.versanet.de) left irc: Read error: 104 (Connection reset by peer)
[18:17] <stakkars> I had a different problem here which distracted me.
[18:17] <stakkars> I can work on it in an hour or so.
[18:17] <stakkars> But I have a concept and started implementing.
[18:18] <pedronis> stakkars: maybe we don't need it
[18:18] <stakkars> uhh
[18:18] <pedronis> Armin assumption
[18:18] <pedronis> that we don't need to call our own function during init was mostly right
[18:18] <pedronis> functions
[18:19] <pedronis> I thought a bit more
[18:19] <stakkars> so you mean there was just a smallish oversight which can be solvedfrom Python
[18:19] <pedronis> I think the only offenders are possibly __new__ methods
[18:19] <pedronis> which btw are from the restrict_integer classes
[18:19] <pedronis> but the point is
[18:19] <pedronis> that the only code
[18:20] <pedronis> ourcls.__new__(ourcls)
[18:20] <pedronis> may call one of our functions
[18:20] <pedronis> I think
[18:20] <pedronis> the __new__ in fact
[18:20] <pedronis> Armin put code like that for the new-style classes case
[18:21] <pedronis> I expented it to deal with subclasses of immutable stuff
[18:21] <pedronis> like r_int classes
[18:21] <pedronis> uses code similiar to pickling
[18:21] <pedronis> and using __reduce_ex__
[18:21] <pedronis> etc
[18:22] <pedronis> but the correct thing to do
[18:22] <pedronis> for most cases
[18:22] <pedronis> is to use the __base__ class
[18:22] <stakkars> we do a PyObject_GetAttr on gcls_r_uint, but the *name* is not initialized. That's an ordinary strring.
[18:22] <pedronis> yes but inside a __new__
[18:23] <pedronis> we should not be calling it
[18:23] <pedronis> if we were not executing during init one of our own functions
[18:23] <pedronis> there would be no problem
[18:23] <pedronis> what I was saying
[18:23] <pedronis> I changed the above code involving __new__ to
[18:23] <pedronis> use the __base__ class
[18:24] <pedronis> of our own class
[18:24] <pedronis> so for a subclass of long
[18:24] <pedronis> we will be calling
[18:24] <pedronis> long.__new__(ourclass, value)
[18:24] <pedronis> we would not be calling ourclass.__new__ anymore
[18:25] <pedronis> this is also the kind of code that unpickling itself uses
[18:25] <pedronis> the failing getattr I suppose
[18:25] <pedronis> was the code retrieving _MASK from r_uint
[18:25] <pedronis> in r_uint.__new__
[18:26] <pedronis> if we call long.__new__(r_uint, value)
[18:26] <pedronis> the problem should go away
[18:26] <pedronis> the value set will be still correct
[18:26] <pedronis> because the value used there is the state
[18:26] <pedronis> from __reduce_ex__
[18:26] <pedronis> which already the correct masked thing
[18:26] <stakkars> I see. Hmm :(
[18:27] <pedronis> very tricky and subtle
[18:27] <stakkars> wouldn't it be cleaner not to rely on this?
[18:27] <pedronis> I 100% sure, but I think __new__
[18:27] <pedronis> well, it is checked in
[18:27] <pedronis> I need to try whether now things work
[18:28] <stakkars> letme do that
[18:28] <pedronis> the fact is that the original code
[18:28] <pedronis> was buggy
[18:28] <pedronis> because in fact pickling calls <__base__>.__new__
[18:28] <pedronis> exactly do avoid passing somehow modified state
[18:29] <pedronis> again to the original __new__
[18:29] <pedronis> I checked in a test btw
[18:29] <pedronis> if we still emitted cls.__new__(cls)
[18:29] <pedronis> it would fail
[18:29] <pedronis> OTOH I was not able to construct a seg faulting test
[18:29] <pedronis> because you need a tricky ordering of things to get that problem
[18:32] <pedronis> stakkars: you can look at my last check-in
[18:32] <pedronis> to also see what I'm referring to
[18:32] <stakkars> File "/home/tismer/dist-pypy/pypy/translator/genc.py", line 263, in nameof_instance
[18:32] <stakkars> assert reduced[1][1] is base_class
[18:32] <stakkars> AssertionError
[18:32] <stakkars> nogo
[18:32] <pedronis> ok
[18:33] <pedronis> so we need more sophisticated code
[18:34] <stakkars> seeking for bugs in genc is harder forme than than to make globals behave transparent
[18:37] <pedronis> yes
[18:37] <pedronis> but anyway the original code was buggy
[18:37] <pedronis> from other point of views
[18:37] <pedronis> I will fix the current problem anyway
[18:39] arg (~arg@sf86.us) left irc: Read error: 104 (Connection reset by peer)
[18:50] <stakkars> I needto change something to make things work under windows. Can I modify genc.h?
[18:51] <pedronis> yes
[18:51] <pedronis> I'm running translate_pypy
[18:51] <pedronis> right now
[18:51] <pedronis> and see where it goes
[18:52] <pedronis> I fixed the glitch you encountered above
[18:52] <stakkars> PyMODINIT_FUNC
[18:52] <stakkars> does the needed declspec
[18:53] <stakkars> ah, oh, ok. I'll check that in
[18:56] <pedronis> I checked in my fix for the glitch
[18:56] <pedronis> anyway is an improvement
[18:56] <pedronis> stakkars: translate_pypy is still going
[18:56] <stakkars> where do you compile? I'm starting now.
[18:57] <pedronis> on my desktop here
[18:57] <pedronis> which is slow
[18:57] <pedronis> but with tcc
[18:57] <pedronis> the compiling itself I think
[18:57] <pedronis> is faster than the analysis
[18:57] <stakkars> absolutely. tcc does it in 10 seconds.
[18:57] <pedronis> I'm still getflowgraph getflowgraph
[18:58] <stakkars> analysis is slow. No idea how to improve that
[18:59] <pedronis> well it is still reasonable
[18:59] <stakkars> yes, but all a bit boring. We are loosing the amenities of Python by bulky compilation.
[19:00] <stakkars> if there was a way to cache this stuff, I would cache it.
[19:00] <stakkars> I think I'm through with flowing.
[19:01] <pedronis> much faster machine
[19:02] <stakkars> well, I don't see anything on the screen. No, there was a longer pause of about 30 seconds, now I'm flowing again.
[19:02] <pedronis> ah
[19:02] <pedronis> yes
[19:02] <pedronis> there one point were computing the graph is slow
[19:02] <pedronis> stakkars:
[19:03] <pedronis> about caching
[19:03] <pedronis> I think being clever
[19:03] <stakkars> is this garbage collection? Or do we have quadratic behavior in some situations.
[19:03] <stakkars> caching,yes?
[19:03] <pedronis> I think the flowgraphs
[19:03] <pedronis> with some pickling like approach
[19:03] <pedronis> and some way
[19:03] <pedronis> to know whether
[19:03] <pedronis> single functions
[19:03] <pedronis> have changed
[19:03] <pedronis> could be cached
[19:04] <pedronis> but is not totally straightforward
[19:04] <pedronis> the fact is that the flowgraph contain constants
[19:05] <stakkars> no, because if I have a cached thing, I still need to examine the contents
[19:06] <stakkars> ah, no, not the problem. If I cache info about a function, then I also *have* the cached flowgraph, and I can record what this function references. !
[19:06] <pedronis> well it's a bit trickier I think
[19:07] <pedronis> but there should be some way to fill the constants in the cache
[19:07] <pedronis> by going through the function and the graph
[19:07] <pedronis> just once
[19:07] dannu (~hpk@a81-14-162-110.net-htp.de) joined #pypy.
[19:07] <pedronis> when we compute the graph we neeed to iterate
[19:07] idnar (mithrandi@idnar.user) joined #pypy.
[19:07] <pedronis> I mean the first time
[19:08] hpk (~hpk@mail.trillke.de) left irc: Read error: 110 (Connection timed out)
[19:09] <pedronis> stakkars: I get a PySys_GetObject
[19:09] <pedronis> is not defined
[19:09] <stakkars> File "entry_point_1", line 46135, in ?
[19:09] <stakkars> NameError: name 'PySys_GetObject' is not defined
[19:09] <stakkars> nAnU
[19:09] <pedronis> well, an improvement
[19:10] <stakkars> you mean this is a sign of less laziness?
[19:10] <pedronis> it is shallow
[19:10] <stakkars> looks like we touch stdin or such.
[19:10] <pedronis> yes
[19:11] <pedronis> I think the code to grab them
[19:11] <pedronis> was not updated
[19:11] <pedronis> we have in the .py file
[19:11] <pedronis> :
[19:11] <stakkars> :-D I see
[19:11] <pedronis> ginst_W_FakeFile.val = PySys_GetObject("stderr")
[19:11] <pedronis> ginst_W_FakeFile_1.val = PySys_GetObject("stdin")
[19:11] <pedronis> ginst_W_FakeFile_2.val = PySys_GetObject("stdout")
[19:11] <pedronis> instead of sys.*
[19:12] <pedronis> let's see where that is
[19:12] <stakkars> yes I remember
[19:12] <pedronis> yes, the code needs to be changed
[19:13] <stakkars> self.latercode.append("import sys")
[19:13] <stakkars> and then "return sys.stdin" ?
[19:13] <pedronis> no
[19:13] <pedronis> a name
[19:13] <pedronis> because it is used
[19:13] <pedronis> also from c side
[19:13] <pedronis> I think
[19:14] <stakkars> hmm
[19:14] <pedronis> we need gsys_stdout = ...
[19:16] <stakkars> ok. Do you do it or di I?
[19:19] <pedronis> I have done
[19:19] <pedronis> about to check in
[19:20] <stakkars> please do
[19:20] <pedronis> checked in
[19:20] <pedronis> you can try again
[19:20] <pedronis> and see how far we go now
[19:20] <stakkars> running.
[19:22] <stakkars> if have plans for some drastic code reductions, btw...
[19:22] <pedronis> yes, anyway we should think/re-think our code gen code
[19:22] <pedronis> but although painful
[19:23] <pedronis> the discovery about doing the right thing about __new__
[19:23] <pedronis> is useful
[19:23] <stakkars> yes, I was talking of genc.
[19:23] <stakkars> if it works, soon, then we can shrink-wash.
[19:24] <stakkars> want to get rid of the massesof decrefs and all the fail-labels. Instead, every step's cleanup can be done with a single call...
[19:25] <pedronis> as we discussed
[19:25] <pedronis> we should also look at having a common framework
[19:25] <pedronis> that the various gen*
[19:25] <pedronis> can reuse
[19:25] <stakkars> yes, I'm thinking of this (and have it on my todo-list)
[19:25] <pedronis> we should probably
[19:25] <stakkars> in fact this isn't really trivial.
[19:25] <pedronis> yep
[19:26] <pedronis> right now we have a naive strategy of mapping
[19:26] <pedronis> object -> name
[19:26] <stakkars> some funcs can be extracted. Others, to be reusable, will force us to provide langauge dependant snippets.
[19:26] <pedronis> but we probably need some intermediary
[19:26] <pedronis> object
[19:26] <pedronis> because I expect for some generators
[19:26] <pedronis> the expression to refer to something
[19:27] <pedronis> may vary by op/context
[19:27] <pedronis> so something object -> repr-object (-> can give e.g. a name)
[19:27] <stakkars> I see. The current assumption is that we want to name the object. WHich is naive.
[19:27] <stakkars> Actually, we are saying "I need this object to exist, and I need to name it".
[19:28] <stakkars> how this thing is rendered is actuallynot required in this context.
[19:28] <pedronis> en example
[19:28] <pedronis> of the problem
[19:29] <pedronis> is the my code
[19:29] <pedronis> to use space.xxx
[19:29] <pedronis> or the builtin.get('xxx')
[19:29] <pedronis> depending on the number of arguments
[19:29] <pedronis> we the current approach I need to hack it in
[19:29] <pedronis> s/we/with
[19:30] <pedronis> that's why I'm saying that we may need an imtermediary object
[19:30] <pedronis> that then could decide what to emit depending on the context, operations operand etc
[19:31] <stakkars> I agree
[19:31] <stakkars> still flowing in the wind...
[19:32] <stakkars> But we also make the bad general assumption that we want to generate code.
[19:32] <stakkars> And we mix the emitted text with the algorithm to find the needed objects.
[19:32] <pedronis> yes
[19:32] <pedronis> we have a one pass approach
[19:32] <pedronis> in some sense
[19:33] <stakkars> exactly. By tiny tricks, it gets sophisticated. Actually, also the initcode.append is bad.
[19:33] <stakkars> That code should not have knowledge about how the object and when it is initialized.
[19:34] <pedronis> it is tricky
[19:34] <pedronis> for example
[19:34] <pedronis> again about the __new__ methods
[19:34] <pedronis> one fun thins
[19:34] <pedronis> thing
[19:34] <pedronis> is that depending on order
[19:34] <stakkars> Running!
[19:34] <stakkars> <entry_point_1.gcls_W_IntObject object at 0x4100872c>
[19:34] <stakkars> 42
[19:34] <pedronis> :)
[19:34] <pedronis> sometimes the old
[19:34] <pedronis> code was
[19:35] <pedronis> using the base class __new__
[19:35] <pedronis> because
[19:35] <pedronis> cls.__new__ = ...
[19:35] <pedronis> wasn't done yey
[19:35] <pedronis> yet
[19:35] <pedronis> after that was done
[19:35] <pedronis> subsequent code
[19:35] <pedronis> cls.__new__(cls,...)
[19:36] <pedronis> was using the class own __new__ (which was the wrong thing but anyway)
[19:36] <pedronis> so ordering is tricky
[19:36] <pedronis> and can still not be very correct probably
[19:36] <pedronis> sometimes
[19:37] <stakkars> yes. There is also not a clean concept of what must be done before an object may be used. It happens to work. But we need amore abstractwayto describe dependencies, but just by coding some lazy things which get iterated at some time.
[19:39] <stakkars> sounds like "complete rewrite aleret".
[19:39] <stakkars> alert
[19:41] <stakkars> do you know what I have to support? Just genc, genjava and geninterp, or do we care about more?
[19:42] <pedronis> genc, genjava, geninterp is what we care right now
[19:42] <pedronis> at some point I need to start
[19:42] <pedronis> a genjava2
[19:42] <pedronis> that is more like genc
[19:42] <pedronis> genjava right now is meant to produce code
[19:43] <pedronis> to be analyzed again
[19:43] <pedronis> that the alternative path wrt the annotator
[19:43] Nick change: dannu -> hpk
[19:43] <pedronis> that Armin wants to explore
[19:43] <hpk> genllvm
[19:43] <pedronis> yes
[19:43] <pedronis> which btw
[19:43] <pedronis> I think is already using
[19:43] <pedronis> a slightly more sophisticated
[19:44] <pedronis> approach than our generators
[19:44] <stakkars> have to look into that.
[19:44] <hpk> armin thought so too IIRC
[19:45] <pedronis> I think it uses some intermediary objects
[19:45] <stakkars> without reducing them to names, too early
[19:45] <pedronis> I think so
[19:45] <stakkars> I would like to rip out the whole nameof_crap thing and make it stand-alone.
[19:47] <stakkars> the abstraction needs to take care of object dependencies and such. The problem is then how do we formulate code generation ina more abstract fashion (AND things must be shorter as well)
[19:48] dlk (~dlk@walter.ita.chalmers.se) left irc: Connection timed out
[19:49] <stakkars> pedronis: well, I'm hesitating. What genc is doing now with initialisation is probably the opposite of what it willdo when we are getting more complete?
[19:50] <pedronis> likely yes
[19:50] <stakkars> should I put effort into that part, then?
[19:50] <pedronis> OTOH we should support the current approach (living on top of CPython)
[19:51] <pedronis> fo a bit more
[19:51] <pedronis> I think that was it does now is sort of ok
[19:51] <pedronis> because with the last changes about __new__
[19:52] <pedronis> is not doing that much active stuff anymore
[19:52] <pedronis> the code now is doing things very similarly to unpickling
[19:53] <stakkars> well, that code is scattered over the diverse nameof_ things. I need a spelling, how do I tech this to a generic improved_nameof_class?
[19:53] <stakkars> yes, you are right. That's basically the tiny engine I was talking about. Unpickling is such an engine.
[19:54] <pedronis> yes, but unpickling does not need
[19:54] <pedronis> to be able to regenerate
[19:54] <pedronis> class definitions
[19:54] <pedronis> that's probably the part that needs more attention
[19:55] <pedronis> although it is true
[19:55] <pedronis> that much later
[19:55] adim (adim@logilab.net2.nerim.net) left #pypy.
[19:55] <pedronis> we will not do/be able to do things this way
[19:56] <pedronis> our class defs will be likely somehow statically defined
[19:56] <pedronis> but it really depends on our target runtime system will look like
[19:56] <pedronis> s/our/how
[20:06] <stakkars> genllvm looks quite good.
[20:10] ludal (~ludal@logilab.net2.nerim.net) left irc: Client Quit
[20:15] dialtone (~dialtone@host116-56.pool80117.interbusiness.it) joined #pypy.
[20:17] <stakkars> the irc bot works fine! look at
[20:17] <stakkars> http://nimrod.terra-link.net/pypy/%23pypy.log.23Feb2005
[20:23] <pedronis> yes, an entire day of work and bad spelling :)
[20:25] <stakkars> bad spelling is spelled speling :)
[20:27] stakkars_ (~tismer@82.144.60.19) joined #pypy.
[20:27] stakkars (~tismer@82.144.60.19) left irc: Read error: 54 (Connection reset by peer)
----- silence for 20 minutes ----- [20:47] arre (ac@ratthing-b3fa.strakt.com) left irc: Remote closed the connection
[20:58] Action: pedronis leaving the office
[20:58] pedronis (pedronis@ratthing-b246.strakt.com) left irc: "ChatZilla 0.9.61 [Mozilla rv:1.7.3/20041007]"
----- silence for 34 minutes ----- [21:32] <stakkars_> uhh!
[21:33] <stakkars_> we get about 470 f_run and fastf_run functions generated!
[21:35] <stakkars_> also there are 476 setfastscope functions.
[21:36] <stakkars_> all caused stuff from gateway.py
[21:41] <hpk> yes, i mentioned yesterday that we generate around 3200 lines of duplicate python code during startup
[21:42] <hpk> (just caching that increases startup time by 50%)
[21:42] <stakkars_> and this is really really bloating things.
[21:43] <hpk> i guess we need to go through the various places that generate code and see how we can avoid duplicate code
[21:44] <stakkars_> yes. Somke of these are simple things which could work easily using local scope. But I fear that exactly is not possible in the context where setfastscope and freindsmust run (rpython)
[21:46] <stakkars_> btw., the huge source compiles and prints 42, again, after all.
----- silence for 18 minutes ----- [22:04] <stakkars_> hpk: I cat get rid of 95 percent of setfastscope functions by caching. Where should Iput the cache?
[22:04] <stakkars_> is it ok to just keep a cache in the class?
[22:05] <hpk> first, it can't hurt to use pypy.tool.cache.Cache
[22:05] <hpk> which gets "frozen" upon translation
[22:05] <hpk> we usually do that for caches
[22:06] <hpk> second i think it's ok to keep it on the class
[22:06] <hpk> actually i did a similar thing yesterday (otherwise i wouldn't have gotten to the 50% figure :-)
[22:06] <hpk> but that was just a rough hack
[22:06] <hpk> feel free to check yours hin
[22:06] <stakkars_> this is a simple case, because the bodies don't contain names.
[22:07] <hpk> i am not exactly sure what you mean but i'll read the diff
[22:07] <stakkars_> I stumbled over it since I spent so much time compiling this huge pile of junk. ;-)
[22:07] <stakkars_> what gives me more headaches is
[22:07] <hpk> do i have any chance with 1GB of RAM, btw?
[22:07] <stakkars_> the 470 run instances. Tjey *have* names in them.
[22:08] <stakkars_> it worked for me with 1 512MB, but quite some swapping.on 1 GB it is quite ok if you use tcc
[22:08] pedronis (~Samuele_P@c-1e8b70d5.022-54-67626719.cust.bredbandsbolaget.se) joined #pypy.
[22:08] <stakkars_> then the slow part is the generating,no longer the compiling.
[22:09] <stakkars_> so how can I change "run" to pick its parameters somewhere else, not compiling a func name in?
[22:09] <hpk> well be careful, some things are specialized for the purpose of the annotator
[22:10] <stakkars_> ouw.
[22:10] <hpk> actually i argued with Armin and Samuele about this code place already
[22:11] <stakkars_> ifso, then it would be great if there were a comment line telling that. "Don't change, or you break XXX"
[22:11] <hpk> but we didn't really discuss how to reduce overall code size
[22:12] <stakkars_> no, sure. These are just so obvious violations of "what like to look at when debugging" that I want to to vanish :-)
[22:14] <pedronis> we can probably unify all the run and setfastscope
[22:14] <pedronis> for the same set of types
[22:14] <pedronis> but indeed they are there for the annotator
[22:14] <stakkars_> then run needs to not to compile the func name in.
[22:15] <pedronis> but is delicate thing
[22:15] <hpk> stakkars_: what exactly do you mean, the BuiltinFrame_for_%s ?
[22:15] <stakkars_> oh,does the annotator need different, disjoint functions, or a special format?
[22:15] <stakkars_> hpk: yes, exactly
[22:15] <pedronis> it's needs disjoint functions
[22:15] <pedronis> for disjoint types
[22:16] <stakkars_> aha! Ok, but this doesn't mean that they all need these clumsy bodies,but could call something common?
[22:17] <pedronis> you want
[22:17] <stakkars_> no problem with tiny stubs. But here I get tons and tons of exception handlers.
[22:17] <pedronis> disjoint paths
[22:17] <pedronis> for disjoint types
[22:17] <pedronis> no common code for disjoint types
[22:17] <stakkars_> ???
[22:18] <stakkars_> source = """if 1:
[22:18] <stakkars_> def run(self):
[22:18] <stakkars_> try:
[22:18] <stakkars_> w_result = func(%s)
[22:18] <stakkars_> except KeyboardInterrupt:
[22:18] <stakkars_> raise OperationError(self.space.w_KeyboardInterrupt, self.space.w_None)
[22:18] <stakkars_> except MemoryError:
[22:18] <stakkars_> raise OperationError(self.space.w_MemoryError, self.space.w_None)
[22:18] <stakkars_> except RuntimeError, e:
[22:18] <stakkars_> raise OperationError(self.space.w_RuntimeError,
[22:18] <stakkars_> self.space.wrap("internal error" + str(e)))
[22:18] <stakkars_> if w_result is None:
[22:18] <stakkars_> w_result = self.space.w_None
[22:18] <stakkars_> return w_result
[22:18] <stakkars_> \n""" % ','.join(self.run_args)
[22:18] <stakkars_> the only interesting stuff here is the w_result = func(%s)
[22:18] <stakkars_> why do we need to repeat the rest a bazillion times?
[22:18] <hpk> i guess we can all see this code at our local side :-)
[22:18] <hpk> we could possibly hide the generic exception handling stuff
[22:19] <hpk> that was introduced to satisfy cpython's regrtests
[22:19] <pedronis> you could have a _run
[22:19] <stakkars_> sorry, I forgot that we get logged
[22:19] <pedronis> the just does
[22:19] <hpk> i am not concerned about logging
[22:19] <pedronis> the w_result = func(%s)
[22:19] <hpk> just that it blocks the channel for some time (because they have a feature which inhibits more than 5 lines of stuff at a time or so)
[22:20] <hpk> so you paste it in quickly but it takes like 20 seconds or so to arrive here line by line
[22:20] <stakkars_> didn't know that. ui
[22:21] <stakkars_> pedronis: yes, that's what I thought. You need different paths, but they may call common code.
[22:21] <hpk> yes
[22:25] <stakkars_> is it exactly the "run" that must be disjoint for annotator? Then I still don't see how.
[22:26] <hpk> except Exception, e: self._except(e)
[22:26] <pedronis> no, it not run
[22:26] <pedronis> it's the call that calls func
[22:26] <pedronis> and access
[22:26] <pedronis> class attributes
[22:27] <stakkars_> you mean, if there is a general run(), that finally calls the special _run somewhere, I'm fine?
[22:27] <pedronis> yes
[22:27] <pedronis> that means run
[22:27] <pedronis> goes on the common superclass
[22:27] <pedronis> and _run
[22:28] <pedronis> goes on the specialized
[22:28] <pedronis> subclasses
[22:28] <stakkars_> great. then _run can be as small as possible.
[22:28] <pedronis> basically I suppose
[22:28] <pedronis> return func(%s)
[22:29] <hpk> but couldn't the generated classes still be reused some more?
[22:29] <pedronis> yes
[22:29] <pedronis> we could try
[22:29] <pedronis> generate one subclass
[22:29] <pedronis> per unwrap_spec pattern
[22:29] <hpk> makes sense
[22:30] <stakkars_> but wouldn't that have the same effect as if I cache the generated functions?
[22:31] <stakkars_> well, the same folding as with setfastscope I meant.
[22:31] <pedronis> what folding?
[22:32] <pedronis> you cannot fold based only on source text
[22:33] <pedronis> because some attribute names
[22:33] <pedronis> don't depend on types
[22:33] <stakkars_> ok, you were not here. I proposed to cache the source text and re-use compiled functions:
[22:33] <stakkars_> : hpk: I cat get rid of 95 percent of setfastscope functions by caching. Where should Iput the cache?
[22:33] <stakkars_> [21:00] stakkars: is it ok to just keep a cache in the class?
[22:34] <pedronis> and I say you cannot
[22:34] <pedronis> fould based on text
[22:34] <pedronis> because in some cases
[22:34] <pedronis> things dealing with different types
[22:34] <pedronis> may get the same text
[22:34] <pedronis> you really need to consider the unwrap_spec
[22:35] <stakkars_> then maybe the annotator should be tought to do the same, instead of relying on code duplication?
[22:36] <pedronis> maybe
[22:36] <pedronis> but it is not trivial at all
[22:36] <pedronis> to write logic that figures this out automatically
[22:36] <stakkars_> the unwrap_spec is the essence here. Yes, I agree.
[22:37] <stakkars_> well, I don't understand the annotator, yet. I thought it would splitbranches anyway when different types appear.
[22:38] <hpk> no, it only does that when you explicitely ask for it (and there are only a few ways to specifiy this)
[22:38] <hpk> it does "specialization" (=generating different versions) by argtypes and by calling location IIRC
[22:39] <hpk> but in the BuiltinFrame-subclass case it would need to do that by instance variables or something
[22:39] <stakkars_> maybe this would be a fine use case.
[22:41] <stakkars_> ahh. It is tied to the function, but not the code itself? Too bad that I cannot split them apart, sharing the code.
[22:41] <pedronis> but the fact is that we want real special code anyway
[22:41] <pedronis> because we want the typechecking that setfastscope does
[22:42] <pedronis> before that
[22:42] <pedronis> not only the annotator was unhappy
[22:42] <pedronis> but we were also leaving out typechecks
[22:42] <pedronis> that would avoid crashing PyPy
[22:43] <stakkars_> pedronis: ok. But pleasedo me the favor andput this line at gateway:248
[22:43] <stakkars_> print 70*"#"##!!
[22:43] <stakkars_> print setfastscope
[22:44] <stakkars_> and then run pypy. Then you will want the same.
[22:44] <pedronis> stakkars: as I said we
[22:44] <pedronis> can merge stuff based on the unwrap_spec
[22:44] <pedronis> I can do that
[22:44] <hpk> i think this will reduce things quite a lot
[22:45] esteban (~esteban@DWM-55-230.go.retevision.es) joined #pypy.
[22:45] <stakkars_> you think that doesn't do many duplicates, because the unwrap_specs are hopefully also similar?
[22:46] <stakkars_> yes, then please do that. 470 functions with the same name - arghh
[22:47] <stakkars_> or hey, if possibloe, this would be much nicer, if it can be managed that they include their unwrap_spec in a way. Then you can understand the generated code a little.
[22:47] <stakkars_> I mean in the name of the function.
[22:47] <pedronis> yes
[22:47] <pedronis> that should be possible
[22:48] <stakkars_> (th is also a thing that I tried in geninterp and will include into genc: I try to get the class names of methods by explicitly fetching the method, not using the dict. Gives much nicer names)
[22:49] <stakkars_> (that is, I inspect the dict of something for names, but then fetch them by getattr(something, name) )
[22:50] <stakkars_> yes, OT
[22:50] <stakkars_> have to go home
[22:50] Action: stakkars_ is leaving work
[00:00] --- Thu Feb 24 2005