All times shown according to UTC.
| Time | Nick | Message |
|---|---|---|
| 03:55 | maggie_s joined #miro-hackers | |
| 04:09 | maggie_s_ joined #miro-hackers | |
| 04:14 | CarlFK joined #miro-hackers | |
| 06:37 | Jarrhead joined #miro-hackers | |
| 08:44 | Jarrhead joined #miro-hackers | |
| 08:51 | janet joined #miro-hackers | |
| 08:57 | Odysimus joined #miro-hackers | |
| 10:53 | Jarrhead joined #miro-hackers | |
| 12:57 | CarlFK joined #miro-hackers | |
| 14:10 | willkg joined #miro-hackers | |
| 14:11 | willkg | hi! |
| 14:45 | Jarrhead joined #miro-hackers | |
| 15:17 | ajonas joined #miro-hackers | |
| 15:35 | bendk | hey willkg |
| 15:35 | glee1: you around? | |
| 15:52 | glee1 | bendk: yep what's up? |
| 15:52 | bendk | so I had some ideas about moving more stuff to the worker process and I wanted to bounce them off you |
| 15:52 | first off, I definitely want to move mutagen there | |
| 15:52 | glee1 | bendk: sure thing, go ahead |
| 15:52 | bendk | and also, maybe it would make sense to move the all_files() calls there |
| 15:53 | try to unload as much as possible of the stuff that we have in idle_iterate | |
| 15:53 | then probably have a priority system, so that all_files() goes ahead in the queue from metadata queries | |
| 15:54 | and maybe even categorize the tasks into IO-bound and CPU-bound and try to run them in parallel | |
| 15:54 | do you think that makes sense? | |
| 15:56 | glee1 | bendk: first I think it'd be great to move long running tasks outside of the eventloop, then, in this case, it's not a matter of whether it is cpu-bound or i/o bound, because any long running task would block the program for a long time, because we run things inline and don't dispatch it elsewhere |
| 15:58 | I mean, for some stuff, like miro_allfiles(), I expect less of a program and you can probably run it in a separate thread, though it would not hurt to run inside a separate address space. Probably mostly i/o bound and it will run outside of the python GIL anyway | |
| 15:58 | bendk | yeah, moving it to another process might be overkill |
| 15:59 | I was excited about moving mutagen there and also echoprint when I added it | |
| 15:59 | I got to thinking of other possibilities and that one popped in my head | |
| 16:00 | the IO-bound vs CPU-bound distinction is just because if we make it so that the worker process could get backed up, then we could run one of each type of task in a separate through to try to maximize the amount of work we get done. | |
| 16:01 | glee1 | bendk: So, in this scheme, you would have 2 worker processes, one for cpu intensive tasks and the other for i/o bound tasks? |
| 16:02 | bendk | I was thinking 1 process, but 2 threads inside of it |
| 16:03 | glee1 | interesting, if that's the case why not just have a thead pool? Because in the worker process you no longer need to care about interactivity only throughput |
| 16:04 | bendk | and I guess a third thread that "schedules" stuff. It would take tasks of the Queue and push them to the worker threads |
| 16:05 | I guess I was thinking of something like a thread poll, but it wouldn't make sense to be running too many mutagen tasks at once, you'd just be spinning the hard drive for no reason | |
| 16:06 | So maybe a thread pool, but we would limit the number of threads that worked on mutagen and other disk intensive tasks | |
| 16:06 | glee1 | disk i/o will happen regardless, e.g. other applications running on top of your operating system, or even the operating system itself e.g. writing into and out of swap space. Probably we can just limit the number of active tasks and then be don with it |
| 16:07 | It's possible that being clever won't do too much what you expect either, due to the underlying implementation of the operating system scheduler | |
| 16:07 | bendk | Okay, I'm convinced |
| 16:09 | So a simple thread pool + a scheduler thread and give each task a priority | |
| 16:09 | I think that parts important because if we can queue up like 10,000 mutagen tasks. If we get a feedparser task in the middle of that we want to run that one as soon as we can | |
| 16:10 | janet | ben glee1 i think this bug is fallout from recent changes: http://bugzilla.pculture.org/s[…]_bug.cgi?id=18508 |
| 16:10 | I think this is a dup http://bugzilla.pculture.org/s[…]_bug.cgi?id=18512 | |
| 16:11 | and 1 more crash I just got similar but related to subrpocess something or another | |
| 16:11 | also - I think related to recent changes is this bug: http://bugzilla.pculture.org/s[…]_bug.cgi?id=18510 | |
| 16:12 | where the opml imports (they all update at the same time) just got stuck in update mode | |
| 16:12 | glee1 | janet: only changed watched folders, so I'm not sure if #18510 was my doing, but I can take a look |
| 16:13 | janet: Ben is going to do some work on subprocess too so maybe it'd be good to talk to him too | |
| 16:13 | janet: the other one I'm pretty sure you are right it is a fallout | |
| 16:15 | janet: | |
| 16:16 | janet | ok |
| 16:16 | glee1 | janet: I'll go and patch it up |
| 16:17 | janet | thanks |
| 16:18 | janet left #miro-hackers | |
| 16:22 | glee1 | z3p, bendk: can you have a look at this guy: https://github.com/pculture/miro/pull/94 |
| 16:23 | GitHub75 joined #miro-hackers | |
| 16:23 | GitHub75 | [miro] bendk pushed 2 new commits to master: https://github.com/pculture/mi[…]916f730...0fc2515 |
| 16:23 | [miro/master] bz18510: Make sure that mediatype is unicode. - Geoffrey Lee | |
| 16:23 | [miro/master] Merge pull request #94 from geoffl/bz18510 - bendk | |
| 16:23 | GitHub75 left #miro-hackers | |
| 16:59 | CarlFK joined #miro-hackers | |
| 18:28 | rikai joined #miro-hackers | |
| 18:28 | rikai joined #miro-hackers | |
| 18:59 | janet joined #miro-hackers | |
| 19:16 | CarlFK joined #miro-hackers | |
| 19:20 | rikai joined #miro-hackers | |
| 19:27 | rikai joined #miro-hackers | |
| 19:27 | rikai joined #miro-hackers | |
| 19:54 | janet joined #miro-hackers | |
| 21:35 | janet joined #miro-hackers | |
| 22:18 | maggie_s joined #miro-hackers | |
| 23:48 | janet | hey glee1 - are you around? |