All times shown according to UTC.
| Time | Nick | Message |
|---|---|---|
| 01:48 | ajonas joined #miro-hackers | |
| 01:59 | ajonas left #miro-hackers | |
| 02:38 | CarlFK left #miro-hackers | |
| 02:43 | willkg left #miro-hackers | |
| 02:46 | DGMurdock left #miro-hackers | |
| 08:32 | morganknutson_ left #miro-hackers | |
| 09:22 | maggie_s joined #miro-hackers | |
| 09:43 | maggie_s_ joined #miro-hackers | |
| 09:45 | maggie_s left #miro-hackers | |
| 09:45 | maggie_s_ is now known as maggie_s | |
| 10:08 | maggie_s left #miro-hackers | |
| 10:09 | maggie_s joined #miro-hackers | |
| 10:18 | maggie_s left #miro-hackers | |
| 10:18 | maggie_s joined #miro-hackers | |
| 10:37 | CarlFK joined #miro-hackers | |
| 11:58 | maggie_s left #miro-hackers | |
| 12:28 | arpu left #miro-hackers | |
| 12:48 | willkg joined #miro-hackers | |
| 12:54 | willkg | hi! |
| 12:55 | funky. because bugzilla was down, i didn't get any bugzilla email. | |
| 12:56 | it felt nice. i kind of liked it. | |
| 13:51 | z3p | good morning |
| 13:56 | neat, we lost some bugs during the downtime | |
| 13:58 | willkg | yeah. i'm not excited about that. but whatever. |
| 13:59 | rob restored from a backup. | |
| 14:42 | paulproteus | z3p: It seems to me that the slow thing with doing bulk_import and other thumbnail-heavy operations is the time we spend blocking on remote HTTP servers while downloading thumbnails. |
| 14:42 | z3p | paulproteus: I would certainly believe that |
| 14:42 | paulproteus | I guess I came up with a super hacky way to do this "fast", which is to do the operation in two runs: first, for all sites, get a list of all URLs we need to fetch, and pass those off to a background job that's doing the fetching for all sites. |
| 14:43 | The background job is a process that is always running, say, and its sole purpose is to put things in the httlib2 cache. | |
| 14:43 | z3p | paulproteus: the first step is easy enough; don't we keep track of the thumbnail URL? |
| 14:43 | paulproteus | We do, exactly. |
| 14:43 | Then run bulk_import etc., but insist that the Python code this time is only permitted to look in the httplib2 cache. | |
| 14:44 | z3p | paulproteus: you can probably do it even easier |
| 14:45 | * paulproteus | is trying to think of the simplest thing that could possibly work. |
| 14:45 | z3p | paulproteus: first time through, don't run save_thumbnail(), but do save the thumbnail_url |
| 14:45 | paulproteus | "save it" to where, this list of URLs that need to be fetched? |
| 14:45 | z3p | paulproteus: on the Video object |
| 14:45 | paulproteus | Right, well, of course. |
| 14:45 | Okay, sure. | |
| 14:46 | z3p | paulproteus: then save_thumbnail() on each Video.objects.filter(has_thumbnail=False).exclude(thumbnail_url='') |
| 14:46 | paulproteus: it'll back-load all the waiting | |
| 14:46 | paulproteus | z3p: Well, sure, but that doesn't permit us to speed up the total time to execute. |
| 14:47 | Because we don't do parallel URL fetching. | |
| 14:47 | z3p | or, don't even do it as part of bulk-whatever |
| 14:47 | paulproteus | I was under the impression that it was a non-permissible optimization to create Video objects the user sees that don't have thumbnails yet. |
| 14:48 | Your way seems like it would 1) Create Video objects we can't show the user, then 2) be just as slow to fetch the thumbnails. | |
| 14:48 | Admittedly in step 2 you could speed it up by doing save_thumbnail plus parallel HTTP fetching. | |
| 14:48 | z3p | paulproteus: right, you can speed up the second part by using save_thumbnail_from_file() |
| 14:49 | paulproteus | Yeah, but at which point in your story do you actually do the HTTP GET? |
| 14:50 | z3p | paulproteus: after we know all the thumbnail_urls |
| 14:50 | paulproteus | And you asynchronously do GETs, and pass them into save_thumbnail_from_file (which I realize can take a ContentFile, and you'd wrap up the image in that)? |
| 14:51 | z3p | paulproteus: right |
| 14:51 | paulproteus: you could even start the GETs before the full import is done, with a queue or something | |
| 14:51 | paulproteus | Yeah... I think that doing it in-process seems to make sense, rather than extracting out the HTTP getting to a different process. |
| 14:51 | z3p | two threads |
| 14:52 | paulproteus | It's just a bit of a sad story to do it that way, 'cause some of our cron jobs run for months for approximately unknown resaons but I presume it's due to blocking on some silly remote network socket. |
| 14:52 | z3p | one for the thumbnail GETs, one to do the rest of the imports |
| 14:52 | yeah, they should probably timeout eventually | |
| 14:52 | paulproteus | Well, I like this way well enough. |
| 14:53 | I'll see if I can do it to bulk_import right now *gulp* | |
| 15:07 | ajonas joined #miro-hackers | |
| 15:25 | paulproteus | z3p: Is there a Django-ish way to say, "Save all these objects at right now", like sqlalchemy has? |
| 15:25 | That way I can avoid some somewhat expensive v.save() calls until the end, too. | |
| 15:26 | z3p | paulproteus: what do you mean? |
| 15:26 | like, take a list of objects and save them all? | |
| 15:26 | paulproteus | Yeah. |
| 15:26 | To effectively batch all the SQL. | |
| 15:26 | I guess I could pretend I'm doing that by using a transaction. | |
| 15:26 | z3p | paulproteus: yeah, you can fake it with a transaction |
| 15:27 | paulproteus | Fair enough. |
| 15:27 | Is there a non-fake way? Just curious. | |
| 15:27 | z3p | paulproteus: not that I, or a quick google search, know of |
| 15:27 | paulproteus | b |
| 15:27 | Me too, then. | |
| 15:31 | zanoi | could someone please try to reproduce bug 17057 |
| 15:51 | kcw | zanoi: is there a bug for the change in e414496? |
| 15:53 | zanoi | yes, i did it as part of the upcoming list view changes, but thought it was a more general bug. Sorry should have written the bug number. |
| 15:54 | kcw | ok. I don't see anything about date there - I'm just wondering whether there's a reason it needs to be so wide |
| 15:54 | zanoi | because the date column seems to be too small to display dates, but it was only noticeable when the date column is not to the rightmost column |
| 15:55 | but since the column order will have to be changed in the list view bug this needed fixing | |
| 15:55 | kcw | Is column width acting differently for the rightmost column? |
| 15:55 | zanoi | i don't know. It seemed like it to me, but I could be wrong |
| 15:56 | also i made it a bit wider than the english language date display because many other languages are longer than english | |
| 15:56 | all i know is that when i changed the column order the date column was too small | |
| 15:58 | kcw | Did you change the column order by resetting your db? That would cause column widths to be reassigned |
| 15:58 | zanoi | yes |
| 16:01 | though we should really adjust the date column size according to the column content. Since dates will have different sizes in each language | |
| 16:02 | paulproteus | Okay, I am repeatedly surprised by what I see in the profiler for bulk_edit. |
| 16:02 | Going to run a big import job on pcf-mcdev (floyd-dev) with a big YouTube video now, and record the entire profile run. | |
| 16:03 | It takes 3 sec per video. | |
| 16:03 | That's a long time. | |
| 16:04 | zanoi | kcw, is there any rule I can follow when I modify the columns without risking breaking stuff? |
| 16:08 | kcw | what kind of modifications? adding or removing columns the list at the top of http://develop.participatorycu[…]php/4.1/List_View has what needs to be changed, but reordering or otherwise modifying the existing columns is generally safe |
| 16:09 | paulproteus | afk a bit for food. |
| 16:09 | Also, vidscraper's opensearch.py is where a whole bunch of time is going. | |
| 16:09 | So much to optimize. | |
| 16:10 | But this looks like a good first one, as here it takes 50% of the runtime. | |
| 16:12 | zanoi | kcw, thanks, that's what I wanted to know. that's also the way I've been handling it until now |
| 16:17 | kcw, also, do you know which list views the changes are meant for? just the podcast one? | |
| 16:18 | ajonas | paulproteus: yay optimization! |
| 16:18 | kcw | zanoi: all list views |
| 16:19 | zanoi | but surely we don't want the music list view to look like that? |
| 16:24 | for instance almost all the columns talked about in the bug don't make sense for music | |
| 16:28 | kcw, ^ | |
| 16:28 | kcw, btw feel free to revert/change the date stuff if you have a better idea | |
| 16:31 | kcw | zanoi: I'm not sure what the question is; a lot of these changes will affect specific columns, wherever those columns are displaying |
| 16:40 | zanoi | kcw, well it also talks about removing, reordering and maybe adding columns. So does that only apply to those views where the columns are present already? |
| 16:44 | kcw | zanoi: are there any column changes other than "buttons and status should be in the same column and on the far right" ? (now bug http://bugzilla.pculture.org/s[…]bug.cgi?id=17094) |
| 16:50 | zanoi | it also says "badges should be in the right hand column", but not sure what he means with that and the mockup has a completely different column order, or was i supposed to ignore the mockup? |
| 16:51 | also I didn't know moving the buttons has its own bug. so I already moved the download button to the status column | |
| 16:53 | kcw | I just made the separate bugs; like I said by email, we're both working on this so lets separate it out into individual bugs so that we can tell who's working on what |
| 16:53 | I think "badges should be in the right hand column" refers to moving the status indicators into the new stuck-to-the-right-side column | |
| 16:54 | zanoi | right. Just noticed all your bugs :-) |
| 16:57 | kcw, but it doesn't seem to be like that in the mockup? or am i misunderstanding it? | |
| 17:02 | kcw | I see what you mean. I don't know what that statement means either. Let's ask on the bug. |
| 17:26 | paulproteus | z3p: It seems we ignore the "etag" field on "class Feed". Is that what you recall, too? |
| 17:27 | z3p | paulproteus: we send it to feedparser |
| 17:28 | paulproteus | I think that's false, but I'll check. |
| 17:28 | z3p | paulproteus: it's in _update_items_generator() |
| 17:28 | paulproteus | Ah hah, there 'tis |
| 17:29 | This is incoherent for joined-up feeds that bulk_import creates, fwiw. | |
| 17:29 | z3p | paulproteus: oh? |
| 17:29 | paulproteus | I mean, we use the .etag from the first feed. |
| 17:29 | I guess that's approximately maybe probably going to work. | |
| 17:29 | I guess it probably is, actually. | |
| 17:29 | It's still pretty odd in my head. | |
| 17:30 | z3p | paulproteus: worst thing that happens if it doesn't work is that the etag doesn't match |
| 17:30 | I think the odds of mistakenly matching the etag are low | |
| 17:30 | paulproteus | Worst thing is the etag does match, and so feedparser decides to return no data (?) |
| 17:30 | Yeah, agreed. | |
| 17:30 | I really should have been committing more often than I have been. | |
| 17:30 | But anyway. | |
| 17:30 | z3p | paulproteus: FYI, requirements.txt is totally borked ATM |
| 17:31 | paulproteus | That's a shame. |
| 17:32 | I slightly knew that, I admit. | |
| 17:44 | kcw | zanoi: ah, I didn't realize we were using long dates in some places - the displays I was looking at just had (mm/dd/yyyy) type dates. Now I see why it's 150px |
| 17:45 | paulproteus | z3p: I'm refactoring that giant function so that it loops over feed_entry objects and passes the feed_entry object to a (slightly) smaller method. |
| 17:45 | z3p | paulproteus: sounds good! |
| 17:45 | zanoi | kcw, ah ok |
| 17:45 | paulproteus | This is so I can pass entries in directly. Question for you: I see ' yield {'index': index, ' on a line -- what is the yield really for? |
| 17:46 | zanoi | kcw, it might be a better solution to only use the short dates because i think they will have somewhat the same length no matter the localization |
| 17:46 | and then we could revert the date size | |
| 17:47 | z3p | paulproteus: see the bulk_import code |
| 17:47 | paulproteus | Oh I see. |
| 17:48 | z3p | you could also do it by building a huge list, I suppose |
| 17:48 | kcw | zanoi: I like that. Filing a design/behavior bug for date formats |
| 17:55 | zanoi: filed it here: http://bugzilla.pculture.org/s[…]_bug.cgi?id=17106 | |
| 17:55 | zanoi | great, thx |
| 18:00 | z3p | paulproteus: is localtv.SiteLocation.enforce_tiers() backwards? |
| 18:01 | paulproteus | I *think* it is not backwards. |
| 18:01 | I have squinted at that function a lot. | |
| 18:01 | z3p | it seems like it's checking paypal when DISABLE is True, not when DISABLE is False |
| 18:01 | paulproteus | It sort of "undoes" the double-negation inherent in the name of the configuration variable. |
| 18:01 | Let me go reread it. | |
| 18:02 | The PayPal thing -- even if DISABLE is true, if you have been through a PayPal transaction, we suddenly enforce. | |
| 18:02 | So user actions end up overriding the configuration. I hope that makes "sense". | |
| 18:03 | z3p | paulproteus: hmm, okay |
| 18:05 | zanoi | bbl, heading home from café |
| 18:08 | z3p | paulproteus: ah, I figured it out; pre_save_adjust_resource_usage needed an extra check |
| 18:09 | paulproteus | I'd be interested to see the patch, but I think this is quite plausible. |
| 18:11 | z3p | paulproteus: or re-arranging the checks; enforce_tiers() fails if there isn't a SiteLocation() around |
| 18:12 | paulproteus | Right, of course. |
| 18:12 | * paulproteus | rolls his eyes. |
| 18:12 | z3p | paulproteus: https://gist.github.com/937261 |
| 18:13 | look okay? | |
| 18:13 | paulproteus | Duh. Looks fantastic, z3p. |
| 19:39 | z3p | paulproteus: what do you think about https://gist.github.com/937424 ? |
| 19:40 | paulproteus: I'm heading out the door, but if it looks okay I'll check it in and add the second cron job | |
| 19:40 | (when I return) | |
| 19:40 | paulproteus | z3p: Rock it. |
| 19:48 | DGMurdockIII joined #miro-hackers | |
| 20:03 | DGMurdockIII left #miro-hackers | |
| 20:03 | DGMurdockIII joined #miro-hackers | |
| 21:03 | mattjohnson_ joined #miro-hackers | |
| 21:03 | mattjohnson left #miro-hackers | |
| 21:03 | mattjohnson_ is now known as mattjohnson | |
| 21:03 | glee left #miro-hackers | |
| 21:25 | willkg | weird. there's no pstats module in python 2.6 on ubuntu maverick. |
| 21:25 | anyone know if that's in a different package or something like that? | |
| 21:29 | ah... it's in python-profiler. | |
| 21:29 | stupid. | |
| 21:31 | ajonas left #miro-hackers | |
| 21:32 | mattjohnson left #miro-hackers | |
| 21:35 | mattjohnson joined #miro-hackers | |
| 21:40 | bendk joined #miro-hackers | |
| 21:40 | bendk | hi all |
| 21:40 | willkg | bendk: hi! |
| 21:40 | bendk | sorry I've been gone away these last 2 days, had some family issues |
| 21:40 | willkg | ahh... sucky. |
| 21:40 | bendk | hey willkg I fixed half of the failed unittests, but I'm unsure about the moviedata one |
| 21:40 | I know it was passing for me when I checked it in | |
| 21:40 | willkg | yay for fixing half of the failed ones! |
| 21:40 | bendk | maybe it got broke later on, maybe it was just working on my system |
| 21:40 | I'm on my macbook right now though, so I can't test it | |
| 21:41 | willkg | want me to get your changes and run the tests? |
| 21:41 | bendk | it's checked into master |
| 21:41 | if you could run them that'd be awesome | |
| 21:42 | on the good news, I think I have a handle on http://bugzilla.pculture.org/s[…]_bug.cgi?id=17059, which I was pretty scared about | |
| 21:42 | willkg | i'm running them right now. |
| 21:42 | bendk: that's awesome! | |
| 21:43 | bendk: that one looks pretty crazy to me, too. and i'm pretty sure janet's seen things like that over the years, too. | |
| 21:45 | bendk: http://dpaste.com/534789/ | |
| 21:46 | bendk: the read_metadata TypeError is probably easy to fix. maybe that fixes the other ones, too? | |
| 21:57 | z3p | paulproteus: was the 'Rock it!' a 'looks good'? |
| 21:58 | bendk | duh |
| 21:58 | sorry will that one is totally my faiult | |
| 22:00 | okay, I know exactly what's happening for the first one, and I have an idea for the other two. I need to go home to fix them though | |
| 22:02 | maggie_s joined #miro-hackers | |
| 22:05 | willkg | bendk: ok. i'm going to grab some dinner, but pingme if you need me again. |
| 22:10 | DGMurdockIII left #miro-hackers | |
| 22:16 | bendk left #miro-hackers | |
| 22:21 | maggie_s_ joined #miro-hackers | |
| 22:22 | maggie_s left #miro-hackers | |
| 22:22 | maggie_s_ is now known as maggie_s | |
| 22:26 | maggie_s left #miro-hackers | |
| 22:26 | maggie_s joined #miro-hackers | |
| 22:41 | maggie_s left #miro-hackers | |
| 22:43 | maggie_s joined #miro-hackers | |
| 22:44 | bendk joined #miro-hackers | |
| 22:45 | bendk | willkg: you around? |
| 22:45 | willkg | bendk: yup. |
| 22:45 | bendk | I'm back home now and working on the tests |
| 22:46 | the metadata one I can fix the error I introduced, although it's still failing on my machine | |
| 22:46 | the other 2 I can't reproduce at all | |
| 22:46 | willkg | bendk: mmm... did you try doing a second clone? is it a missing file? |
| 22:46 | bendk | can I connect to your automated testing box and try it there? |
| 22:47 | willkg | bendk: are they failing on the build box? if so, then connect there. if not, it'll take me a while to reconfigure my world to allow you to connect to my laptop. |
| 22:47 | bendk | I just did a fresh pull —rebase and I don't see any extra files |
| 22:47 | let me try the build box | |
| 22:47 | willkg | bendk: amongst other things the AssertionError is pretty useless. so it'd help to fix those lines with some useful error message. like what the assertion values are. |
| 22:48 | paulproteus | z3p: Yes |
| 22:48 | maggie_s_ joined #miro-hackers | |
| 22:49 | maggie_s left #miro-hackers | |
| 22:49 | maggie_s_ is now known as maggie_s | |
| 22:49 | willkg | bendk: also, that test looks weird. in either case (should_run == True or False), the same assertion is run. that's weird. |
| 22:49 | oh, wait. i missed a not. nevermind. | |
| 22:58 | bendk | argh, they pass on the build box as well |
| 22:58 | maybe it's a dependancy issue? | |
| 22:59 | and interestingly enough, the metadata test pass on the build box, but not my computer | |
| 22:59 | z3p | paulproteus: awesome; I'll push it up now |
| 23:02 | bendk | I just pushed a chageset that adds more info when the tests fail |
| 23:02 | can you run again and tell me what's outputted? | |
| 23:02 | willkg: ^^ | |
| 23:04 | willkg | bendk: just did a fetch and now i'm running it. |
| 23:05 | bendk: i made sure to run ./clean.sh this time around. maybe that was part of the problem before. | |
| 23:08 | bendk: http://dpaste.com/534809/ | |
| 23:08 | bendk: that's on my machine again. | |
| 23:09 | also, +1 on using dean.jpg. whoever added that test gets a gold star. | |
| 23:09 | bendk | thank you very much! |
| 23:10 | willkg | woah! dean.jpg was added ages ago! |
| 23:10 | we should make it a coding standard that all test data must involve dean in some way. | |
| 23:14 | afk a bit to make muffins. | |
| 23:20 | paulproteus | z3p: From RDS read I/O it looks like you're doing a migration (-: |
| 23:20 | z3p | paulproteus: making sure I don't totally hose anything before doing it on the live site |
| 23:20 | paulproteus | (-: |
| 23:20 | z3p | so it'll be doing it again later on |
| 23:29 | paulproteus | z3p: skip_constraints=True # or similar |
| 23:29 | I have to run, but look at other migrations for what I mean | |
| 23:29 | and/or git log -p -Sconstraints | |
| 23:30 | z3p | paulproteus: will do, thanks! |
| 23:43 | bendk | willkg: any results? |
| 23:45 | still making muffins huh? | |
| 23:45 | can you email it to me, I have to run | |
| 23:45 | thanks for all the help! | |
| 23:45 | bendk left #miro-hackers |