All times shown according to UTC.
| Time | Nick | Message |
|---|---|---|
| 00:25 | paulproteus | deardiary: working on deploying some improvements to bulk_import |
| 00:25 | (to the dev servers, for now) | |
| 00:35 | You should never have to "Wiping your environment for a clean-slate" (but it's okay to have a section on it, but just do also say you should never have to) | |
| 02:58 | paroneayea: waaaahhhh, django-tagging is SO SLOW | |
| 02:58 | I think that, plus thumbnailing, are ruining my day w/r/t improving bulk_import performance | |
| 02:58 | And part of the sucky thing about thumbnails is that there's something slow about the way we're using S3. | |
| 02:59 | paroneayea | paulproteus: :( |
| 02:59 | paulproteus: time to ditch django-tagging in favor of something more custom? :O | |
| 03:00 | paulproteus | Well, or maybe <evil>I could do the tag saving in the background, in a separate job, but report success ... |
| 03:00 | ...actually one sec. | |
| 03:01 | ...okay, well, "good", the problem isn't that we're using MyISAM tables. | |
| 03:02 | I'm trying to use SQL transactions to simulate bulk INSERTs. | |
| 03:03 | paroneayea | paulproteus: obviously the answer here is to rewrite everything to use a NOSQL database! |
| 03:03 | * paroneayea | not helpful |
| 03:03 | paulproteus | This works well for most things, but for some reason django-tagging is still slow. |
| 03:03 | paroneayea | paulproteus: I think it's because it's using the GenericField field type |
| 03:03 | which does evil things, SQL wise | |
| 03:04 | paulproteus | Interesting, that does seem reasonably likely to be the culprit. |
| 03:04 | paroneayea | paulproteus: I remember using django-tagging initially when starting miro community |
| 03:05 | and at the time I was of the opinion that using things in a reusable and generic way is really great. But that was because of my previous work at imaginary landscape when we'd have any number of general in-house applications hooked together on one site | |
| 03:05 | miro community sites don't have that need | |
| 03:05 | there *are* no other django applications that need to share the same tagsw | |
| 03:05 | tags | |
| 03:05 | you're probably better off using your own tag table | |
| 03:05 | it'd be a lot faster. | |
| 03:06 | paulproteus | it's only the *saving* of the tags that's slow, it seems. |
| 03:06 | I guess that's to be expected. | |
| 03:06 | * paulproteus | is still investigating. |
| 03:07 | willkg | time for me to jet. see you all tomorrow! |
| 03:07 | willkg left #miro-hackers | |
| 03:07 | paulproteus | (bye willkg!) |
| 03:09 | This is "only" a problem for bulk_import, where it's a pain if a video takes 4 sec to import from a YouTube feed with 350 videos. | |
| 03:11 | paroneayea | https://bugzilla.mozilla.org/e[…]ages/mozchomp.gif I'm glad that bugzilla keeps the animated 3d gif dream alive. |
| 03:11 | paulproteus | (-: |
| 03:13 | I wish I could use greenlet to concurrently multiprocess everything. | |
| 03:18 | paroneayea: I'm extremely tempted to create a sub-thread and do the django-tagging work in there, so that I can do other operations like resizing thumbnails in parallel. | |
| 03:18 | OTOH if I can do resizing thumbnails in greenthread, doing ca. 5 at once, that would make more of a difference. | |
| 03:18 | Still, though, it's so tragic that it takes even 1 sec per video, AKA 5 min for a 300-video feed. | |
| 03:19 | Yeah, there's more to win by speeding up thumbnailing. | |
| 03:19 | I'll work on that next. | |
| 04:12 | jeremyb | whoa, wall of text |
| 04:13 | miro is django? | |
| 04:13 | or MC? | |
| 04:13 | (looking at django-tagging above) | |
| 04:14 | * jeremyb | will bbl (another day) |
| 04:22 | paulproteus | Miro Community is Django, indeed. |
| 06:02 | maggie_s_ joined #miro-hackers | |
| 06:04 | maggie_s left #miro-hackers | |
| 06:04 | maggie_s_ is now known as maggie_s | |
| 07:16 | CarlFK joined #miro-hackers | |
| 07:17 | Jarrhead joined #miro-hackers | |
| 10:01 | janetPCF joined #miro-hackers | |
| 10:10 | maggie_s left #miro-hackers | |
| 11:52 | arpu left #miro-hackers | |
| 12:08 | arpu joined #miro-hackers | |
| 12:18 | morganknutson_ left #miro-hackers | |
| 12:18 | willkg joined #miro-hackers | |
| 12:20 | arpu_ joined #miro-hackers | |
| 12:25 | arpu left #miro-hackers | |
| 12:35 | willkg | hi! |
| 12:53 | glee: 13530 is totally a feature. | |
| 12:53 | glee: it changes the way we're determining the language from the trackname/filename. | |
| 12:53 | glee: please just leave it alone and don't keep fiddling with it. | |
| 12:54 | glee | willkg: Fair enough. |
| 12:54 | willkg | also, since it's TED specific, i really want an example from TED. that's what i'm waiting on the original reporter for. |
| 12:54 | does that make sense? | |
| 12:55 | regardless, this is not a priority for getting a beta out tomorrow. | |
| 12:55 | we should be focusing on P1 bugs for 4.0 that are stability-related, testing, and doing what we can to make sure the beta doesn't crash on people. | |
| 12:56 | everything else gets deferred. | |
| 12:58 | * willkg | sighs. |
| 12:58 | willkg | glee: please please please please please leave that bug alone and stop doing things on it. |
| 13:03 | zanoi_ joined #miro-hackers | |
| 13:03 | arpu__ joined #miro-hackers | |
| 13:05 | arpu_ left #miro-hackers | |
| 13:05 | zanoi left #miro-hackers | |
| 13:05 | zanoi_ is now known as zanoi | |
| 13:09 | arpu__ left #miro-hackers | |
| 13:13 | willkg | afk for coffee and food while natty downloads.... then i'm going to do a big upgrade. |
| 13:21 | arpu__ joined #miro-hackers | |
| 13:46 | paulproteus | Hello everybody. |
| 14:18 | willkg | hi! |
| 14:23 | z3p left #miro-hackers | |
| 14:35 | z3p joined #miro-hackers | |
| 14:36 | z3p | good morning from the café |
| 14:37 | willkg | morning! |
| 14:37 | z3p: you're on maverick, right? | |
| 14:38 | and by "you're on maverick", i mean, "you have ubuntu maverick 10.10 installed on your computer". | |
| 14:38 | z3p | willkg: yes, I am still on 10.10 |
| 14:39 | willkg | z3p: natty released today. i'm going to upgrade either today, tomorrow or this weekend depending on how things go. |
| 14:39 | z3p: ben's on lucid. i figure if he stays on lucid, you stay on maverick, and i switch to natty, then we have all our bases covered. | |
| 14:39 | for some loose definition of "covered". | |
| 14:40 | z3p | willkg: yeah, I'm not upgrading until they fix click-to-raise |
| 14:40 | * willkg | nods. |
| 14:42 | z3p | deardiary: I don't think I've mentioned recently how much I hate windows |
| 14:45 | janetPCF | I currently have maverick and lucid |
| 14:46 | willkg | janetPCF: that's helpful, too. i'm thinking dev coverage because then we're guaranteed that someone is using it day after day. |
| 14:46 | it's definitely the case that all of us are using miro in different ways, so covering as many ways as possible is good. | |
| 14:49 | paulproteus | Like test coverage. |
| 14:50 | deardiary: Going to finalize the changes I made to parallel thumbnail downloading for bulk_import and then finally deploy on dev server. | |
| 14:50 | willkg | yes. unit tests are better with 4.0 than 3.5. still a bunch of things we can do better, but we'll get there. |
| 14:51 | and i'm pretty sure janet's testing stuff is also a lot better. | |
| 14:56 | ajonas joined #miro-hackers | |
| 15:14 | paulproteus | Morning, ajonas. |
| 15:14 | ajonas | paulproteus: *wave8 |
| 15:20 | * paulproteus | is reasonably happy with the state of bulk_edit now, and goes off to try it on the dev server. |
| 15:22 | willkg | z3p: when you say "false positive", what do you mean by that? |
| 15:22 | z3p: after that checkin, the video and music tabs _never_ get populated with anything. | |
| 15:22 | z3p: i can't tell if i'm not communicating the problem statement well enough or not. | |
| 15:22 | z3p | willkg: the reason that revision is fast is because it's only scanning like 5 files |
| 15:22 | willkg | z3p: this isn't a fast/slow issue. this is a "does stuff show up" or not issue. |
| 15:23 | Jarrhead left #miro-hackers | |
| 15:23 | z3p | willkg: okay, your comment has 'INSANELY FAST' in it, so I got confused; I'll keep looking into it |
| 15:24 | willkg | z3p: i only mentioned the speed stuff because that's the stuff ben was working on. |
| 15:24 | paulproteus | ajonas: Some facts about bulk_import: |
| 15:25 | 1. I think it works now (testing on dev server) | |
| 15:25 | 2. It takes 0.5-2sec per video | |
| 15:25 | When a feed has 377 items in it, that means 2-10 minutes | |
| 15:25 | If you go to e.g. http://floyd-dev.mirocommunity[…]8d30-1227ca712211 it says "Just a moment" rather than "oh my gosh this will take ten minutes!!" | |
| 15:25 | ajonas | paulproteus: once upon a time we used to show x of y videos imported |
| 15:26 | paulproteus: i liked that | |
| 15:26 | paulproteus | Yeah -- how long ago was that? It makes sense to me. |
| 15:26 | ajonas | paulproteus: may still be up, let's check |
| 15:26 | paulproteus | Check on a non-dev site, since the dev server has my newer stuff. |
| 15:27 | ajonas | paulproteus: *nod* i'm kind of shocked that it still takes that long - that seems worse than it used to be - but i think if it works and we properly communicate what's happening, that's enough for now |
| 15:27 | paulproteus | So it's far far worse for YouTube than any other service. |
| 15:27 | willkg | z3p: i added a comment clarifying the problem in that external drive bug. |
| 15:27 | paulproteus | Well, mostly because YouTube videos have tags. |
| 15:27 | The way we store tags is inefficient, adding 0.3 sec per video. | |
| 15:28 | z3p | willkg: great, thanks! |
| 15:28 | willkg | z3p: and like i said last night, i can do some more analysis on it. |
| 15:28 | paulproteus | Plus generating thumbnails takes about 2 sec. |
| 15:28 | Importing everything else takes about 0.1 sec. | |
| 15:28 | ajonas | paulproteus: still, i think something must have changed? because i don't think it used to take so long |
| 15:28 | paulproteus | (per vid) |
| 15:28 | z3p | willkg: I'm just finishing up the windows doesn't show devices bug, then I'm switching to that |
| 15:28 | ajonas | paulproteus: so we still say that we'll show periodic updates. i think we must have dropped those updates somewhere, though |
| 15:28 | paulproteus: by accident | |
| 15:29 | willkg | z3p: ok. i'll see what happens with my other external drive. that one is just media and has no symlinks. |
| 15:29 | paulproteus | I'll read the history of the Python code that serves the http://floyd-dev.mirocommunity[…]8d30-1227ca712211 page. |
| 15:30 | ajonas | paulproteus: i do get status updates on bigger feeds |
| 15:30 | paulproteus: in the form of "x videos imported" | |
| 15:30 | paulproteus: http://d.pr/2V1C | |
| 15:31 | paulproteus | b |
| 15:31 | Which site is that, out of curiosity/debugging-help? | |
| 15:31 | ajonas | paulproteus: they jump a lot, though - 29 to 37, etc. not sure if that's good or bad. but for now, i'm satisfied if it continues to do this and works |
| 15:32 | paulproteus: coffee | |
| 15:32 | paulproteus | Cool. |
| 15:34 | z3p: Got a sec? I don't yet see how localtv.admin.feed_views.add_feed_done seems to refresh periodically. | |
| 15:34 | z3p | paulproteus: a <meta refresh> in the template header |
| 15:34 | paulproteus | It seems to me it blocks on a single task, so it gets updates only once, at the end of that, unless I misunderstand. |
| 15:34 | Right, but how can the output of the page change? | |
| 15:35 | Won't it just report "Working..." then "Success! 12 videos imported" | |
| 15:35 | Rather than "Working..." "12 done!" "37 done!" "All done." | |
| 15:35 | z3p | paulproteus: let me pull up the code |
| 15:35 | paulproteus | Thanks, I appreciate it. |
| 15:36 | z3p: Figured it out!! | |
| 15:36 | It's in the template. Sorry 'bout that. | |
| 15:36 | z3p | paulproteus: oh good, I was about to explain it! |
| 15:36 | paulproteus: no worries | |
| 15:40 | * paulproteus | thinks for a sec. |
| 15:43 | paulproteus | For some reason, I find S3 to be the bane of my existence. |
| 15:44 | I see... you can't delete a thumbnail image that doesn't exist, maybe. | |
| 15:50 | Yeah, that's one bug here. | |
| 15:50 | z3p | willkg: you're talking about an external HD, with plates, right? |
| 15:50 | willkg | z3p: uh... what's a plate? |
| 15:50 | z3p | willkg: not a solid state drive |
| 15:51 | willkg | z3p: yes. one is a SATA drive and the other is an EIDE, i think. |
| 15:51 | z3p: both connected by USB. | |
| 15:51 | i wish i had an SSD. that'd be really cool. | |
| 15:51 | z3p | willkg: because even with a clean slate and 20k files, stuff shows up from my flash drive within a couple seconds |
| 15:52 | but it's also a FAT drive | |
| 15:52 | willkg | z3p: ok. sounds like i should do some more analysis, then. pass the bug back to me and i'll spend some time trying to figure out what's different about my drive. |
| 15:55 | deardiary: sent out email about the beta channel just now. going to work on external drive issues next. | |
| 15:55 | z3p: i'll test that windows fix, too. | |
| 15:56 | z3p | willkg: if it doesn't work, flip the super debugging flag and send me the log; it's possible there are other data formats windows is return |
| 15:57 | *returning | |
| 15:58 | paulproteus | Pseudo-random fact: feedparser is really slow. |
| 15:58 | z3p | whoops, C-wed the wrong window |
| 15:59 | janetPCF1 joined #miro-hackers | |
| 15:59 | janetPCF left #miro-hackers | |
| 16:03 | janetPCF1 left #miro-hackers | |
| 16:05 | janetPCF joined #miro-hackers | |
| 16:05 | willkg | z3p: i think we need to call that new progress thing CylonProgress. |
| 16:06 | z3p | willkg: haha, it does look like that |
| 16:06 | willkg: you're seeing it; that's good! | |
| 16:07 | willkg | well, actually, i'm seeing it in the browser display. |
| 16:07 | still not seeing my external drive on windows. i'm double-checking that i didn't goof something up while testing now. | |
| 16:08 | paulproteus | ajonas: I have restored the progress information we were giving, and I just fixed a few bugs in the speedups. |
| 16:08 | z3p | willkg: damn |
| 16:08 | willkg | it could be goofy testing. i'll let you know in a minute or two. |
| 16:09 | ajonas | paulproteus: on a call, back in a bit |
| 16:09 | paulproteus | ajonas: no prob, just updating you |
| 16:16 | arpu__ left #miro-hackers | |
| 16:17 | willkg | z3p: yeah, my external drive on windows isn't working. it's an older laptop drive, so maybe the standards changed or something like that. |
| 16:17 | z3p: i reopened that bug and attached the log with super debugging enabled. | |
| 16:18 | z3p | willkg: awesome, windows isn't showing it as a USB device |
| 16:18 | willkg | z3p: w00t! |
| 16:18 | z3p | willkg: it's detecting it though, so that's something |
| 16:18 | willkg | z3p: it's in a 2.5 enclosure. |
| 16:21 | ajonas | paulproteus: shall i test things on dev? |
| 16:21 | paulproteus | Still making some changes. |
| 16:21 | z3p | willkg: can you do me a favor, and export HKLM\SYSTEM\CurrentControlSet\Enum and send it to me? |
| 16:22 | willkg | z3p: sure. how do i do that? |
| 16:22 | arpu joined #miro-hackers | |
| 16:22 | paulproteus | willkg: start->run->regedit |
| 16:22 | z3p | willkg: regedit, right click on Enum, and select 'Export' from the context menu |
| 16:22 | paulproteus | nm, z3p's got your back |
| 16:23 | z3p | willkg: a text file would be most useful |
| 16:26 | willkg | z3p: sent. |
| 16:26 | z3p: also, i screwed up--i'm using vista on that machine. | |
| 16:27 | z3p | okay |
| 16:27 | willkg | i'm doing too many things at once. not getting the details right. sorry about that. |
| 16:27 | also, i think i left my coffee in the microwave. bah. | |
| 16:31 | holy crap! it's noon! | |
| 16:34 | afk... | |
| 16:49 | paulproteus | ajonas: I'm pretty happy with the state of things on the dev server. Please give it a shot. |
| 16:49 | I'm doing a hsus import, and it's showing me progress, and doing thumbnails in parallel. | |
| 16:55 | aw man, except it said "there was an error" at the end. | |
| 16:55 | Things were going *so* well. | |
| 16:58 | * paulproteus | fixes that bug. |
| 17:00 | tests locally and eats lunch. | |
| 17:15 | janetPCF left #miro-hackers | |
| 17:19 | z3p | willkg: can you ping me when you've got a second to test something? |
| 17:19 | willkg | z3p: back! |
| 17:20 | z3p | willkg: awesome! |
| 17:20 | willkg: I'm going to pastebin something; can you run it on the windows box and tell me if it gives you any output? | |
| 17:20 | willkg | sure! |
| 17:20 | ajonas | paulproteus: i'm back from lunch, should i test? did you push that fix? |
| 17:20 | paulproteus | I just did push a fix, and tested it locally. |
| 17:21 | You should see videos import about 5x faster, and they should import 25 at a time. | |
| 17:21 | And the process should actually succeed. | |
| 17:21 | I'm doing another test run on floyd-dev with the http://gdata.youtube.com/feeds[…]orderby=published feed which was what we were testing with before. | |
| 17:21 | z3p | willkg: http://pastebin.ca/2051627 |
| 17:22 | willkg: you should be able to run it as a script | |
| 17:22 | willkg | ok. give me a minute. it's on a different machine, so i ahve to do some finagling. |
| 17:22 | paulproteus | Also, fwiw, there is a 1-20 second period (with the code the way it is right now) where, if a feed is set to autopublish, a video could appear on the site without a thumbnail. It's not a very long period, but I guess it's probably a regression, and I can work on closing that issue if we're happy with this so far. |
| 17:23 | z3p | willkg: no hurry; you're helping me a ton |
| 17:23 | ajonas | paulproteus: cool - i do think we should close that issue, will test i a few (realized i need to do one other thing first) |
| 17:24 | paulproteus | b |
| 17:27 | willkg | z3p: running the script gives me output. |
| 17:28 | z3p: http://pastebin.ca/2051631 | |
| 17:29 | z3p | willkg: you've got a camera and the single drive connected, right? |
| 17:29 | willkg | z3p: it has a built-in camera. there's one external drive connected. |
| 17:30 | z3p | willkg: cool, so I'll refactor the search a bit |
| 17:30 | willkg | z3p: w00t! |
| 17:30 | z3p: ping me if there's anything else i can do. i'm going to look into the external drive problems i'm having on linux now. | |
| 17:30 | z3p | willkg: awesome |
| 17:31 | willkg: btw, I'm sorry that it's working so poorly for you | |
| 17:31 | willkg | z3p: no worries. it's really good to know that my problems are edge cases and not "it's totally broken for everyone". |
| 17:34 | also, i should probably share my clean-slate-environment run scripts. they make some of this testing so much easier. | |
| 17:37 | paulproteus | I'll be back in ca. 20 min. |
| 17:38 | ajonas | paulproteus: testing now |
| 17:39 | paulproteus: how much of a pain would it be to show people an update earlier than 25 videos? | |
| 17:40 | paulproteus: to show updates 10 at a time, maybe, instead | |
| 17:41 | zanoi | willkg, would you mind having a quick look at my patch for 17057 and see if you agree with it? |
| 17:43 | ajonas | paulproteus: seems to be working swimmingly other than that. can you mark the ticket (i think there's a ticket???) as fixed when you have a chance? |
| 17:54 | janetPCF joined #miro-hackers | |
| 17:55 | paulproteus | ajonas: So it'll probably cause a 5% performance decrease and a 45 minute time investment on my end. I would actually suggest just making the page refresh a bit more often, since 25 videos often happens in less than 20 seconds. |
| 17:55 | It's somewhat complex, and 45 minutes might be actually too low a guess. | |
| 17:56 | ajonas | paulproteus: can we do what you're suggesting in much less time? |
| 17:56 | DGMurdockIII joined #miro-hackers | |
| 17:56 | paulproteus | ajonas: Huh? (-: |
| 17:56 | ajonas | paulproteus: as in, how much time to make the page reload more often? |
| 17:56 | paulproteus | Oh, yeah -- that's approximately a 2 sec change. |
| 17:56 | ajonas | paulproteus: let's do that |
| 17:56 | paulproteus | I'll just do it right now, and make it refresh every 5 sec. |
| 17:56 | (-: | |
| 17:56 | willkg | zanoi: that code doesn't handle files with no extension very well. probably better to use os.path.splitext(). |
| 17:57 | zanoi | willkg, ok, thx |
| 17:57 | ajonas | paulproteus: then i'd say mark it closed and move on! maybe maggie can give it some more thorough testing tonight and ideally then we coul dpush it live tomorrow? |
| 17:57 | willkg | zanoi: i'm still looking through it.... |
| 17:57 | arpu left #miro-hackers | |
| 17:57 | paulproteus | ajonas: There's also the issue to close-up about some videos becoming published before their thumbnails are ready. |
| 17:57 | ajonas | paulproteus: ah, yes yes |
| 17:58 | time estimate on that? | |
| 17:58 | paulproteus | Other than that, though, great. |
| 17:58 | willkg | zanoi: the popen line looks fishy in regards to \", too. |
| 17:58 | paulproteus | 45 min? It mostly requires careful reading and thinking, and then a 3 line change. |
| 17:58 | willkg | i think that generates a command like: |
| 17:58 | paulproteus | I'll get on that straight away. |
| 17:58 | willkg | explorer /select,foo.mp3" |
| 17:59 | zanoi | yeah, true. Strange that it still works |
| 17:59 | willkg | zanoi: mmm... looks like it's copied from the other Popen line. but that also looks wrong. |
| 17:59 | zanoi | i'll fix that |
| 17:59 | willkg | zanoi: well, i'm not positive it's wrong. it just looks fishy. |
| 18:00 | zanoi | I wrote the original line, so I'll check where i found the instructions |
| 18:00 | have to find them first though | |
| 18:00 | egor joined #miro-hackers | |
| 18:00 | * willkg | nods. |
| 18:01 | paulproteus | ajonas: So what I'm going to do is make it so that videos can possibly possibly appear in the review queue before their thumbnails are fully ready, but they can't appear as published until their thumbnails are ready. |
| 18:01 | The video will wait for the thumbnail to become published. | |
| 18:01 | zanoi | willkg, ok it was wrong. Sorry about that. |
| 18:02 | ajonas | paulproteus: that's fine. i'm sad to spend so much of your time on this semi edge case, but i do think it's important for sites that rely on having videos be presentation ready |
| 18:02 | willkg | zanoi: other than that, it looks ok as far as i can tell. might want to have z3p, bendk, or glee take a pass at it to verify. |
| 18:02 | zanoi: no worries. :) | |
| 18:02 | paulproteus | I think handling this edge case is fine. |
| 18:02 | zanoi | oddly enough the explorer thing still worked though |
| 18:03 | well the sucky thing is if we change how we compute the program name in setup.py it breaks the code in WindowsApplication | |
| 18:03 | but there isn't really a better solution for now i think | |
| 18:03 | * paulproteus | ponders. |
| 18:03 | paulproteus | It'd be even better to just make the video have a new status, like VIDEO_STATUS_IMPORT_IN_PROGRESS, actually. |
| 18:03 | zanoi | willkg, ^ |
| 18:03 | willkg | zanoi: what's wrong with the prog_id? |
| 18:04 | zanoi | well it's just that we now compute the prog_id at two different places and if they don't stay in sync the comparison will fail |
| 18:04 | paulproteus | I'll do it the quick way, though. |
| 18:05 | willkg | true. but that's not an issue i have with the code. you can toss a big FIXME in both places and then we can deal with it in another version. |
| 18:05 | zanoi | willkg, yes, that's what I did |
| 18:05 | willkg | we have that kind of thing in a bunch of other places, too. it sucks, but ... so it goes. |
| 18:06 | paulproteus | z3p: Quick sanity-check: I think I'm going to create a new video status called PENDING. |
| 18:06 | willkg | the baby is giggling right now. it's so cute. |
| 18:06 | paulproteus | willkg: (-: |
| 18:07 | z3p | paulproteus: how is that different from UNAPPROVED? |
| 18:07 | willkg | he's like a siren lulling me away with his giggly tune. |
| 18:07 | paulproteus | z3p: The time when a video is .save()d but its thumbnail isn't ready, it'll be PENDING. Then the thumbnailer will change the status to the right value (ACTIVE/UNAPPROVED). It seems to me this should be a pretty simple change with not much major effects on the codebase. |
| 18:07 | z3p: Well, I would rather things not even appear in the review queue until they're finished getting the thumbs. | |
| 18:08 | z3p | paulproteus: can it be 'PENDING_THUMBNAIL' then? |
| 18:08 | paulproteus | Heck yeah. |
| 18:09 | * paulproteus | does this. |
| 18:20 | paulproteus | Okay, should be implemented now. Testing locally. |
| 18:20 | bendk joined #miro-hackers | |
| 18:20 | bendk | hey all |
| 18:21 | so I'm working on this bug: http://bugzilla.pculture.org/s[…]_bug.cgi?id=17013 | |
| 18:21 | which is fairly crazy, shutil.move is failing for what I think are dumb UAC issues | |
| 18:22 | willkg | bendk: that's awesome! |
| 18:22 | for some really funky version of "awesome". | |
| 18:22 | bendk | hmm, I was about to propose something, but then I realized it wouldn't help |
| 18:27 | willkg | npr mentioned that bugzilla wasn't working well for him today. is anyone else having bugzilla slowness issues? |
| 18:27 | zanoi | not me |
| 18:28 | bendk | I've noticed it |
| 18:37 | willkg | z3p: so, i can reproduce the "external drives don't work on linux" issue with the drive i've been using for the last few months (ext3 or 4, symlinks, lots of media files, lots of non-mediafiles, ...), but not the new drive i created (FAT32, no symlinks, mostly media files). |
| 18:37 | z3p | willkg: okay; I've got a drive like that around at home, so I'll play around when I get back there |
| 18:38 | * paulproteus | fixes one last bug... |
| 18:38 | willkg | z3p: i think i can fix this. let me take a stab at it and i'll pass to you when i run out of oomph. |
| 18:38 | z3p | willkg: cool |
| 18:39 | paulproteus | ajonas: FWIW it seems to do the first 25 videos in about 20 seconds. |
| 18:40 | willkg | z3p: also, there's a "Permission denied" on this drive. i bet that returns None and that's causing the scan to stop. |
| 18:40 | ajonas | paulproteus: cool |
| 18:44 | paulproteus | Since this does the video metadata download as a separate step from thumbnailing, I'm going to change the UI so that it only shows the # of videos that are fully imported, thumbnails and all. |
| 18:46 | (done) | |
| 18:47 | ajonas: You might enjoy the slow but realistic up-ticking of http://floyd-dev.mirocommunity[…]a261-e09c61306961 | |
| 18:47 | ajonas | paulproteus: nice! (note that these might go to the review queue cuz i think it's set to basic) |
| 18:47 | * paulproteus | nods. |
| 18:47 | hopes it successfully finishes. | |
| 18:48 | paulproteus | Wow it did!! |
| 18:48 | Okay, I am quite happy with this now. | |
| 18:48 | * paulproteus | looks for the bug. |
| 18:49 | paulproteus | ...is there a Bugzilla bug here? I recall there's a ZenDesk ticket. |
| 18:49 | http://bugzilla.pculture.org/r[…]munity&target=1.2 I think is the winner | |
| 18:50 | ajonas | paulproteus: did you find it? also, yikes! we should do those last 1.2 stuff and kick anything non-crucial down to a new milestone |
| 18:50 | paulproteus | Oh, hmmph. There is a deadlock possibility that I should probably pay attention to. |
| 18:50 | ajonas: Do you happen to watch _Pushing Daisies_? | |
| 18:51 | ajonas | paulproteus: indeed |
| 18:51 | paulproteus | (Sweet.) Remember the episode about the flower-power car? |
| 18:51 | The complex circumstances of creating the explosion... | |
| 18:51 | ajonas | paulproteus: ...not sure |
| 18:52 | paulproteus | "So what if – in the unlikely, but not impossible event – that the car gets up to a speed of 70 miles an hour with the headlights on and the seat warmer on low, a short circuit in the radio causes a cataclysmic chain reaction that blows the car and its precious human cargo to smithereens!" <http://pushingdaisies.wetpaint[…]ript%3A+Episode+2> |
| 18:52 | If there are 8 bulk_imports going on at once, it's possible they will block each other out from ever totally succeeding. | |
| 18:53 | ajonas | paulproteus: ah yes |
| 18:53 | paulproteus: worth looking into | |
| 18:53 | paulproteus | Sadly, yes. |
| 18:53 | ajonas | paulproteus: no exploding ghosts here! |
| 18:53 | paulproteus | or hidden crash test dummies. |
| 18:54 | willkg | paulproteus: you need more forks! |
| 18:54 | paulproteus: or chopsticks. | |
| 18:55 | z3p: ok. i finally added some logging to miro_allfiles and you're right. it is continuing to scan. it's just taking a minute break every x number of files. | |
| 18:55 | z3p: because this drive has a gazillion files on it, this takes forever to finish. | |
| 18:55 | z3p | willkg: *nod* |
| 18:56 | willkg: so the break is so other stuff can run | |
| 18:56 | willkg | z3p: and in the meantime, the Music tab says "Importing file details: 0 of 3097". |
| 18:56 | z3p | willkg: but I'm not sure what else is taking up all the time |
| 18:56 | willkg | z3p: ok. but it's literally like a minute long and miro isn't doing anything else. |
| 18:56 | z3p | willkg: maybe the idle handler isn't being called frequently enough? |
| 18:57 | willkg | z3p: and this most recent break took 3 minutes. |
| 18:57 | 14:47 to 14:50. | |
| 18:57 | bendk: you on? | |
| 18:57 | bendk | yup |
| 18:57 | z3p | willkg: if you remove the yields in scan_device_for_files() it should go all the way through |
| 18:57 | at the cost of having nothing else run | |
| 18:58 | bendk | interesting |
| 18:58 | I made a change to idle_iterator lately, maybe it didn't work | |
| 18:58 | willkg | z3p: when you say "nothing else" is that nothing miro-wise or nothing else on the computer? |
| 18:58 | z3p | willkg: mirowise |
| 18:59 | bendk | willkg: can you try doing update all feeds? |
| 18:59 | does that make the delays slower? | |
| 18:59 | willkg | i'm not sure i could measure that without spending an hour on it. |
| 18:59 | z3p | willkg: it'll dominate the eventloop |
| 18:59 | willkg | z3p: ok. so... why take breaks > 1 minute? |
| 18:59 | bendk | willkg: I think the 1 minute break is the issue |
| 19:00 | if you do update all feeds, maybe those breaks will get less long | |
| 19:00 | willkg | z3p: i understand yielding and idle_iterators. i'm not against that. i'm against the "it's taking so long to scan my drive it seems like it's not doing anything" thing. |
| 19:00 | i'll try it out | |
| 19:00 | z3p | willkg: right, but the minute break isn't the fault of allfiles() or scan_device_for_files(); it's not being returned to often enough |
| 19:01 | willkg | z3p: i know. |
| 19:01 | z3p: i'm just tracking backwards through the issue. | |
| 19:02 | bendk: if i update feeds, it immediately goes into a flurry of scanning, too. every time. | |
| 19:03 | bendk | willkg: maybe add some logging statements in SimpleEventLoop.loop(), I'm pretty sure the issue is in there |
| 19:03 | willkg | bendk: good call. |
| 19:03 | bendk | actually, I'd be willing to bet that we're waiting in select() |
| 19:04 | willkg | bendk: i think so, too. i'm not sure how to debug this--i'm not familiar with the eventloop code at all. |
| 19:05 | bendk | willkg: can you try this: http://dpaste.com/536912/ |
| 19:06 | the background on this one is before idle_iterator() called add_idle() directly, but that would mean that we would always have an idle pending, so it would starve the network callbacks and the other stuff in the event loop | |
| 19:06 | so I changed it to finish the current loop, then add the idle | |
| 19:06 | but I forgot the wakeup() call in add_idle, which is pretty important | |
| 19:07 | willkg | that's working a lot better. |
| 19:09 | bendk: that fixed it. w00t! | |
| 19:09 | z3p, bendk: thank you both! i'll check in this fix, update the bug, and all that. | |
| 19:22 | z3p | willkg: awesome |
| 19:24 | paulproteus | Well, I think I fixed the deadlock by implementing a "poor-man's" version of celery within the bulk_import management command. |
| 19:24 | willkg | it's still weird that state isn't a default column for the Music tab (in list mode) for devices. |
| 19:24 | * paulproteus | waits for a local run to finish before deploying. |
| 19:27 | paulproteus | z3p: You're going to totally hate what I did to bulk_import, I fear. It deserves a clean-up pass, but I think it'll do for now. |
| 19:28 | * paulproteus | marks bug as resolved. |
| 19:33 | willkg | zanoi: looks like you checked in that patch for 17057, but didn't fix the issues i think it has. |
| 19:33 | zanoi: was that a mistake? | |
| 19:37 | z3p left #miro-hackers | |
| 19:38 | z3p joined #miro-hackers | |
| 19:40 | willkg | z3p: i checked that fix in. i need to let my machine cool a bit, then i'll re-check 17168. |
| 20:17 | z3p: no dice. i can still reproduce 17168. i handed it back. | |
| 20:17 | z3p: it's not something we need for the beta, though. so it can wait until next week. | |
| 20:19 | z3p | willkg: yeah, I'm trying to get the windows stuff in for the beta, but it's more complicated than I want it to be |
| 20:19 | willkg | z3p: that sucks. |
| 20:20 | deardiary: done fiddling with external drives for today. onwards to beta channel! | |
| 20:22 | bug 17170 covers the beta channel stuff. | |
| 20:24 | zanoi: you on? | |
| 20:29 | zanoi: nevermind. i wrote a comment in the bug. | |
| 20:31 | afk a smidge for a snack. then beta channel. | |
| 20:42 | z3p | willkg: http://pastebin.ca/2051695 |
| 20:42 | willkg: can you see if that prints the right USB data for you? | |
| 20:42 | willkg | z3p: give me a minute.... |
| 20:44 | z3p: i ran the script and it prints no output. | |
| 20:44 | z3p | willkg: frig |
| 20:44 | willkg | sorry dude. |
| 20:44 | afk for a snack for realz this time. | |
| 20:47 | janetPCF1 joined #miro-hackers | |
| 20:49 | janetPCF left #miro-hackers | |
| 21:12 | janetPCF1 left #miro-hackers | |
| 21:15 | z3p | willkg: I checked in a new usbutils which might work for you; http://git.participatorycultur[…]ows-devices-16728 |
| 21:15 | willkg: I'm leaving the café; shoot me an e-mail if it works, or doesn't | |
| 21:21 | z3p left #miro-hackers | |
| 21:25 | janetPCF joined #miro-hackers | |
| 21:29 | z3p joined #miro-hackers | |
| 21:30 | maggie_s joined #miro-hackers | |
| 21:35 | bendk left #miro-hackers | |
| 21:39 | ajonas | hi maggie_s ! |
| 21:40 | maggie_s: we have a bug fix for you to test if you have time tonight - http://bugzilla.pculture.org/s[…]_bug.cgi?id=14937 | |
| 21:41 | maggie_s | ajonas: hi! *waves* |
| 21:41 | ajonas: sure, let me have a look | |
| 21:43 | ajonas: no problem, I'll look into it in a few minutes | |
| 21:43 | ajonas | maggie_s: thanks! |
| 21:43 | maggie_s | ajonas: you are always welcome! |
| 21:45 | janetPCF left #miro-hackers | |
| 21:53 | z3p | willkg: nevermind, it's probably not going to work |
| 21:54 | willkg: but I can reproduce the issue with an external HD | |
| 21:54 | willkg: so that's something | |
| 21:54 | willkg | z3p: cool! glad you can reproduce it. |
| 21:54 | z3p: i'm testing it right now. | |
| 21:58 | egor left #miro-hackers | |
| 21:59 | willkg | z3p: yeah... no dice. |
| 21:59 | z3p | willkg: windows is doing something different for this kind of device |
| 21:59 | willkg: it shows up like a built-in disk, rather than an external drive | |
| 22:00 | willkg | z3p: what "kind of device"? is it drives in enclosures? |
| 22:01 | z3p | willkg: I think so; it doesn't appear as part of the USB tree |
| 22:03 | willkg | screwy. |
| 22:03 | is this just vista? or xp and/or 7 as well? | |
| 22:03 | z3p | willkg: I'm testing W7 now; definitely XP |
| 22:04 | willkg | does it happen with all external drives or only some of them? |
| 22:12 | afk for dinner. | |
| 22:27 | maggie_s left #miro-hackers | |
| 22:27 | morganknutson joined #miro-hackers | |
| 22:33 | maggie_s joined #miro-hackers | |
| 22:35 | mattjohnson joined #miro-hackers | |
| 22:57 | z3p | willkg: it looks like both XP and W7 are making my drive in an enclosure look like fixed HDs |
| 22:57 | willkg | z3p: but flash drives look "regular"? |
| 22:58 | z3p | willkg: where "regular" is "removable", yes |
| 22:58 | willkg | screwy. |
| 22:58 | z3p | willkg: I don't see an easy way to tell the difference between 'drive installed inside the machine' and 'USB enclosure' |
| 22:58 | willkg | i know i had to assign a drive letter to it in order for it to show up in vista. |
| 22:59 | z3p | ooh, wait |
| 23:00 | DGMurdockIII left #miro-hackers | |
| 23:01 | willkg | z3p: was that useful to know? |
| 23:01 | z3p | willkg: no, just saw a note in the MSDN docs which might be helpful |
| 23:02 | willkg | ahh... got it. |
| 23:03 | ajonas left #miro-hackers | |
| 23:13 | CarlFK left #miro-hackers | |
| 23:17 | maggie_s | paulproteus: hi! |
| 23:17 | paulproteus | Hi maggie_s! |
| 23:17 | z3p: I'm curious to read the MSDN note, for my own curiosity's sake. | |
| 23:17 | maggie_s | paulproteus: I'm playing with importing feeds into MC |
| 23:18 | * paulproteus | gulps! |
| 23:18 | z3p | paulproteus: http://msdn.microsoft.com/en-u[…]939(v=VS.85).aspx |
| 23:18 | maggie_s | paulproteus: your feed passes with flying colours |
| 23:18 | z3p | not actually that helpful |
| 23:18 | paulproteus | Alas. |
| 23:18 | maggie_s | paulproteus: but all vimeo feeds seem to consistently throw errors |
| 23:19 | paulproteus | Interesting. If you submit URLs, I can test 'em locally shortly-ish and see if I can fix them. |
| 23:19 | maggie_s | just any vimeo feed, for example, http://www.vimeo.com/user1538555/videos |
| 23:19 | oops, http://www.vimeo.com/user1538555 | |
| 23:21 | paulproteus | maggie_s: Put them in the bug report, and I will get them fixed. |
| 23:21 | maggie_s | ok |
| 23:31 | z3p | new strategy: guessing! |
| 23:32 | paulproteus | maggie_s: Wait a sec... it fails on the non-dev servers, too? |
| 23:32 | I guess that page is a vimeo page, and we're not recognizing it as a feed. | |
| 23:32 | Can you try something that is definitely an RSS feed, from vimeo? | |
| 23:32 | maggie_s | paulproteus: just a second |
| 23:33 | paulproteus | By "I guess that" I mean "maaaaybe, I'm just making wild guesses here" |
| 23:35 | maggie_s | paulproteus: yeah, on live servers as well |
| 23:35 | paulproteus | Interesting. So the changes I made don't affect it. |
| 23:41 | maggie_s | not so fast - here's an RSS link: http://vimeo.com/channels/influence/videos/rss |
| 23:42 | paulproteus | Does that fail on the live sites? What about on the dev server? |
| 23:43 | maggie_s | both failed eventually... but the live server at least made a good try :) |
| 23:43 | zanoi | willkg, when is the deadline for pushing changes that will make it into the beta? |
| 23:43 | willkg | zanoi: are you asking for a specific time? |
| 23:43 | zanoi | more like whether it's today or tomorrow |
| 23:44 | willkg | it the earliest, it'll be my friday. |
| 23:45 | zanoi | well i kinda follow your timezone lifestyle wise |
| 23:45 | but I take it there'll be time for changes tomorrow then | |
| 23:46 | willkg | yes. |
| 23:46 | zanoi | k |
| 23:46 | willkg | i'll send out an email tomorrow morning asking where everyone is with things they're working on and we'll schedule the release moment accordingly. |
| 23:47 | paulproteus | maggie_s: Huh, good to know... |
| 23:47 | blip? | |
| 23:47 | willkg | amongst other things, i need to finish up the beta channel changes. |
| 23:47 | paulproteus | I'd also be interested in trying a random RSS feed from the Miro Guide. |
| 23:47 | maggie_s | I've got a feedburner in progress - so far, so good |
| 23:47 | will try blip next | |
| 23:48 | bhakem joined #miro-hackers | |
| 23:53 | * zanoi | nods |
| 23:53 | bhakem left #miro-hackers |
![https://bugzilla.mozilla.org/e[…]ages/mozchomp.gif](https://bugzilla.mozilla.org/extensions/BMO/web/images/mozchomp.gif){kind=link}