setting cache metadata on files in s3

i use Amazon’s AWS Console for uploading files to S3 and i always have to remember to manually add in the Cache-Control metadata so that i don’t get boned on bandwidth fees. i wish they would put in a nice default for that but oh well. i searched around for a better tool to use that would let me also recursively update the metadata on all my S3 files but the only one that seemed to have that feature and run on OSX cost $70! youch!

well it turns out to be easy to code up a script to do this in python using the boto library so to save you all some time (and money!!) here’s what i ended up writing :) this recursively runs through all the objects in all the buckets on an S3 account and sets the Cache-Control for JPG and PNG files.

from boto.s3.connection import S3Connection

connection = S3Connection('aws access key', 'aws secret key')

buckets = connection.get_all_buckets()

for bucket in buckets:
    for key in bucket.list():
        print('%s' % key)	

        if key.name.endswith('.jpg'):
            contentType = 'image/jpeg'
        elif key.name.endswith('.png'):
            contentType = 'image/png'
        else:
            continue

        key.metadata.update({
            'Content-Type': contentType,
            'Cache-Control': 'max-age=864000'
        })
        key.copy(
            key.bucket.name, 
            key.name, 
            key.metadata, 
            preserve_acl=True
        )

one month of Sketch Club!!

it’s been a month now since Sketch Club hit the app store! it’s been so much fun seeing what people create and upload to the gallery! i’m really inspired and awed by what people are able to pull from their minds and share. i’ve hoped to spend a lot more time sketching myself but when i do i just find more things to do or fix in the app or on the site and wind up coding haha. :P

along with the online gallery i’ve made somewhat of an offline gallery for Sketch Club as well. on the wall behind my desk at work i’ve been putting up 4×6″ prints of some of the best sketches. it’s quickly grown from just a shmattering to nearly the whole wall! i’ve had to move my big NIN poster out of the way and soon my generative art and tech prints will have to find new homes as well. here you can see a bit of the progression… :)

Sketch Club wall 0 Sketch Club wall 1 Sketch Club wall 3

sketch club!

yay apple approved my new app Sketch Club and it’s now in the app store!! (here’s a direct link to it in the store)

what is Sketch Club??? it’s a super rad drawing / sketching / painting app for ipads, pods, and phones. :) it’s got sweet procedural brushes like sketchy and fur (based on trace aka mr. doob‘s epic html5 sketching experiment harmony and zefrank’s scribbler). it’s also got smooth (which is my own take on the connect to nearest points action) and side lines (umm great for fuzzy caterpillars?!). along with that procedural madness there are regular brushes like you’d find in photoshop but pared down to the most useful functionality and a vector brush for simplified smooth or polygonal drawing. also there are foreground and background layers to draw in, a photo layer for background or tracing, and canvas color.

that’s all pretty sweet, but wait there’s more! :) sketch club provides not just the means of creating cool images but also the motivation! integrated with the app is an online gallery where you can upload what you make and enter it into fun competitions! the online bit is very much inspired by the wonderful demoscene portal pouet.net. you can view all the sketches people upload and rate them (rules, piggy, sucks) and as you do you earn glops and after accumulating enough glops you get hearts. it takes a lot to earn hearts so you give them to only your most favorite sketches. one departure from pouet is that while you can give sketches piggy and sucks votes (and they do factor into rankings) they are never displayed to anyone else. this way you don’t have to worry about hurting someone’s feelings and people won’t get discouraged.

you’ll notice there’s already some pretty amazing sketches in the gallery! while i was working on the app and it was still littered with horrible bugs and missing features some very brave friends tested it out and helped drive its features. omg i lurve you guys, thank you!!!!! :) there are still of course some smaller bugs and tooons of features i didn’t get to yet so expect plenty of updates!

okay so even if you don’t have an ipod or ipad you can still participate by going to sketchclub.com and logging in with your facebook account. you’ll be a guest so you can rate and comment but won’t be able to earn hearts or upload sketches. if you do get a device later you can go in the app and link to your facebook account and that will upgrade you to a full member!

so yeah get sketch club and try it out, it’s pretty sweet! :)

stroke my ball!

a few months ago i wrote my first iPad code and my first shaders for iOS devices. i made a fun 3d globe control for ABC News’ iPad app! it’s a bunch of cards on a sphere that you slide around to browse through news stories. it doesn’t use any standard projection method but instead some really crazy stuff so that cards in the center are nice and big and cards get small really quick as they fall off from the center. it feels really nice sliding it around. it calls back on its delegate as cards scroll into view so it allows for scrolling through an unlimited set of data in both directions and allows asynchronous loads and updates of the cards.

bad news: i got no monies or greets for this and my nice spec highlights and mip-mapping were disabled in the final for some reason :(
good news: tons of people are now touching, swiping, and stroking my ball! ;)

here’s a link to the full info for the app so you can download and install it on your iPad! be sure to rate it five stars in the app store and comment that you loved stroking blackpawn’s ball!! :D

still lovin’ the svg

i posted a few months back on using the scalable vector graphics (SVG) web format for visualizing data structures. i keep finding more great uses for it! by now i’ve also put it to work visualizing GPU memory allocations, asynchronous file operations and access patterns, hierarchical per render frame CPU work, and location based data map overlays. this is quite the handy visualization tool to have around! :)

in addition to lines which i covered in the first post, i’ve now been using circles, rectangles, text, images, and layers. these are all super easy just like the lines. for example:

<text style="font-size:10px;fill:#ff0000" x="4" y="16">ReticulateTheSplines</text>

<image x="8" y="30" width="256" height="256" xlink:href="base.jpg" />

i’ve found it really handy to go into InkScape and make a quick SVG by hand then load it up in a text editor to see the right markup to use for different things.  for example with layers InkScape has it’s own namespace and extra options so you can put really spammy data in topmost hidden layers and only reveal them when you’re zoomed in on some area of interest.

here’s a link to the full W3C SVG standard which is a great reference for all the possibilities. and here are some new examples of this madness in action.

@party entries

with @party this weekend came the first releases of a new demo group vrtx (pronounced “vertex”). we did two entries for the freestyle graphics compo and got 2nd and 6th place. kirill did this sweet spider web scene and guybrush, kirill, and i did the mazelized text. watch out for more cool graphics and demos coming from vrtx in the future!

vrtx web VRTX

blockparty 2010

i couldn’t make it out to blockparty again this year but i did participate online through the ustream stream and chat which was a blast! i really hope more demo parties do this in the future. watching live party streams is fun but being able to chat at the same time with people at the party and others watching the stream during the compos is just so much better. the archived video is available on ustream but unfortunately the hilarious chat doesn’t replay with it haha. and of course lots of the entries are available for download on pouet.

though we didn’t make it out in person, xplsv took 1st in the hi-rez graphics compo with kirill’s rad render! it’s actually using assets from a demo we started after finishing our invite for blockparty 2009. we wanted to release the demo at blockparty 2009 but didn’t have time to finish and then again for 2010 we wanted to do it and again didn’t find the time. don’t worry though we’ll release this guy eventually. :)

XPLSV

congrats to everyone who did manage to make a release or make it to the party! and how great to hear the fifth edition of blockparty will be on the west coast in california! see you there? :)

beta, wires, and blobs

if you’ve encountered me at all in the last week either in person or online then you know i got into the Starcraft 2 beta!!@ with Starcraft still being my favorite game of all time i can’t help but be exploding with excitement over this! now i’m not going to write a big review or anything like that here because everyone already knows it’s fantastic and will be buying at least one copy when it’s released. but well okay… it’s amazing fantastic wonderful true to the original super fun crazy addictive and all these things which is making it really hard for me to do anything productive!!! gg blizzard. :)

but anyway sole has been diligently blogging daily on her breakpoint demo progress so i should at least be able to manage weekly updates right? thanks for being a good influence sole!

so recently i’ve added what i think are some really neat animation and sync controls to the demo system but haven’t capped a video of that yet so you’ll just have to wait for those details. :) also got in material support, more controls over copied instances, metaballs, and mesh displacement.

here are some new pictures of a simple scene generated in demo studio and rendered offline:

splash4 splash5

splash7 splash

and once again here’s a peek at the construction:

okay now i need to get in a match of SC2… or maybe two… :)

first test with demo studio v4

this year i’ve been working on the next version of demo studio. so in brief, version 1 (tokyo, mudballs, ccc) was your typical drop effects into a timeline and edit the parameters, version 2 (hofn, sokuseki) had more powerful sync and scripted effects, version 3 (n-0505, blockparty invite) was entirely code driven with pop-up ui only for tweaking. with version 4 i hope to get the best of both worlds between artist/designer support with ui but without sacrificing the handcrafted codery goodness that comes from not having ui. :)

with lots of the core of the new system done now i can get things animating and syncing to music. here’s one of the quick tests from this weekend and a peek at the ui showing how it was done. the music is from Zardonic’s remix of Nine Inch Nails Ghosts track 35.

logging in the next decade

so 16 years ago i started keeping a journal, 14 years ago i started saving all my source code, 10 years ago i started saving regular screenshots of my projects and since then have been accelerating the rate at which i store off snapshots from originally around one every month to by the end of 2009 one image a day and one video a week. this is quite a lot of data but the rate of technological advance in storage has far exceeded the increasing rate of data i store. right now you can buy 1.5TB of storage for $100. this is insane and it makes my data set look pretty tiny and pathetic!

so to get started on the next decade i’ve upped the ante and created some software to help. inspired by gordon bell’s research and latest book Total Recall: How the E-Memory Revolution Will Change Everything my app grabs a screenshot of my multi-monitor desktop every 15 seconds and stores it off with lots of metadata. this was actually a bit too much data for my tastes since each compressed snapshot comes out to many megabytes. so i added an additional layer of inter-frame compression similar to what is done with video and this gets me down to well under 100 kilobytes for most snapshots and it’s currently trending around 100 megabytes per day. this still sounds like a lot but that 1.5TB drive is actually large enough to store that rate of data for me for the next 40 years! ridiculous right?

of course you’re probably wondering what the point is… well, there are lots! for one, just like the journal it is fun and rewarding to go back in time and see what my life was like in the past and recall the things i was thinking about and doing. it helps to gain perspective on things. with this automated capture now so many more possibilities unfold. my simple playback app at the moment can already give me fast replays of the past as well as statistics on my activity (how much time spent coding, web browsing, chatting, etc). in the future i’ll be able to run OCR on the data to recognize any text on screen and then be able to search effectively and quickly extract text from my computer at any moment in the past. further in the future i’ll be able to feed all this into generic AI software to train it to respond, work, and think like me so i can have great digital assistants and even further in the future provide a much more accurate history and memory to my simulated consciousness after all of our brains have been scanned and moved to processors in space! :)

anyway this work is the reason i didn’t make as much progress as i wanted on other projects this holiday. :P

4sceners.de best of 2009

wow, cool! the invitation demo that kirill, gloom, and i made for blockparty 2009 was deemed one of the best invites of the year by 4sceners.de! that makes me happy. :) here’s the google translation of their best of 2009 page.

i’m still bummed i didn’t make it to the actual party. oh well in 2010 i should make it. i’m also planning on attending the new @party in massachusetts. oh and breakpoint in germany in 2011. fellow usa demosceners please go to all those parties too! :)

4sceners.de-bestof2009

visualizing data structures with svg

so i found dropping SVG to disk is an easy and fun way to visualize or debug data structures. here’s all it takes in C:

FILE * fp = fopen("kdtree.svg", "w");
fprintf(fp, "<svg version=\"1.1\" width=\"1024\" height=\"1024\" xmlns=\"http://www.w3.org/2000/svg\">\n");
fprintf(fp, "<line stroke=\"#%06x\" stroke-width=\".1\" x1=\"%f\" y1=\"%f\" x2=\"%f\" y2=\"%f\"/>\n", ...);
fprintf(fp, "</svg>");

what’s nice is then you can load the file up in firefox or inkscape and zoom in. also you can easily draw in other data like triangles and circles or put nodes you’re interested in in different colors and what not.

here’s a 3d kd-tree seen from above subdividing space evenly
kdtree with even splits

here’s a 3d kd-tree seen from above using surface area heuristic
kdtree using surface area heuristic