It’s been a while since I’ve posted any updates for pypeline. I’ve been recently getting familiar with OpenCV, which has a great feature recognition API. I’ve ditched PIL/pillow in favor of OpenCV, and it’s looking promising. Also I was getting sick of dealing with the .NEF files output by my camera, so I’ve switched over to using Dave Coffin’s dcraw. It’s great for converting just about any RAW filetype into tiff. The code has undergone substantial changes, it’s still available publicly at github. The only downside is OpenCV is currently only compatible with Python 2, so I guess py3k is out for now. Now I just need to get some decent photos so I can start working with them…
For the fun of it, I made a twitter bot. It searches for usages of the phrase “in a world where” and combines them. The results are occasionally comedic, poignant, and nonsensical.
In a world where you can be anything, nothing is in your control
— In A World Where (@TweetInAWorld) January 13, 2014
In a world where there is no drama, my cat pictures only gets five likes
— In A World Where (@TweetInAWorld) January 12, 2014
In a world where everyone hates me, everyone does this one thing
— In A World Where (@TweetInAWorld) December 28, 2013
I have found occasionally tweets get parsed incorrectly, and there are a few phrases I’ve seen repeated. Apparently the pool of “in a world” tweets isn’t all that large. It’s a little hit-or-miss, but occasionally turns up good ones. I built it with Python and tweepy. It wasn’t difficult with the tweepy API, and the learning experience was fun. You can follow it at @TweetInAWorld.
I’ve pushed some changes to the pypeline repository, adding basic stacking functionality. There isn’t any registration, it only takes the median of each channel (R, G, B) for each pixel. It’s currently way slow, but I suspect there is substantial room for improvement there. Wrangling NEF files has proven more difficult than I anticipated, so currently the state of the art in pypeline is JPGs.
My camera is rated down to 32°F, and nightly lows have been around 0, so I’m scared to take it out into the elements. On the plus side, the stacking works with regular images too! Any particular pixel just needs to have the “right” value for at least half the shots.
There is a little ghosting, but quite good considering, I’d say. I am not sure how to get rid of that totally. More pictures should quash the error, but at 5 I would have thought it would wipe out any traces of the marker. Also, a better algorithm should be able to push down the > 50% requirement to only a plurality. Maybe with some sort of clustering of values? I’m also taking the mean of each channel independently, maybe a better way would be to use luminance. In any case, baby steps!
Inspired by this post at DataGenetics, I implemented a quick-and-dirty script in python to test it out. The first takes an input image and iterates over it pixel by pixel, splitting it into two output images. Ideally, the outputs are randomly assigned, so it is impossible to recover the original without both outputs. The outputs can be combined to recover the original. Here’s an example of it in action:
Intermediate images (hopefully look like static):
Not perfect, but it is definitely recognizable. The idea can apparently be extended to 9×9 (and 16×16, and 25×25… I presume) images, for a wider-shared secret. In any case, this scheme should make it possible for any number of people to share a secret, but none of them individually can recover it. I uploaded the code here on github.
I’ve been a little busy lately, but should be able to pick things up again with the new PC I built. Made a time-lapse of assembling it too:
I’ve just enabled SSL on the site. Technically it’s TLS, but the name SSL seems to be sticking. Now you can access it at https://www.foxrow.com (note https). The not-encrypted version should still be available at http://www.foxrow.com. I’m not sure with Heroku hosting the apps if I can get it set up for ergs and weather, still looking into that.
Apparently PIL, and therefore Pillow, do not support Nikon RAW (.NEF) files. From what I can tell, my camera shoots RAWs in a 14-bit grayscale format. With the help of nefarious, I’ve found a way to get the image data out of my RAW files. Images coming soon!
As part of my ongoing effort to consolidate my various projects onto foxrow.com, I have moved the rowing weather app to weather.foxrow.com. Eventually I hope to get some interactive form of pypeline online as well. I am not sure yet what form that might take. Because RAWs are so big, uploading enough to get a reasonable outcome may be prohibitively bandwith-intensive. Also processing power on the server side may prove too big a bottleneck, but I won’t know until I try it.
I’m using Linux Mint 15 (“Olivia”) for my pypeline development – and ran into a little difficulty right off the bat getting pillow installed. Virtualenv and Virtualenvwrapper are pretty much a must-have if you’re developing multiple projects, and
pip install pillow was failing with messages like
Python.h: No such file or directory despite following the installation instructions to a T. Turns out since I’m using Python 3,
sudo apt-get install python-dev doesn’t cut it. What you want is
sudo apt-get install python3-dev. Run that once and you should be good to go in your virtualenv.
Since I plan on taking quite a few night sky pictures, I’d like to automate the process as much as possible. The project is hosted on github here. Right now it’s essentially empty, but eventually features I’d like it to have:
- Image calibration – support for flat frames, bias frames, dark frames
- Image registration
- Image stacking
- An optional GUI
Why roll your own pipeline, when other programs exist?
I’d like to learn more about the process and get some experience with astrophotography.
The majority of stacking/alignment applications I’ve found are windows-only.
Why open source?
The majority of stacking/alignment applications I’ve found are closed-source, even if they’re free as in beer. I like free as in everything.
What formats/algorithms do you support?
Right now, none. Since my camera is a Nikon D5200, the first thing I’ll implement is Nikon RAW (NEF) files. Various stacking/averaging algorithms will have to come later.
What’s your release schedule?
When I get around to it. There’s no hard timetable on anything, but I’d like to get a good chunk implemented sooner rather than later. Got some pictures to process!
Python 3? Really?
Yes, really. Python 3 is the future of the language, and Django and PIL were the two major things holding me back. Well, now Django is py3k compatible, and now we have pillow, the friendly PIL fork. Lucky!
That’s all for now, follow the project on github or fork it and contribute!