What’s got me irked this time around (see Nobody Masturbates on Hacker News for other reasons) started out with this lazy and inane blog post: Android vs iOS: A Developer’s Perspective.  I’m going to dissect and refute the post later on here, but what got me heated was that the comments dispelling the myths put forth in the blog post about memory management were being consistently down voted in the thread on HN and, instead, the glib, snarky and unhelpful ones scored much higher.

The fuck Hacker News?  If it wasn’t for the sanity on reddit, I’d be totally losing my shit right about now.

Anyways, let’s get to memory management in Objective-C, because that’s what started this all off.

Memory Management in Objective-C is Easier Than James Gregory Thinks

I know the concept of memory management scares the shit out of script kids and the average java developers, but it really isn’t that scary.  But before we get into it, let’s see what James Gregory has to say about it:

iPhone doesn’t have it. Seriously, Apple: What The? Various luminarieshave waxed geeky on why garbage collection is so great, so I’ll keep this brief. Basically there’s two reasons it’s awesome: I really don’t have time to learn the rules of memory ownership and when to free, and all that stuff for yet another language. Secondly, I just can’t think of another language feature that accelerates development as much as garbage collection. Android has it, and for anything not pushing the envelope in terms of what the hardware can do (ie, almost everything), it works really well. I might have understood why Apple didn’t include it in the first couple of iPhones — it does come at a computational premium — but they really don’t have an excuse any more. It’s a big enough deal that I would advise the first-time mobile app developer to start on Android for this reason alone.

Jesus, there are so many things wrong with this, but mostly it reinforces wrong and lazy ideas about memory management.  His statement, “I really don’t have time to learn the rules of memory ownership and when to free, and all that stuff for yet another language” is about the most ignorant thing I’ve ever heard a developer say.  Basic object-oriented memory management is universal, no matter if you are doing C++, ObjectPascal or any other non-garbage collected language.  It’s some pretty basic computer science and the concepts, specifically how they apply to Objective-C, are nearly trivial.  If I were interviewing someone for a job and they said something similar, the interview would be concluded at that point.

I agree with him that the iPhone should have GC though, so don’t think for a second I don’t see its advantages.  That’s why you should listen when I tell you that manual memory management is not a significant effort, specifically in Objective-C.  I sometimes, though rarely, use it for Cocoa desktop apps that I build, and I build a lot of them.  The typical use case for me to use GC is when a quick app idea I’m sketching out in XCode becomes a fairly full featured thing.  So while I am in the process of exploration, GC is on because I’m not typically writing code I’ll reuse.  Once my idea is proven, however, the GC goes off and I refactor all of that code.  The other case where I have it on is when I discover a leak in cocoa itself, but even that is pretty rare.

So if I use GC sometimes, why don’t I use it all of the time?  Because it’s so easy to do memory management that I don’t need it.  I also tend to write a lot of private frameworks that I re-use across all sorts of projects, which may or may not have GC turned on.

The thing is, the retain/release model in Objective-C is roughly a middle ground from no memory management to a basic GC so you are already ahead of the curve.  Unlike C++ or ObjectPascal, you actually never technically “free” objects you are using, you simply are saying “I’m done with you” and then let the runtime take care of the rest.

The Four Basic Rules

There are only 4 basic rules when it comes to memory management in Objective-C:

• If you own it, release it.
• If you don’t own it, don’t release it.
• Override dealloc in your classes to release the fields that you own.
• Never call dealloc directly.

That’s it.  The first two are the most important and we’ll get into the basics of object ownership next.

Rule #1.  When You Own It.

You own an object in Objective-C when you alloc it, copy it or new it.  For example:

How easy is that?  I can’t even think of anything more to write about this, it’s that simple.  But to reiterate:

• You own it if you alloc it.
• You own it if you copy it.
• You own it if you new it.  (New is simply a shortcut for alloc/init).

James if you are still reading this, you are 1/4th the way to learning it buddy.  Just stick with me for the next couple of bits and you’ll be writing software I trust in no time.

Rule #2.  When You Don’t Own It.

This is the tricky part.  You don’t own it when you don’t own it.  So if you did not alloc it, or you did not copy it, or you did not new it – you don’t own it.

What about the following?

Do you own that NSString?  Nope, you didn’t alloc/copy/new it.

The NSImage?  Yes you own this, you alloc‘d that bad boy.

The NSData?  Nope, again, you didn’t alloc/copy/new it.

Here is how that method should really look, btw:

Yes, it’s so much harder than GC.  How do us Cocoa devs ever get anything done?

Rules #3 and #4.  Dealloc.

Ok, so here’s the hardest part of the whole thing.  If you have any class with object properties that you retain, you’ll have to release them when your object has been deallocated via the [dealloc] message.  An example to demonstrate:

In the above example, our SomeObject has two fields: things and someOtherThings.  You can see in our init method that we create the objects and assign them to our properties.  For things, because we are using [NSMutableArray arrayWithObjects:], we have to call retain.  Remember, we didn’t alloc, copy or new, instead we called a convenience method that returns an autoreleased object that we must explicitly retain.  What is autorelease you might be asking?  I’ll explain in the next part, but for now all you have to remember is the basics of object ownership described above.

In the [dealloc] method you’ll see that we are releasing the objects we created.  Nothing more, nothing less.

James if you have made it this far, you’ve essentially learned all you really need to know.  Everything below here is optional, but recommended to know.  That wasn’t so hard was it.

Properties

Objective-C 2.0 includes properties, and while I won’t get into the particulars of them, there is some confusion around the retain/release cycle with regards to them.  There is really only two rules:

• If you retain or copy your property, you need to set it to nil in your dealloc.
• If you initialize the property during init, autorelease it.

Let’s look at some properties:

The title property we are calling a convenience method on NSString that returns an autoreleased object, so we don’t have to do a thing.  For subtitle, we are allocating a new NSString ourselves, therefore we need to autorelease it.  The reason being is that when the value is assigned to the property, the retain count is incremented by 1, so without the autorelease, that object will maintain a retain count of 2 (one for the alloc/init and another for the assignment to the property).  With a retain count of 2, the object will never be released and you’ll leak memory.

Autorelease?

Ok, I totally lied as there is one more complicated aspect to this whole thing, and that complexity comes in the form of Autorelease Pools.  The heck is that?  Autorelease pools maintain a list of objects that will be sent release messages when those pools are destroyed.  You really needn’t worry about when all of that happens, but rest assured that your autorelease objects are, if everything is done correctly, being released without your explicit telling it to do so.

Following our rules of object ownership, one has to wonder what happens to objects that are returned from methods.  Who owns those?  This is where autorelease and autorelease pools come in to play.  For example:

Who owns that NSImage?  Well, in this example, the caller of this method is the owner and it is, therefore, the caller’s responsibility to release it.  However, this method is bad form.  The correct form:

Notice the autorelease message at the end.  What happens when autorelease is called, the object in question is added to an autorelease pool so that it will receive a release message when the pool is destroyed.  The caller needn’t have to worry about it, nor does the callee.

Autorelease pools can be nested, too.  This is great for situations where you are creating thousands upon thousands of temporary objects and want to keep your maximum memory footprint slim and tight.  Also, if you are doing threaded work, anytime you create a new thread, you’ll have to create an autorelease pool for that thread.

Responding to Hacker News

Below are some responses to some of the more incorrect responses on HN.  From orangecat:

And don’t get me started on those lazy kids and their assemblers. I mean, how hard is it to remember a few dozen opcode hex values?Less snarkily: Developer resources are not infinite. Time spent futzing with memory management in non-performance-critical areas is time not spent improving performance where it actually matters, adding features, or improving the user interface.

For example, Angry Birds on the Galaxy Tab versus Angry Birds on the iPad are no where near the same experiences. The Galaxy Tab is jerky and slow, while the iPad is smooth.

The Android code for Angry Birds is primarily in native code, so garbage collection is unlikely to be the cause of your observations. And it’s perfectly smooth on my Nexus One.

He is somehow brilliantly equating the difference between garbage collection and manual memory management in objective-c to the difference between writing raw machine code and writing assembly.  Bravo. The last line, he tells me Angry Birds is written in native code, which means no garbage collection (the NDK doesn’t do GC) and doing C++ memory management, which renders the whole point of James Gregory’s post moot. Slow clap.

binaryfinery says:

I’ve written memory managers used in console games for the N64 (4Mb), PS2 (32Mb) and a complete memory tracking system for Unreal Engine 3 on XBox360 (OMG 512Mb).I use MonoTouch for iOS development. Why? Because I can build complex object graphs without having to maintain, in my head or in external documentation, the acyclic version of the graph. In a GC language I can create graphs that are happily cyclic, and have every reason to be, and yet have them deallocated when the application nulls the last reference.

In a reference counted system that desires cyclic pointers, one or more of those pointers have to be chosen as non-reference-incrementing pointers – and likewise one must remember not to release them either.

Memory management in Objective-C (without GC) is no effort provided one is making applications with simple object interactions, such as the plethora of tree-based hierarchies. Reference counting is great for that.

So it could be that the OP is lazy, or it could be that they have experience with problems that you do not. If you believe that OP actually requires a lesson on the basic rules of reference counted memory management (since thats what you provided) then I suggest that you are underestimating the experience of your detractors, which in turn leads me to believe that you are overestimating yours.

Cyclic object graphs in Objective-C are not a problem.  At all.  In fact the retain/release facilitate them pretty easily, and some very basic comp sci 101 OOP theory will have you moving around them just fine.  Again, it boils down on what you own and what you don’t own.  Also, if you’re shit is too complicated to document, you are doing it wrong.  Straight up.  I don’t care if you wrote the internet or not, btw.

And finally, the one that made me the most irate, from the blog poster himself:

It’s totally about laziness! But that’s what computers are all about — I could store printed versions of all of my documents in a filing cabinet, and go and manually sort them every time I needed a different ordering, but I’m lazy! I use a database!I just don’t see why laziness should be restricted to users. Developers are lazy too.

You’re right that there are only 4 rules (or more or less depending on your formulation), but I don’t care. I’d rather take the time to have another martini. Or, y’know, implement features that make my users happy.

And it definitely gets harder when there are more moving parts. You’re right that the rules are simple, but the execution of those rules gets more complex as you add more components, more threads, remoting, etc. I never said it was impossible, or up there with Fermat’s Last Theorem or anything like that. Just that this is work the computer could be doing for me. I want to be lazy, but Apple won’t let me.

Brother, I am all about laziness, but I am also all about doing it right.  So, go ahead and drink up friend, because while you are wallowing in your own ignorance, people are out there doing it better than you, smarter than you.  I’m not sure about you, but in my profession and my career, I am constantly striving to do the best job that I can, and a large part of that goes towards ways to allow myself to be lazier, but to get it right at the same time.  Also, it’s not a matter of what you want, it’s a matter of making these damn computer things do what you want.  The trip, the technical aspects of it, have to be done right or your ship is going to sink.

In Summary

As you can see, this whole thing is not as complicated or scary as it seems.  Sure, GC is a lot easier, but even if you had it, there are so many instances where manual memory management is preferred: games and media apps.  Memory management is such a crucial aspect of the performance of these types of apps that not knowing how to do it wouldn’t work.  Thankfully, Cocoa and Objective-C make it pretty painless.

For More Info

Really, the only document you need to read is the one apple provides here.  But, in case you need some more help, the following links are fantastic:

• http://www.cocoadev.com/index.pl?MemoryManagement
• http://www.wooji-juice.com/blog/cocoa-6-memory.html
• http://www.dikant.de/2007/08/23/cocoa-memory-management-101/
• http://mattpatenaude.com/ch3-memory-management.html
• http://aspirement.com/2009/09/mastering-cocoa-memory-management-on-the-iphone/
• http://www.duckrowing.com/2010/03/08/the-cocoa-memory-management-rules-of-the-road/
• and on and on and on…

The framework I used to build the app uses a lot of separate javascript and css files, though the entire app is housed in a single html file.  So I end up with a <head> element jam packed with stuff.  While this is perfect for development, it’s no good for deployment or QA.

For an example of what the <head> element looks like during development:

That’s a lot of stuff.  To optimize for mobile, I’m going to need to smash these files together and run them through yui-compressor to make it as small as possible (and then serve it up with GZip to make it even smaller).

Now after some googling, I ran into Juicer.  Juicer is a great tool, but it still didn’t do everything I wanted, and specifying the dependencies within the javascript and CSS files was not as explicit as I desire.  It just seemed to prone to error.  Finally, after merging everything with Juicer, I’d have to manually update the html files by hand to replace all those script and link tags with the new ones.

Next up on the google list was Apache Ant.  I haven’t used Ant since doing a random Flex project a few years ago, and I was afraid it was simply too complicated to setup for my small little project.  But in my quest to avoid NIH syndrome, I gave it a spin anyhow.

I gave up after an hour or two as it was just too much.  Maybe I’m getting lazy in my older years, maybe I am suffering insufferable NIH syndrome, I’m not sure.  Also, I hate XML.

Tick Tock

So the clock is ticking down on my self-imposed deadline to get a system setup that “builds” my little HTML5 app into a production ready thing.  There were a couple of other options presented to me by google, but again, they either used the dependency thing of Juicer, or were horribly complicated.

So I did what any self-respecting developer would do: fired up TextMate and started hammering.

Meet makejs

When I set about building it, I had a few simple goals in mind:

• YAML based configuration files.  I love me some YAML.  I love being explicit.
• Ability to modify HTML files to replace <script> and <link> tags
• Generate an HTML5 cache manifest
• Run in continuous integration mode where changes to files would trigger a re-build.

So five hours later I had a working system.

Using makejs

makejs uses a YAML based configuration file to direct the build process.  Here is an example:

Now that’s a lot to process, but it’s pretty simple once you break it down.  The general layout for the build configuration is thus:

• Build Target
  • Version (Major.Minor.Build)
  • Pre-Requisite Build Targets
  • Build
    • Build Phase
      • Settings

And that’s essentially it.  But let’s look at it in more detail.

In the configuration above, you’ll notice I have two distinct build targets: “glamrock” and “example”.  GlamRock is the mobile framework I built, Example is a sample project that uses GlamRock.  For the GlamRock build target, you’ll see that I have 3 build phases: copy, compress and manifest.

The copy build phase does exactly as it sounds, it copies files to a destination in the build directory.  Nothing more, nothing less.  In this case, I like to keep jQuery and jQuery’s template stuff separate and not mashed into all of the other files.

The compress build phase takes a list of files, joins them together and then runs them through yui-compressor.  Let’s take a look at that build phase again:

If you look at the files element, you’ll see a filename (in this case, js/glam.rock.min.{VERSION}.js) with a list of files associated with it.  These files are the ones that are joined together and compressed to generate js/glam.rock.min.{VERSION}.js.  So what’s the deal with the {VERSION} thing?  That tells makejs to replace {VERSION} with the newest build version, which makejs takes care of automatically for you.  For example, if you’re configuration says version: 0.0.81 – the next time you build, makejs will auto-increment the build #, re-save the conf file, and then when it generates the compressed files, you’ll end up with a file name js/glam.rock.min.0.0.82.js.

The manifest phase is what builds the HTML5 cache manifest, a definite must for mobile HTML5 apps.  For GlamRock, it won’t actually build a manifest (see below for why), but it is important to list what files that this build produces which can be included in the cache manifest.

Now, let’s turn our attention to the example build target.  This build target has a copy, compress, alter and manifest build phase.  But before we dive into those, you’ll see that this build phase has a pre-requisite for GlamRock.  When this build target is built, GlamRock will be built first and the output of that build will be put into whatever the build directory for example is.  This is nice because we treat each build target as independent projects and get some nice dependency functionality out of it.  Keeps things clean and relatively simple.

Back to the build phases for the example build target.  I won’t go over copy and compress because there is nothing different happening here then there was in the GlamRock target.  So let’s look at the alter build phase:

The alter build phase takes any input file and replaces any denoted blocks with the proper <script> and <link> tags that have been generated during the entire build process.  If you look at the <head> example at the top of this post, you’ll see that there are actually two blocks surrounded by <!– BEGIN {target} –> and <!– END {target} –> comments.  During the alter phase, makejs will replace these blocks with the output of the specified build target.  For example, after building, this is what our new <head> looks like:

Pretty nice, right?  Note, that this never alters the original, it simply transforms the input into the destination in the build output directory.

The example build phase has a manifest phase, like GlamRock, but with one important difference: it has a name setting.  When building a manifest phase, if there is a name specified, than a physical cache manifest is generated.  So running this phase on the example build target yields a cache manifest saved to example.manifest in the output directory.  The example.manifest looks like this:

Perfect-o.

Getting It

Head on over to GitHub and check out the project: http://github.com/jawngee/makejs

Hope someone finds this useful!

However, what strikes me most about the Wired app is how amazingly similar it is to a multimedia CD-ROM from the 1990′s. This is not a compliment and actually turns out to be a fairly large problem…

1990′s Here We Come … Again

The only real differentiation between the Wired application and a multimedia CD-ROM is the delivery mechanism: you download it via the App Store versus buying a CD-ROM at the now defunct Egg Head store at your local strip mall. And I really mean that comparison. For all of the interactivity that was touted in the Flash prototype, what we’ve really ended up with is a glorified slide show. Instead of the “Next” and “Previous” buttons you might have been used to on those old CD-ROMs of yore, you instead swipe left and right to change pages (well *cough* images of pages).

There are certain interactive elements to the articles, but – and I apologize to all of the people who put in a lot of back breaking work into this – they’re pretty lame. Tapping on a button-looking element switches out part of the page with another image. You can drag your finger across certain images to make them sort of animate like a flipbook (and in truth, that’s what it is – a series of PNG or JPEG images). There are videos you can tap on to view fullscreen. There are audio clips that you can play. The interactivity in the Wired application is very 1990′s. I am not trying to be insulting either, it’s simply the truth. The Wired application has pretty much brought back image rollovers.

And that’s about the extent of the interactivity. Which makes me wonder if that should be the extent of it or should we be wanting more? I don’t have an answer to that, though my gut feeling is that there is a massive opportunity to reinvent the concept of a magazine – yet we end up with something akin to what the web was like in the mid to late 90′s. This basically boils down to a print designer’s vision of what the web should be like – but in this case it’s a print magazine person’s vision of what an interactive magazine should be like.

If you can think back that far, and you were doing web development during that time, you will glumly remember the frustration of web development driven by print design, where pages were essentially huge images cut and sliced and then reconstituted back into insane table structures for pixel perfect layouts. That’s what we’ve essentially gotten with the Wired app: a giant step backwards and a complete dismissal of the lessons we learned from that dark period of interaction design and development.

Holy Shit, That’s Big

With the Wired app weighing in at a whopping 500 megabytes – just 100 shy of a full CD-ROM – how do they intend to maintain new editions of the magazine? 500 MB is too large for a 3G download (no help from AT&T’s less than spectacular network performance) and for those with iPad’s with the smaller storage, each issue will take a significant chunk of space on the device. With no apparent means for managing which issues you keep on your device, this will become huge issue for a lot of people. Obviously they will fix this with updates to the application, but I’m still wondering what they were thinking to begin with. I’m hoping there were voices of dissent that pointed out the end product was not worth it’s weight in megabytes. A PDF version would have been a tenth of the size, though without the interactivity. But is the interactivity worth the 500MB price? I personally don’t think so.

Why is the magazine so large? Being the intrepid hacker that I am (*wink*) I mounted my jail broken iPad via AppleTalk and quickly tore into the app itself to see how it was constructed. Similar to the PopSci+ magazine application, each Wired issue is actually a bunch of XML files that lay out a bunch of images. And by “a bunch of images” I mean 4,109 images weighing in at 397MB.

Each full page is a giant image – there are actually two images for each page: one for landscape and one for portrait mode. Yes, I’m laughing on the inside too. There is no text or HTML, just one gigantic image. The “interactive” pieces where you can slide your finger to animate it are just a series of JPG files. When you press play on the audio file and see the progress meter animate? A series of PNG files.

Something is wrong with this picture. Something wrong and something very lazy and/or desperate.

Over Architect Much? (or How Desperation Ruins Good Ideas)

I have no inside knowledge on how the Wired app was produced, so the following is all conjecture on my part. That said, my guess is that Adobe sold Conde Nast on doing the thing in Flash. Or if Adobe didn’t do the selling, some Flash loving technologist at Conde Nast sold them on it. Either way, since Flash CS5 was going to be able to target the iPhone/iPad, they’d be able to publish the thing as it had been shown to the press. But then Steve Jobs came along and threw section 3.1.3 into the iPhone licensing terms and … well … Adobe and Conde Nast were pretty much fucked. So fast forward to this moment in time and the best short term solution they could come up with was some jury rigged XML based layout framework and an epic shit ton of images.

The Wired app isn’t alone in this weird architectural choice either. The PopSci+ magazine is based on a very similar architecture. There are also other magazines that work along the same lines, or simply go the route of PDFs with a customized PDF viewer application.

The problem with these XML + images architectures is that they are essentially reinventing HTML with no added benefit. When I showed the Wired app to a colleague of mine, someone I consider to be one of the top HTML/Javascript developers in NYC, his assessment was the same: Why the heck didn’t they use HTML5? We stepped through each “page” of the Wired application, looked at each interactive piece – but failed to find anything that ruled out the use of HTML and JavaScript.

The argument might be that it needs to be cross platform – the very thing Adobe and Conde Nast were banking on by going with Flash – but guess what? For all of the tablets about to fall on the heads of consumers in the coming years, each one of them uses WebKit. If anything was built for this type of application, it most certainly is WebKit. And even for harder interactivity puzzles – in terms of how do we do X and Y – one can easily hook into WebKit to enable that stuff that might otherwise be more difficult to do in straight HTML + CSS + JavaScript. I have yet to see anything in any magazine application on the iPad that would really require this though.

So why didn’t they choose HTML5 and build a custom viewer application around WebKit? It comes down to either a sense of desperation, a sense of Adobe overselling a bad idea or simply a dumb technology decision. Possibly all three. It certainly isn’t a development challenge and it certainly isn’t because WebKit isn’t capable. I had a thought that perhaps memory management was an issue, but by going with HTML5/WebKit, you wouldn’t be showing pages and pages of huge images – you’d actually be able to build those pages the right way. And doing it this way, in my professional opinion, the magazine itself would be slashed dramatically in size, as well as acting and reacting in ways familiar to people who’ve been browsing the web for the last 15+ years. Furthermore, the cost savings from a production standpoint would be drastically lower as Wired already maintains a staff of web developers. There wouldn’t be an impetus for Adobe to create some “solution” at Conde Nast’s expense and a lot of the great interactivity you saw in the YouTube videos of their prototype could very easily come to life.

But as it stands now, the Wired iPad app is even far behind their own website. That’s embarrassing. Did anyone at Conde Nast look at this and wonder why someone would choose to use this over their very own website? That iPad – unless you are in a subway – is constantly connected to the intertubehighway. That fact alone makes one wonder what the point of the whole thing is. Specifically since they’ve not done any sort of interactivity or visual presentation that I think anyone can say is amazing. Sure, it’s a print designers wet dream – but it really should be a consumer’s wet dream. And it most certainly is not that.

So, from a technical perspective, I think what we are looking at is the result of equal parts desperation and ignorance. Desperation on the part of Adobe to carry forward their relationship with Conde Nast in this new publishing market and ignorance on the part of their development team for ignoring the best solution to their “No Flash Allowed” problem: HTML5.

Is This The Future For Magazine Publishing?

I hope not.

I actually think it’s a huge step backwards and I think the wrong people are working on the problem – just like the wrong people were working on the web problem back in the day. Sure, we corrected course and we’re seeing the web done correctly more and more these days – but can the magazine publishing industry afford to get this wrong for any amount of time? Once could argue that the internet is quickly making their industry irrelevant. By the time an article is published in Time, I’ve read six or seven different takes on the same story on the web well before it hits the newsstands. I don’t think that’s a unique or new insight. But now you want me to download 500MB a month just so some print designer can have pixel perfect layouts with custom fonts?

Unfortunately, as long as Conde Nast allows Adobe to dupe them into believing Adobe has a solution, it’s going to fail. They need to go outside the box to get this right, and all the technology to do it is sitting right there in front of them. And they’ve been using it very well … until now when they’ve chosen not to even use it all.

Update

People have been mentioning that you can’t do application specific fonts with the WebKit browser. Not sure why.

Here is a simple iPad app that loads up a custom open type font, and then displays an html file in a UIWebView with that custom font: http://github.com/jawngee/iPadFontExample

So it can be done. Enjoy.

Off the top of my head:

• She can read her books in the dark.
• She can use it in the kitchen to look up recipes and watch cooking videos.
• She can use the GPS to help my dad drive their RV around the country.
• She can be better connected to her iPhone wielding grandchildren.
• She can play Words with Friends on a bigger screen.
• She can  browse the web on a readable screen.  While in the RV.
• Watch all of her NFL games while sitting on the porch.

And that’s just to start.  Later down the road, maybe after revision 2, she’ll also be able to:

• Draw her knitting patterns and sync it with her knitting machine via Bonjour over WiFi.
• Have video chats with the kids.
• Augmented reality tour guides for the new places they travel to.
• Telemedical (god forbid) monitoring to save trips to the doctor.
• Control surface for manipulating certain controls of the RV.

The best part?  I won’t have to show her how to do those things, not most of them anyway.  So to do 80% of what she uses her laptop for, she can do all of those things without having to lug the thing out, plug it in, hook up the mouse, etc.  She’ll be able to take the information she is looking for to the place where she needs it to be: the kitchen, the sewing room, the co-pilot’s seat.

So thinking about my mother using it, and loving it, I started to think how I would use it.  My list is almost equally as long.

• Control Boxee.
• Use it as a control surface for Final Cut or Ableton Live or Traktor via Bluetooth or WiFi.
• Home automation.
• The ultimate universal remote (extra hardware required)
• Pornography
• Better in-flight DVD viewing experience

There are so many more integration points the iPad can make with day to day living and day to day information needs.  The key aspect of it all is mobility and portability, having the information you need not only when you need it, but where you need it.

I understand the let down from the perspective of people who were wishing for a multi-touch Mac Book.  But I think they are thinking about it wrong.  People are thinking about it as if it’s a computer, but it isn’t just a computer, it’s a whole new category of device, a whole new definition of what a computer is.

I also agree with most that the lock down to the App Store is a let down, though you don’t have to think too hard about it to understand it’s a necessary evil.  It reduces any potential confusion on how to make the thing do something new.  My Mom and I were playing Words with Friends without me having to walk her through how to install it and get it running.  Now compare that with what it was like to show her how to do something similar on her Windows laptop … yeah no thanks, I’ll take the App Store.  Not to mention not having to run that crappy antivirus and anti-spyware junk – the stuff that sucks the performance out of her laptop and yields endless complaints about how slow everything is.  Perhaps it’s a false sense of security, but I can live with that.

Some of the complaints about usability and related things are complete nonsense and even defy logic.  I’m pretty sure Apple did some usability testing on this thing … but maybe that’s just wild speculation on my part.

I think Apple is definitely onto something, maybe it’s not quite there yet, but they’ve definitely accomplished more than bringing an oversized iPhone to market.  But only time will tell.

And, yes, she does kick my ass in Words with Friends.  She hit me with two 70+ pointers yesterday.

One thing that is truly great about Slicehost is that they offer an awesome API that allows you to do most everything you can do in their management console, but instead through web services. You can create new slices, reboot them, change and add DNS, as well as rebuild or destroy slices.

The only problem is that the API follows a “standard ActiveResource pattern” that comes straight from the bowels of Rails and is way too convoluted for something this simple.

My biggest complaint is that instead of POSTing standard post variables, you POST XML, but none of the data being posted approaches any kind of structural complexity that warrants it’s usage. It just a layer of crap that doesn’t need to be there. Every modern language has some sort of HTTP request mechanism that handles all this stuff without some arbitrary object model forced on top of it.

Curiously, there appears to be another way where you can POST regular form data, but the variable names are in the form “resource[field]” which, again, seems completely unnecessary. I also couldn’t get it to work reliably.

As a point of reference, I was able to implement S3, EC2 and SQS API’s from amazon in just under 6 hours. It took me two days to figure out the Slicehost API. The ActiveResource pattern is a failure in my estimation.

The following are my notes on implementing the slicehost API. Hope you find it useful.

NOTE: This post doesn’t cover their DNS API as I don’t have a particular need for it. I will be posting the PHP library I wrote for the Slicehost API to github over the weekend. Perhaps someone can build on top of that…

Authentication

I actually had a bit of a bitch with this. The docs and ruby samples specify that you put your API key in the url, like this:

https://APIKEY@api.slicehost.com

Problem is that I just could not get it to work. Even the ruby samples in their docs didn’t run without authentication errors. Copying and pasting the URLs into Safari didn’t work either (simply pasting your API key into the user input when the Basic Authentication popup shows does work though).

So, make sure you always set the USER part of Basic Auth with your API key. In PHP it looks like this (using PEAR’s HTTP_Request object):

$request->basic_auth(YOUR_API_KEY,’’);

Dealing With Slices

This is the part that drove me nuts. I ended up having to debug their ruby examples through Charles, after patching activeresource.rb to support proxies. And even then, their examples didn’t work due to a typo and another error I couldn’t sort out.

Listing Slices

This is easy as pie. Simply make a GET request to:

https://api.slicehost.com/slices.xml

You will get back an XML document as a response:

<?xml version="1.0" encoding="UTF-8"?>
<slices type="array">
	<slice>
		<name>slicename</name>
		<status>active</status>
		<addresses type="array">
			<address>XXX.XXX.XXX.XXX</address>
			<address>XXX.XXX.XXX.XXX</address>
		</addresses>
		<id type="integer">666</id>
		<progress type="integer">100</progress>
		<bw-out type="float">0.0</bw-out>
		<bw-in type="float">0.0</bw-in>
		<image-id type="integer">10</image-id>
		<ip-address>XXX.XXX.XXX.XXX</ip-address>
		<flavor-id type="integer">2</flavor-id>
	</slice>
</slices>

The one issue I have with this is that the <address> element doesn’t specify if the address is internal or external. Right now, you can only make the assumption that the first address is external and the second one is internal. Not a big deal though.

Also, note the .xml extension. I believe this is arbitrary, but it would be dope if you could specify .json or .yaml or some other format. Again, not a big deal.

Getting Slice Info

This is equally as simple:

https://api.slicehost.com/slices/666.xml

Where 666 is the ID of the slice you want to get info about. You get another XML document in return containing a <slice> element as above.

Creating a Slice From an Image

This is relatively simple, but there are a couple of steps to it.

First you need a list of the images that Slicehost offers. Easy enough. GET request to :

https://api.slicehost.com/images

Will return an XML document that looks like this:

<images type="array">
	<image>
		<name>CentOS 5.2</name>
		<id type="integer">2</id>
	</image>
	<image>
		<name>Gentoo 2008.0</name>
		<id type="integer">3</id>
	</image>
	<image>
		<name>Debian 5.0 (lenny)</name>
		<id type="integer">4</id>
	</image>
	{ .. and so on .. }
</images>

Once you have this list, you’ll have need to figure out what flavors of slices you can create. Do this by sending another GET request to:

https://api.slicehost.com/flavors

Again, you’ll get another list that looks like this:

<flavors type="array">
	<flavor>
		<id type="integer">1</id>
		<name>256 slice</name>
		<price type="integer">2000</price>
		<ram type="integer">256</ram>
	</flavor>
	<flavor>
		<id type="integer">2</id>
		<name>512 slice</name>
		<price type="integer">3800</price>
		<ram type="integer">512</ram>
	</flavor>
	<flavor>
		<id type="integer">3</id>
		<name>1GB slice</name>
		<price type="integer">7000</price>
		<ram type="integer">1024</ram>
	</flavor>
	{ ... and so on ... }
</flavors>

So now we know the ID for the image and the ID for flavor of slice. Let’s keep going!

This next bit took me awhile to get going. When I tried to go through the method of posting good old post data, I would get back really obscure errors. So debugging the HTTP traffic of the ruby examples, I unearthed the secret sauce.

To create the slice with the image of your choosing, POST an XML document to:

https://api.slicehost.com/slices/

But, make absolutely sure that you have set the “Content-Type” of your request to “text/xml” otherwise it no worky. Your XML document will look like:

<slice>
	<name>NAME OF YOUR SLICE</name>
	<flavor-id type="integer">1</flavor-id>
	<image-id type="integer">2</image-id>
</slice>

The <image-id> must be a valid ID from the list above, of course. Their API docs have a typo where the sample code to create a slice uses an ID of 1, which doesn’t exist. A little hair pulling on that one, even though the bug was so obvious. The same thing goes for the <flavor-id>.

The response you receive will be information about you slice, including the root password. You only get this root password once, so make sure you take note of it. The response looks like:

<slice>
	<status>build</status>
	<name>SLICE NAME</name>
	<addresses type="array">
		<address>XXX.XXX.XXX.XXX</address>
		<address>XXX.XXX.XXX.XXX</address>
	</addresses>
	<id type="integer">666</id>
	<root-password>ROOT PASSWORD</root-password>
	<progress type="integer">0</progress>
	<bw-out type="float">0.0</bw-out>
	<bw-in type="float">0.0</bw-in>
	<image-id type="integer">2</image-id>
	<ip-address>XXX.XXX.XXX.XXX</ip-address>
	<flavor-id type="integer">1</flavor-id>
</slice>

And that’s it, you’ve created your slice!

Creating a Slice From a Backup

This is nearly the same as creating from an image, but this time we’ll need a list of available backups. Backups are things you create through the Slicehost management interface. What I like to do is build a base system and then take a snapshot. With that snapshot I can create new slices based on it.

To grab a list of backups, send a GET request to:

https://api.slicehost.com/backups

You will receive a list (if you have any backups):

<backups type="array">
	<backup>
		<id>3-666</id>
		<slice_id>666</slice_id>
		<name>daily</name>
		<date>Sat May 09 04:15:05 UTC 2009</date>
	</backup>
	<backup>
		<id>3-667</id>
		<slice_id>666</slice_id>
		<name>webserver_snapshot</name>
		<date>Sat May 09 04:15:05 UTC 2009</date>
	</backup>
</backups>

And, then, in the XML document you POST:

<slice>
	<name>NAME OF YOUR SLICE</name>
	<flavor-id type="integer">1</flavor-id>
	<backup-id type="integer">3-666</backup-id>
</slice>

Everything else is the same as creating a slice from an image.

Updating a Slice

The only thing you can update with your slice is it’s name. This is accomplished by PUTing an XML document to:

https://api.slicehost.com/slices

The XML doc looks like:

<slice>
	<id type="integer">666</id>
	<name>NAME OF YOUR SLICE</name>
</slice>

All you need is the ID of the slice and it’s new name. The response will be an XML document with a <slice> node.

Deleting a Slice

Delete a slice by sending an XML document using the DELETE HTTP verb:

<slice>
	<id type="integer">666</id>
</slice>

NOTE: You will have to have this option enabled in the API settings on your account. You can edit those settings at: https://manage.slicehost.com/api/

Rebooting Your Slices

To reboot your slice with a hard reset, send a PUT request to:

https://api.slicehost.com/slices/YOUR SLICE ID/hard_reboot.xml

To do a soft reboot, a PUT request to:

https://api.slicehost.com/slices/YOUR SLICE ID/reboot.xml

Rebuilding Your Slices

To rebuild your slice, you’ll need to use an image ID or a backup ID. Once you’ve picked one, send a PUT request to:

https://api.slicehost.com/slices/YOUR SLICE ID/rebuild.xml?image_id=X

or:

https://api.slicehost.com/slices/YOUR SLICE ID/rebuild.xml?backup_id=X

Conclusion

So that’s all I know. As you can see the API is pretty simple, but not as simple as it could be, nor are the documents even remotely near as useful as they should be. But, it’s awesome they have this API in the first place so I’ll stop bitching now!

Again, thanks to Slicehost for making awesome. The world sure does need it.

There are some caveats.  You will need to be mildly technical to pull this off, but Google is there to hold your hand through some of the tough stuff which, honestly, isn’t that tough.  And while some of the concepts might fit Microsoft developers, this article is Linux/BSD/Unix focused.  I will tell you, though, from years of being a .NET consultant, trying to bootstrap a Microsoft centric solution is futile if you’re trying to sell your ideas as cheaply as possible.  I’m not trying to start a flame war, I’m just sayin’.

Register Your Domain

Let’s start at the top.  You’ve got a great idea for a website.  You’ve done the leg work, written up your business plan and are eager to get going on actually building out the product and the supporting internals for the company. The first thing we’re going to need to do is register a domain.

You have a variety of options here, and depending on the domain and it’s TLD, you’ll have various pricing levels to work with.  For the sake of the argument, let’s assume you are going to register a .COM instead of the more expensive .TV, .ME, .JOBS and others.  Any one of these is a fine registrar:

• namecheap.com - $9.68
• godaddy.com – $9.99
• powerpipe.com – $10

I also prefer to do private domain registration, which can add $15 to the purchase.  This is entirely optional but it’s a good way to protect your private information.  If you are registering a .US domain, though, you can’t do private registrations.  Also, some places protect the privacy of your registration differently, so do some research if this is important to you. Namecheap has free private domain registration.  Godaddy charges extra and I’m not sure about PowerPipe.

Startup cost: $10

DNS Made Easier

This part is entirely optional, but after the register.com fiasco, it’s worth the peace of mind to run your DNS through a third party.  I love DNSMadeEasy for a variety of reasons, chiefly DNS changes are live nearly instantly, but they also scale incredibly well, providing options for doing round robin DNS to balance DNS requests between load balancers, as well as fail over monitoring.  They also have an API which you can use in your applications for all kinds of neat stuff.

Best part?  Cheap.

Run Rate: $0.08 a day

The Nuts and Bolts

So we have a domain now.  The next thing we’ll need to do is setup some infrastructure so our team can communicate and collaborate as we start building this mother out.

I will confess that I made a very dumb move when we were getting Massify going.  Having spent the prior 7 years as a .NET consultant, all I really knew was Windows and the host of enterprisey things you find on Windows.  Having done some SharePoint consulting, I decided SharePoint was the way to go.  An avid Outlook and Blackberry user, the only option I could see through my rose tinted glasses was Microsoft Exchange.  So we hooked up Microsoft Exchange with BES for my blackberry addiction and off we went to focus on bigger and better things. Fast forward a year and a half and the company has grown to 14+ people and our monthly hosted Exchange bill was close to $450 a month.  The worst part?  Massify is an entirely Apple OS X shop.  The only Windows machines we own are the ones we run in virtualization software.  So while we were running Exchange (and have all moved on from Blackberries to iPhones) we were not reaping any of its benefits.  No calendaring, contact sharing, none of that.  Had we been on Exchange 2007, we might have been able to capitalize on some of those basic things, but we were stuck on Exchange 2003 which doesn’t work with any OS X productivity apps without some spendy third party utilities that only sometimes work.

So, yeah, dumb move Gilkison!

Well I am here to tell you that you can get NEARLY all the benefits of Exchange for FREE.  Yes, you read right:  FREE.  My friends, I would like to introduce you to Google Apps for Business.

Shared calendaring?  Check.  Contact syncing?  Check.  Kick ass web interface for reading/writing mail?  Check.  IMAP support for desktop and handheld clients?  Check.  Push email?  Negative.

I did say NEARLY.

And the best part, beyond the awesome price tag?  I don’t have to install/maintain shit. I’ve been test driving Google Apps for Business for awhile for my personal stuff, to make sure that it has legs, and I couldn’t be happier.  So we transitioned off Exchange and we are all now much happier individuals.

Run Rate: FREE

Working Together

There are a ton of collaborative apps out there and they’re all like gloves.  Each has their own utility and fits its wearers in different ways.

The most popular, of course, is BaseCamp from 37Signals.  I won’t get into what I think of 37Signals, but suffice to say a lot of people have a huge app crush on BaseCamp.  I personally, and this is just my opinion, don’t particularly care for it.  Everything starts out well, but once you get into the thick of big projects it becomes a bit of a mess.  But, like I said, different strokes for different folks.  Nothing I’m preaching here is gospel, so find out what works best for you.

Now you get a variety of collaborative tools with the Google thing, in addition to the email.  Google docs is great for document sharing, but the other apps we’re not so keen on.  We’ve found a nice balance between PivotalTracker and PBWiki.  Both apps cost money and there are a lot of free, open source alternatives.  In fact, we use Deki Wiki from Mindtouch for our development wiki.  For agile development, there are quite a few free backlog management apps, but Pivotal is easy enough for everyone to use, so that’s what we went with.

However, the goal of this article is to do things cheaply, but functionally.  There is no open source equivalent of BaseCamp, so let’s price with BaseCamp in mind.

Run Rate: $0.78

Let’s Talk Hosting

The following is based on the assumption that you’re planning to build a web app based on today’s standards.  This means you’ll be using some kind of framework like Rails, or a language like PHP, with a database backend like PostgreSQL.  Yeah I could have said MySQL, but why when PostgreSQL exists?

I imagine this will be the most “controversial” aspect of this article because every techie has their own vision of the right way of setting things up.  I also want to talk to you about planning for scale.

“Premature optimization is the root of all evil” is a phrase people like to bandy about when people approach the topic of planning for scale before the need for it.  Well, first things first, the phrase is often taken way out of the context in which it was used.  Obviously you don’t need to unroll your loops in PHP, because listen, you’re using PHP and optimizing it’s performance doesn’t have a substantial effort/reward ratio.  Same goes with Rails/Ruby and Django/Python. But, it doesn’t mean that you can’t plan to be successful and get things in the right position for when you do need to scale.  There are some easy things you can do to make sure when the need comes at you full bore, you’ll be ready for it.

So let’s talk about hosting.  The Web 1.0 way was to colocate racks of metal in a datacenter somewhere or to pay for managed hosting through someone like RackSpace.  But that is a crazy thing to do now, it only makes sense when your site hits a certain threshold of scale.  If you’re at that scale now, what are you doing reading this article?

We’re working under the pretense that your business idea has to prove itself in the marketplace and that you need to get it into the marketplace as cheaply as possible.  So what options do we have? You can sign up for a VPS like dreamhost, but you’ll be cursing yourself when your site hits the front page of Digg or Reddit and the social news monkeys are kicking the shit out of your application.  A day is forever in internet time, so what are you going to do when you need to take your site down for a couple of days to migrate it to something else?

The next step up, then, is this “cloud” thing everyone is talking about.  And it’s the right step.  However, a number of people instantly assume we are talking about Amazon’s EC2 offering, and that is the wrong way to go.  Bandwidth on Amazon is not the cheapest thing around and I’m of the fundamental belief that EC2′s best purpose is for bursts of additional computing resources.  For instance, on Massify, we use EC2 to do parallel processing of media.  We can scale it up and down at will, taking it entirely off-line if we need to.  Having priced out hosting our site on EC2, the end costs were not that far off of managed hosting. So where does that leave us?  I heartily recommend SliceHost.  And I recommend the following starting setup:

Obviously, this isn’t the absolute cheapest way to start out, but it prepares you for growth.  If you develop with a share nothing architecture, you can easily slot in more application servers as needed, and take them down as needed, until such a point as your database becomes a bottleneck.  But that shouldn’t happen for awhile and the total cost for this rig on Slicehost?  $160 a month.  If you were to stuff it all into a single slice, you’d go for their 4GB or 8GB slices which run $250 and $450 respectively.

Run Rate: $5.20

Spread The Love Around

Like the DNS management, this step is entirely optional, but if you’re serving a ton of static content or media, I can’t imagine going without for very long.  The advantage of serving all your static content (this includes static html pages, javascript, css, images, media) is that it’s way cheaper than serving off even Slicehost and it takes a ton of load off.

Good news for you is that CDN’s are getting cheaper and cheaper.  We use Panther Express at Massify and our monthly bill is amazingly low.  I can’t really publish those numbers, so let’s talk about Amazon’s S3 and CloudFront pricing to get a rough number.  The basic way this works is that all of your static content gets pushed to S3 and then served up to your users via CloudFront.  CloudFront has a lot of nice features, though if you’re planning on doing video through it, you might look elsewhere as they don’t support HTTP chunking – which is the poor man’s way of doing video streaming and seeking.  Panther Express does support this, though BitGravity has some awesome looking capability as well.

Anyways, assuming we’re going to transfer 250 GB of data via our CDN, the numbers will surprise you:

S3 Storage and Transfer: $37.50 for 250GB

CloudFront delivery: $42.50 for 250GB

Combined, you would be paying roughly $80 a month for 250GB of data transfer.  For most sites that’s overestimating, but if you’re doing a user content generated site, you should probably aim higher.  I’m not at liberty to divulge how much we transfer per month, but it’s significant.  And despite that our CDN bills are shockingly low.

Run Rate:  $2.63

Total It Up

So we’ve covered everything from domain registration to company infrastructure to hosting and content distribution.  Here’s how the costs breakdown:

For a grand total of: $8.69 a day. Not too shabby.

So those are my tips based on my experience for the last two plus years.  Would love to hear comments and tips from others!

Some Background

For the last 2+ years, we’ve been running Apache with mod_php at Massify. And, for the most part, it’s been painless and hassle free. But as traffic increases, we’ve noticed Apache struggling to keep up. It chews up memory like crazy and don’t even get me started about CPU. Add in the fact that we are currently moving the site from co-located metal to hosting on the cloud, we’ve needed to find a faster way to serve up these pages on machines that are a quarter of the spec we had with the metal.

It’s a pretty well know fact by now that Nginx (pronounced engine-x, though I call it n-jinx) in typical scenarios outpaces Apache on all kinds of fronts: i/o, cpu, memory, reqs/sec. Feel free to google around for more comparitive information. I can tell you from my own informal load testing on both setups of Massify that I’ve seen a pronounced difference between the two, specifically in the configuration I’ll be writing about in this post. I won’t be posting numbers, because my testing wasn’t scientific and it’s not really the focus of this article, but I am more than confident that we’ll be getting a level of performance a few steps above Apache. (Please don’t write me telling me I can recompile and reconfigure Apache to approach Nginx’s performance. I’m certain you are correct, but I don’t really care at this point.)

PHP-FPM?

The typical nginx configuration involves using spawn-fcgi from the LightTPD project to get nginx serving up PHP. There are a few issues with spawn-fcgi that are covered on more in-depth posts across the interwebs, so I won’t rehash them here. Suffice to say, it can lead to major headaches, so we want to do what we can to avoid those headaches.

Enter PHP-FPM, which stands for PHP FastCGI Process Manager. It is actually a set of patches for the PHP source that bake in FastCGI process management into your PHP binaries.

Note: Even if you are sticking with Apache, there are a variety of reasons to skip mod_php and go straight for PHP via FastCGI. For starters, with mod_php, each request that Apache handles loads PHP – and all of it’s libraries. That’s an epic shit ton of overhead. With FastCGI, PHP acts more akin to an application server. PHP-FPM, as well as spawn-fcgi, manage this; loading and killing PHP instances as needed. This has a lot of benefits, not the least of which is reduced memory overhead.

Let’s Get Rocking

We’re using a fresh install of Ubuntu 8.10 Intrepid. With a little tooling, this can work equally well on other linux distros. I’ve deployed this on CentOS, with minor changes, just fine.

First things first, we’re going to have to install all the dependencies for building everything:

sudo apt-get install make bison flex gcc patch autoconf subversion locate
sudo apt-get install libxml2-dev libbz2-dev libpcre3-dev libssl-dev zlib1g-dev libmcrypt-dev libmhash-dev libmhash2 libcurl4-openssl-dev libpq-dev libpq5 libsyck0-dev

Now that we have everything we need, now it’s time to …

Compile PHP

We’re going to download the source for PHP 5.2.8, as well as the matching patches for PHP-FPM. We’ll then apply the patches and get compling.

cd /usr/local/src/
sudo wget http://us.php.net/get/php-5.2.8.tar.gz/from/this/mirror
sudo tar zvxf php-5.2.8.tar.gz
sudo wget http://php-fpm.anight.org/downloads/head/php-5.2.8-fpm-0.5.10.diff.gz
sudo gzip -cd php-5.2.8-fpm-0.5.10.diff.gz | sudo patch -d php-5.2.8 -p1
cd php-5.2.8
sudo ./configure --enable-fastcgi --enable-fpm --with-mcrypt --with-zlib --enable-mbstring --disable-pdo --with-pgsql --with-curl --disable-debug --enable-pic --disable-rpath --enable-inline-optimization --with-bz2 --with-xml --with-zlib --enable-sockets --enable-sysvsem --enable-sysvshm --enable-pcntl --enable-mbregex --with-mhash --enable-xslt --enable-memcache --enable-zip --with-pcre-regex
sudo  make all install
sudo  strip /usr/local/bin/php-cgi

The above should have been completely auto-pilot. If you run into any errors, more than likely you don’t have some of the dependencies installed. Note: Make sure you enable and disable whatever PHP configuration options you need for your particular setup. The above are what we use at Massify, more or less. (We actually apply a couple of other custom patches, but that’s another post for another day).

While we are at it, let’s install some of the modules we’re going to need via PECL:

sudo pecl install memcache
sudo pecl install apc
sudo pecl install syck-beta

Now, when you build the APC extension, make sure you turn off the option to compile for apache. It will prompt you to make this change.

Let’s copy the stock php.ini:

sudo cp /usr/local/src/php-5.2.8/php.ini-recommended /usr/local/lib/php.ini

Finally, let’s setup symbolic links to make things easier to find:

sudo mkdir /etc/php/
sudo ln -s /usr/local/lib/php.ini /etc/php/php.ini
sudo ln -s /usr/local/etc/php-fpm.conf /etc/php/php-fpm.conf

That’s it for compiling PHP. The only thing we have left to do is change some setting in the php-fpm conf. Open up /etc/php/php-fpm.conf in your editor of choice. Do some searching and change the users who own the processes to www-data. The file is too big to post in here, but the values we want to change are on lines 51, 52, 63 and 66. Should look something like:

<value name="owner">www-data</value>
<value name="group">www-data</value>
<value name="user">www-data</value>
<value name="group">www-data</value>

Done!

NGINX

This is just as easy as compiling PHP was:

cd ..
sudo wget http://sysoev.ru/nginx/nginx-0.6.35.tar.gz
sudo tar zxvf nginx-0.6.35.tar.gz
sudo rm -f nginx-0.6.35.tar.gz
cd nginx-0.6.35
sudo ./configure --sbin-path=/usr/local/sbin --with-http_ssl_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module
sudo make && sudo make install

Let’s set up some links:

sudo ln -s /usr/local/nginx/conf /etc/nginx

Now the next step is optional, but this is what I’ve been using so far. Let’s open up /etc/nginx/nginx.conf in our editor and go to town. Salt and pepper to taste:

user  www-data;
worker_processes  6;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  10 10;

    gzip  on;
    gzip_comp_level 1;
    gzip_proxied any;
    gzip_types text/plain text/html text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript;

        log_format main '$remote_addr - $remote_user [$time_local] '
                  '"$request" $status  $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

        access_log  /var/log/nginx_access.log  main;

        error_log  /var/log/nginx_error.log debug;

        include /usr/local/nginx/sites-enabled/*;
}

We’re also going to have to set up some FastCGI parameters so that PHP doesn’t choke and you’ll avoid random 503 errors from nginx. Open up /etc/nginx/fastcgi_params and add the following:

fastcgi_connect_timeout 60;
fastcgi_send_timeout 180;
fastcgi_read_timeout 180;
fastcgi_buffer_size 128k;
fastcgi_buffers 4 256k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
fastcgi_intercept_errors on;

Finally, let’s create a SystemV style init script and store it in /etc/init.d/nginx. This script was lifted from here.

#! /bin/sh

### BEGIN INIT INFO
# Provides:          nginx
# Required-Start:    $all
# Required-Stop:     $all
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: starts the nginx web server
# Description:       starts nginx using start-stop-daemon
### END INIT INFO

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/local/sbin/nginx
NAME=nginx
DESC=nginx

test -x $DAEMON || exit 0

# Include nginx defaults if available
if [ -f /etc/default/nginx ] ; then
        . /etc/default/nginx
fi

set -e

case "$1" in
  start)
        echo -n "Starting $DESC: "
        start-stop-daemon --start --quiet --pidfile /usr/local/nginx/logs/$NAME.pid
                --exec $DAEMON -- $DAEMON_OPTS
        echo "$NAME."
        ;;
  stop)
        echo -n "Stopping $DESC: "
        start-stop-daemon --stop --quiet --pidfile /usr/local/nginx/logs/$NAME.pid
                --exec $DAEMON
        echo "$NAME."
        ;;
  restart|force-reload)
        echo -n "Restarting $DESC: "
        start-stop-daemon --stop --quiet --pidfile
                /usr/local/nginx/logs/$NAME.pid --exec $DAEMON
        sleep 1
        start-stop-daemon --start --quiet --pidfile
                /usr/local/nginx/logs/$NAME.pid --exec $DAEMON -- $DAEMON_OPTS
        echo "$NAME."
        ;;
  reload)
      echo -n "Reloading $DESC configuration: "
      start-stop-daemon --stop --signal HUP --quiet --pidfile /usr/local/nginx/logs/$NAME.pid
          --exec $DAEMON
      echo "$NAME."
      ;;
  *)
        N=/etc/init.d/$NAME
        echo "Usage: $N {start|stop|restart|force-reload}" >&2
        exit 1
        ;;
esac

exit 0

Don’t forget to set executable permissions on it!

Set Up Your Site

This part you’ll have to tool yourself, but here’s a rough guide.

First we’ll have to create a directory to store our site(s) conf files:

sudo mkdir /usr/local/nginx/sites-enabled
sudo ln -s /usr/local/nginx/sites-enabled /etc/sites

And now let’s add a conf file for our default site at /etc/sites/default.conf. The contents:

server {
        listen *:80;

        location / {
                root   /var/www/default/pub;
                index index.php;

                # if file exists return it right away
                if (-f $request_filename) {
                        break;
                }

                # otherwise rewrite the fucker
                if (!-e $request_filename) {
                        rewrite ^(.+)$ /index.php$1 last;
                        break;
                }

        }

        # if the request starts with our frontcontroller, pass it on to fastcgi
        location ~ ^/index.php
        {
                fastcgi_pass 127.0.0.1:9000;
                fastcgi_param SCRIPT_FILENAME /var/www/default/pub$fastcgi_script_name;
                fastcgi_param PATH_INFO $fastcgi_script_name;
                include /usr/local/nginx/conf/fastcgi_params;
        }
}

The conf file above is for front controller style sites, which include wordpress, cake, etc. It also takes care of serving up static content without going through FastCGI. Note: You will need to change /var/www/default to wherever the files for your site reside.

Start It Up

Now all we have to do to get it rocking:

sudo php-fm start
sudo /etc/init.d/nginx start

You should be able to test your site out and see it working. Any questions, feel free to ask.

Reference

• Nginx
• Nginx vs Apache Death Match
• 10,000 req/s with Nginx
• Replacing Apache With Nginx For Static File Serving
• PHP-FPM
• How to Install PHP-FPM
• PHP-FPM, A Smoother PHP FastCGI Process Manager

…

Back in April of 2008, I came across a proposal by Daniel Glazman and David Hyatt for using variables in CSS stylesheets.  I thought the proposal was absolutely brilliant, filling a much needed void for sites using complicated stylesheets across a variety of different pages.  Another part of their proposal was being able to include/import other stylesheets.  I don’t know anyone that couldn’t find this useful.

I put together a quick class for implementing most of their proposal using PHP.  Nothing fancy going on here, most of it is some simple regexes.

Download the code here.

Using the CSS Compiler

Let’s first talk about how to setup your CSS stylesheets to use variables and imports.

Important: Any stylesheet that uses variables or includes stylesheets that use variables must end with the extension ‘.cssp‘.

To declare variables, you place them within an @variables at-rule like so:

@variables
{
    titleFont:bold 18px "Verdana, Arial";
    titleBG: #FF0000;
}

To use the variables:

div.title { font: var(titleFont); background: var(titleBG); }

Couldn’t get much simpler.

Importing CSS and Variables

You can also import/include css and variables from an external stylesheet using @import:

@import "vars.cssp";

div.title { font: var(defaultFont); };

Alternately:

@import url("vars.cssp");

div.title { font: var(defaultFont); };

Evaluating Expressions

You can also perform math operations with measurement units. For instance, you could do something like:

@variables {
    lc_width: 240px;
    rc_width: 180px;
    all_padding: 20px;
}

#some_div {
    width: eval((lc_width+rc_width)-(all_padding*2));
    padding: eval(all_padding/2);
}

You can only do this with px/em/% measurement units and the units must match. For instance you can’t add a variable that is 2em with another that is 120px. Doing so will throw an exception. Also, if you use a variable reference that doesn’t exist, it will also throw an Exception.

Using in HTML Pages

To use these stylesheets, you’re going to have to write a little PHP, but not too much:

<?
    // include the compiler
    include 'css_compiler.php';

    // set the compile environment variable
    // otherwise it will spit out a static file
    // in a production environment comment this out:
    define('COMPILE_CSS',true);
?>
<html>
    <head>
        <? style(dirname(__FILE__),'testing'); ?>
    </head>
    <body>
        <div class="nice">Hello World</div>
    </body>
</html>

At the top of this file, we’re including the css compiler and then defining a global variable COMPILE_CSS.  COMPILE_CSS tells the compiler to do the compilation and save the resulting .css files.  You will not want to do this in a production environment, so comment out the define when deploying to the masses.

In the head tags, the style() function is a helper function that compiles the css.  The first argument is the full path to where the css is.  The second argument is the css file to use without it’s extension.  Note: if your css files have a relative path of /css/whatever.css on your webserver, the relative path should be part of the second argument and not the first.

For example, let’s say our PHP application is located at /var/www/.  Our css files are located at /var/www/css/.  Our style function is going to look like:

style('/var/www/','/css/testing');

Our actual usage of this code at massify is way different.  I’ve added the style() function so that people reading this blog can use it, but you should definitely do your own legwork to blend it into whatever framework you are using.  This should be enough to get started however.

Also, make sure that your css directory is writable by your webserver or the whole thing won’t work at all.

Leave a comment or drop me a line if you find this useful/find bugs/have suggestions.

Enjoy!

One of the things I miss the most (besides the sanity) are attributes (annotations for Java peeps). To be able to ascribe metadata to class, method and property definitions opens up a whole new world of introspection which enables you to do some pretty wicked hacks.

In this post, I present a PHP class that allows you to do metadata/attribute programming with PHP.  You can download the class here.  But before we dig in, we must understand what attributes are and how they are useful…

Introduction to Attributes

For those that don’t know about attribute based programming, let me quote an excerpt from OnDotnet:

Attributes are a mechanism for adding metadata, such as compiler instructions and other data about your data, methods, and classes, to the program itself. Attributes are inserted into the metadata and are visible through ILDasm and other metadata-reading tools.

Reflection is the process by which a program can read its own metadata. A program is said to reflect on itself, extracting metadata from its assembly and using that metadata either to inform the user or to modify its own behavior.

One of the last things I wrote in C# before switching over was a simplistic REST-esque framework for web services for C#. I used attributes on classes to mark which ones were exposed to the outside world and then used attributes on the methods to specify such things as the URI, description, etc. This allowed the services to be discoverable versus declared and to be self documenting. For example:

namespace SimpleWebService {
[RestService(baseURI='/simple/',Name='Simple', Description='Simple service', Persistent=false)]
class Service
{
[RestMethod(URI='something',Name='Do Something',Description='Does something.')]
public void DoSomething()
{
// …
}
}
}

In the above example, the attributes are declared in the brackets.  On our Service class, we give the service a name, define it’s base URI, give it a description and declare if the service is persistant between requests (meaning that the class is only instantiated once, or is instantiated every time).  On the DoSomething method it’s a similar deal: we declare the URI endpoint for the method, the name of the method and it’s description.

By going to http://example.com/simple/something would eventually invoke the DoSomething method on our Service class.  The beautiful thing here is that my classes needn’t implement any interfaces or descend from any parent classes to be exposed as a web service.  I could drop the attributes on the class, recompile and they would then be instantly available.  When the application loaded, it scanned all of the classes in the loaded assemblies, did some reflection on them and created a cache of exposed services.  An HTTPHandler would dispatch the incoming requests to the corresponding service, map any POST/GET or URI fragments to the parameters of the method being called and then returned the results of the method call as serialized XML.

Nice and easy.

Attributes and PHP

PHP has no built-in mechanism for declaring attributes, but it does have a primitive retrospection capability.  For any class or function in PHP, there are a set of functions that you can call that will provide information about the class.  For instance, with methods, you can get it’s parameters and any associated block of comments:

class DumbClass
{
    /**
     * Comment block
     */
    public function thing($parameter,$another)
    {
        return false;
    }
}

$method=new ReflectionMethod('DumbClass','thing');
echo $method->getDocComment();

In the above example, we create an instance of the ReflectionMethod class and echo the comment block.  If we run this script, the output would look like:

    /**
     * Comment block
     */

Now if we put 2 and 2 together, you’ll see that we could use the comment block to insert our class metadata.  Now all we need …

Introducting AttributeReader

The AttributeReader class is a simple class that extracts YAML from the comment block on a class, it’s methods or properties.  You can download the class here.  Note:  You must have the pecl syck package installed, details are here.

Since we have access to the comment block, it’s the most obvious place to express our metadata.  Obviously, the 100% correct solution would be to patch PHP to support attributes out of the box, but that’s a much bigger effort that would require some serious coding.  The performance of this method is fast enough for most use cases.

So how do we use this?  At massify, we use metadata for our ORM layer.  All of our models have metadata attached to them that define what database table to use, the column name, etc.  Here’s an example:

/**
 * Sample model
 *
 * [[
 * table: sample.item
 * database: default
 * read_only: false
 * ]]
 *
 */
class Item extends Model
{
    //@ fields

    /**
     * [[
     * label: Title
     * type: string
     * length: 32
     * description: Title of the item
     * validate:
     *   required: true
     *   length: 8-32
     *   unique: true
     * ]]
     */
    public $title;

    /**
     * [[
     * label: URI
     * type: string
     * length: 32
     * description: URI of the item
     * validate:
     *   required: true
     *   length: 4-32
     *   unique: true
     *   format: alpha_numeric
     * ]]
     */
    public $uri;

    /**
     * [[
     * label: Description
     * type: text
     * description: Description of the item
     * ]]
     */
    public $description;

    //@ end fields
}

In the above example, our metadata is nestled between double brackets [[]].  Inside the double brackets is YAML expressing the metadata.  The cool thing here is that we can nest attributes of the metadata.  Let’s look at what we’ve done:

On the declaration for the Item class, we’ve described metadata that tells us which database table this model represents, which database it resides in and if it’s read_only (a view, in database parlance).  On the properties of the model, the metadata describes the label to use for forms, what the database type is (simplified), the length (for string types), a description and a list of validators.  For instance, on the $uri property, we’ve declared validators that make sure the property has a value before saving (required), is of a specified length (4-32 characters), is unique in the database (unique) and matches a specific format (alpha_numeric).

Since the class extends from Model, the model knows to use the metadata to build the correct SQL statements for insert, updates, etc.  It also knows how to validate itself when saved to the database, returning the correct errors to the user if any of the validations fail.  Finally, we are able to automatically build forms to edit the models by using the metadata as a guide when constructing the form in code.

All of this is completely possible to do without using metadata, but would require a lot of redundant code with a predetermined set of use cases.  Metadata frees us from this because we can write use cases that work with a known quantity, rather than the other way around.

Using the AttributeReader

Using the class is straight forward.  Here’s an example.  Let’s assume we have the Item model from the previous example loaded and want to extract metadata for it and it’s URI property:

$item=new Item();

$class=AttributeReader::ClassAttributes($item);

echo $class->database;
echo $class->table;

$method=AttributeReader::PropertyAttributes($item,'uri');

echo $method->label;
if ($method->validate->required)
    echo "Required field";
else
    echo "Not required.";

It’s that simple.

Caveats

Make sure you are familiar with the rules of YAML.

Performance is acceptable, but always make sure you measure performance for yourself.  We use a caching strategy via APC at massify for caching model metadata.  This requires that you restart apache if your models change (you can disable APC on your development/staging to get around this).  With the APC caching strategy, it’s super fast.  Find out what works best for you.

I hope people find this useful, I think it opens a whole new world for serious PHP development.  Feel free to drop me an email if you have questions or successes.