How to Get Open Source Project Ideas and Execute Them

A common question people ask me is how I come up with open source project ideas. It is a pretty simple process:

1. Have A Problem
If you are a developer, when you are building stuff you will have to solve problems. On top of those, as a human being you will have problems every day. How can I share responsive web design examples to educate my clients? How can I make a more flexible grid system for responsive web design? How can I help my kid learn his ABC’s? Are my passwords manly enough?

The first thing most of us do is see if that problem has been solved before, and how. It may be that the problem has been solved effectively, and there is no need for you to solve it again. Many times, there are different approaches to solving the problem, but none that work quite the way you’d like. On rare occasions, you have encountered a problem that doesn’t have any good solutions yet, which can be quite exciting.

1a. Come Up With a Cool Name
This is probably the most fun part of the process, haha. Cool names are cool!

2. Solve The Problem
Get on it. Solve that problem. It could be that solving a particular problem is too large and ambitious to pull off. Either find some friends to collaborate with, or break the problem into smaller components then go after them one at a time.

3. Abstractize Your Solution
When possible, craft your solution in such a way that it can be applied more generally, for people who encounter the same problem.

4. Publish
Throw it on Github, of course, but if you want your project to stand out, go the extra mile. Make a great demo page to showcase your project with some clear, easy-to-follow documentation.

5. Don’t Be Shy
If you release a Github project in the woods, does anyone notice it? To get your project noticed, you need to publicize it. Tweet it out and share it up. Send it off to newsletter publications (e.g. JavaScript Weekly, HTML5 Weekly, etc.) Even better, create a tutorial and try to get it published online somewhere like Codrops, Tuts+ or Smashing Magazine.

6. Repeat
The more you projects you publish, the more ideas you will have. I recommend maintaining a list of your project ideas on Google Drive or Trello.

One Year of Open Source Traffic

Screen Shot 2014-02-02 at 1.36.13 PMI’ve put Google Analytics on some of my Github project pages, but hardly ever look at it. Well, today I decided to take a look at a year’s worth of traffic, and I am absolutely blown away.

In one year, there were 743,557 unique visitors to my various open source projects. SuperScrollorama was the most popular with 487,475 pageviews. Unbelievable! And I noticed that I wasn’t even tracking Responsivator or BigVideo.js, so it was undoubtedly a lot higher, probably over a million. Amazing.

AWS Notes: Mastering NoSQL – Advanced Amazon DynamoDB Design Patterns for Ultra-High Performance Apps

DynamoDB Engineer David Yanacek started things out by giving an overview of the types of tables and queries that exist in a typical social network app, and how they look in DynamoDB. A second example he gave was image tagging, doing queries like images by users, by date, by tags and then combinations of those. He showed how to set up DynamoDB queries that would be useful and inexpensive for this type of datasets. He introduced their new concept of a Local Secondary Index, which is a temporary table created by DynamoDB, that allows you to run a secondary lookup on a result set. For tags, in DynamoDB you would have another table tie the tags to image IDs.

Then he revealed a new feature for DynamoDB: Global Secondary Index. This lets you do fast, inexpensive with more flexible queries.

The next example was managing states of a tic tac toe game, show some problems that arise and how to avoid them. David showed a problem in which a tic tac toe player could cheat by sending multiple update item requests, perhaps in separate browser windows. You can solve for that by doing conditional writes. Another fix was to add an expect clause to prevent bad writes.

He moved on to talking about finer-grained access control for user data. He proposed moving from a three-tiered architecture, to two-tiered where users do updates directly to DynamoDB rather than a service layer. Having just heard the ___ presentation, I knew where he was going with this: using Web Identity Federation to get temporary credentials and a session token. A new feature in DynamoDB is to limit access to particular hash keys and attributes. For example, allow all authenticated Facebook users to query the images table but only when their facebook id matches their user id. This allows us to expose the database directly to users, resulting in lower latency, cost and complexity. Some problems with this is less visibility into user behavior, more difficult to make changes without a service layer and requires work in scoping items to specific users.

Next up was David Tuttle, Engineering Manager of Devicescape. His talk would cover how they have implemented DynamoDB in their service. Prior to adopting DynamoDB, they faced challenges in scaling, DB maintenance, slow queries on large table and difficulty making schema changes. DynamoDB forces a different way of thinking about data.

He quickly gave a bunch of pro-tips: Organize data into items around a primary key, evenly distribute workload across hash keys, use parallel scans for high speed jobs. DynamoDB will not give you the expected throughput unless you distribute across hash keys. By using conditional writes and an atomic increment in the first item, they avoid collisions. Plan for failure. Use local files to resume data operations after a failure. Extract data to S3 to do non-realtime operations like analytics. Use DynamoDB local to speed up developer workflow.

Last it was Greg Nelson from Dropcam, the Wi-Fi enabled camera company that does intelligent activity recognition and cloud recording. They moved away from managed hosting to AWS and eventually DynamoDB. He showed how they use it to manage all their camera data, as well as user sessions. He drilled into what their table updates, queries and deletes look like for cuepoints (to keep track of important footage within the video tracks).

He talked about some of the complexity within DynamoDB, such as the concept of “eventual consistency”. With NoSQL, choosing a hash key is the most important up-front design decision. You need to keep very close attention on IOs. Throttling behavior is opaque, so you need to actively watch for problems.

AWS Notes: Writing JavaScript Applications with the AWS SDK

AWS Developer Loren Segal introduced the AWS SDK for Node.js It is open source, apache-licensed and on Github. They have full service coverage across 30 services.

Install through npm, of course:
npm install aws-sdk

Configuration is easy and can be done programatically (there’s a global configuration object). Loren ran through a bunch of code snippets for how to do various things. It wasn’t long before he got into some live coding in the terminal. First up some simple S3 requests and response handling. Next, his next livecoding example demo’d response pagination for scanning large DynamoDB datasets. Then he showed request event handling, and demonstrated the AWS global events object, allowing you to do global event handling on request success response for example. Last, he went over programmatically configuring credentials, showing that you can set temporary creds that automatically expires.

Loren moved on to show off the AWS SDK for JavaScript in the Browser which is now in developer preview. This literally allows you to build web applications with static files. Some examples of things that could be easily created are forum software, blogging, commenting systems, browser extensions and mobile apps. This is freaking awesome.

He created a sample blogging application (check it out on github). The app.js script is a mere 260 lines of code.

Loren live deployed the app. It had permissions set up, so he did a social login, then created a blog post using markdown and an open source WYSIWYG editor. Did I mention this is all with static HTML/CSS/JS files?! Nice. You can see the demo at (demo link no longer available).

He went over the key differences between doing traditional three-tiered architecture vs two-tier development with AWS in the browser. He did say that in order to use it with S3, you need to configure CORS settings. For other services, CORS is not necessary because the requests for them are already authenticated.

He stressed the importance of not hardcoding your credentials. They get around this by using Web Identity Federation which trades access tokens provided through other services like Facebook, Google or Amazon for AWS keys. He showed quickly how to set up a Facebook application to create IAM roles and set up permissions for users. He showed some code for working with Facebook Access Tokens.

He finished by talking about the open source community. They love to get feedback, pull requests, issue reports and third-party plugins.

Open Source For The Wins

awwwards-articleI was browsing around some web design award sites today. So much inspiring stuff out there to be found. What do all these beautiful and often amazing award-winning websites have in common? They all use open source projects from Github and elsewhere. As someone who releases open source, it is thrilling when you view the source of one of these sites and see your own code mixed in there.

So I want to thank all the people out their using my code to help them power their creations. Knowing that my projects are being used by talented people all over the world is both tremendously motivating and also a bit mind-blowing.

Here’s a list of some of the latest award-winning sites that use my open source stuff:

Sites using SuperScrollorama (and also Greensock Animation):

Ikea, Life Of One Kitchen
www.zivotjednekuchyne.cz
Won a Smashing Award and was a CSS Awards Site of the Day.

Alpina, Life is Vertical
lifeisvertical.alpina-since1883.com
Won a CSS Design Award and a Smashing Award. Also a Design Inspiration Selection.

New Škoda Octavia
www.nouvelleoctavia.fr
A CSS Awards Site of the Day and Design Licks Site of the Day
FWA Shortlist

Sites using BigVideo.js:

WhiteShoe by Ferragamo
whiteshoe.ferragamo.com
A Smashing Awards winner.

KOJO
kojo.com.au
Awwwards Honorable Mention and featured on Smashing Awards
FWA Shortlist

herrlich media
www.herrlich-media.de
An Awwwards Site of the Day and CSS Design Award Winner and Make Better Websites selection.

Whiteboard Agency
whiteboard.is
An Awwwards Site of the Day and CSS Awards Site of the Day

rocking.tv
rocking.tv
No awards yet, but I think it is pretty slick!

And thanks also to all the other award winning sites using my open source projects that I’ve mentioned in the past.

Lastly, a special shoutout to Alexander Prinzhorn and his magnificent skrollr project. Many a time I have viewed source of a really cool scrolling website in hopes that it was using my code, only to discover that it is a Skrollr site. At which time, I clench my fist and shout “Skrollr! You got me again, haha.”

Update: Here’s a new BigVideo.js winner
L’Attrape Rêve
lattrapereve.fr
Awwwards Site of the Day

Update: BigVideo.js has just got its first FWA Site of the Day via Nectar Estudio!
Lois Jeans
www.loisjeans.com/aw2013
FWA Site of the Day

The Bootstrap Trap

The promise of Bootstrap is so appealing. A quick, easy way to get a web project up and running that just works, and works well.

It is a promise that comes with a trap.

If you’re trying to use a front end framework to offload all the brain-hurting, detail-obsessing, bug-crushing, browser-wrangling mental gymnastics of modern web development, then you are falling into the trap.

What happens when you build your website in Bootstrap 2.0, then Bootstrap 3.0 comes out? What happens when you have bugs? Is it a Bootstrap bug or your own implementation? Maybe it doesn’t work quite the way you need. How much of Bootstrap is a blackbox to you?

If you know your Front End Framework of Choice like the back of your hand, then maybe you’ve avoided the trap. Even so, I bet there are things about it you’d like to change.

Am I suggesting you or your team should consider rolling your own framework for every project? Yes, I’m absolutely suggesting that.

Maybe this sounds like a lot of work. It should also sound like a lot of fun. Just like there is no crying in baseball, there is no lazy in web development.

If you had to hire a great chef, would you hire one who knows how to heat up food, or one who makes all their dishes from scratch? If you want to prepare the best recipes, you need to understand the ingredients.

Dig in and cook up your own flavor of gourmet web deliciousness, but do so by standing on the shoulders of giants. Projects like Bootstrap and Foundation represent thousands and thousands of dev hours. They offer a free peek inside the minds of the best web teams in the world. The work they are doing is incredibly valuable!

Study frameworks. Pick the source code apart and figure out what you like. When starting a new project, always take a fresh look at what’s new on the front end scene.

Re-use from your past. Borrow from the new. Mix them together. Use tools to be efficient. Invent when inspiration strikes. Construct it all together in a way that makes exact sense for that particular project.

Research. Build. Learn. Repeat.

Techweek Day 2: When Humor is a Serious Business

This was an interview session with Dick Glover, President & CEO, Funny or Die, which launched 6 years ago. User generated content was the future. It started with a question. Why is there not a ‘Hot or Not?’ for YouTube?

Instead of following a Hollywood model, they did a Silicon Valley model. They made Will Farrell and Adam McKay an equity partner in the business. They didn’t get money unless the company was successful.

There was no budget. Go find friends and cameras, and do stuff. This led to the creation of “The Landlord”.

After that viral success, there were other videos, shortly followed by the 2008 market crash and then the emergence of social. They focused on combining A-list talent with social media. Though anyone can contribute videos, most of their popular content features professional talent.

Chicago Tech Events

chicagotechevents-logo-180Throughout the past year, I have been keeping a running list of cool local Chicago tech events and sharing it with friends and the people at work. I’ve decided to make it into a thing and put it out there. So I present to you… chicagotechevents.com

I managed to have some fun building it too. It was my first time using Twig templates, and really liked it. I expect I’m going to be using that on a lot of things in the future. Also, I am trying out a couple AWS products. I built the backend on DynamoDB because why not. I made a simple admin page for adding the events to DynamoDB. Every day, a CRON job runs that goes through the events and makes a JSON file with all the data, and puts it onto S3. When you go to chicagotechevents.com, it is loading that JSON file and smushing it together with a Twig template. Nice and simple.

I am also using Amazon SES for the first time for sending out the email newsletter, because the price was right and integration was easy (so far).

When Inpiration Strikes

The dev ops guy on our team comes up with some creative and hilarious passwords for our web servers. Every time he sends me a new ssh credential, it inevitably cracks me up. He follows the multiword approach with some special characters sprinkled in.

archer-posterSome other guys I work with recently launched a new high-end line of quality room spray. Called Archer Air Superiority it comes in scents like distillery, hunting lodge and european sports car. We have Archer posters all around the office.

When inspiration hits, it is usually when the combination of two or more things collide together in your brain and give birth to something else. This is what happed to me when I was walking around the 18th floor and, boom!, I had an idea. A few days later, I launched Passwords for the Manly Man.

I’ve written before about how I come up with ideas. The key to shipping your ideas out into the world is to strike while the iron is hot. Don’t overthink it, just do it. I’ve posted before about my criteria for the types of ideas I like to pursue. It is good to have that, but ultimately what you want is to come down with a bit of the madness over an idea and just execute.

manlymanpasswords-lg

I enjoy doing funny little bite-sized projects like this. It is great to see people getting some chuckles out of it on Twitter and Reddit.

Awesome Responsive Screenshots

responsive-screenshot

Below is a quick, simple way to generate screenshots of Responsive websites like this (note: link is a big image, so it will take a while to load).

Go to the Responsivator website and enter the url of the website. Check the hide header option. This will generate you a url like this: http://johnpolacek.github.io/Responsivator/?site=gesture.com&header=hide

Go to that url and use the Capture Webpage Screenshot Chrome Extension to take a screenshot. After a few seconds of processing time, you will have yourself a nice looking screen capture of your website across a variety of screen sizes (like this).

If you’d like to customize the screen sizes, click the customize button in Responsivator and adjust the frames settings

Note: This article has been updated since originally published. The tool previously recommended is no longer available (as you may notice in the comments).