Lessons Learned from Talks Selection for PyBay
The accepted talks list for PyBay has been posted for a while now but I wanted to talk a little bit about what I learned from the process.
We received over 90 total talk submissions for PyBay and had the task of picking out 30 or so to go along with the keynote speakers we had already lined up. To be honest I initially thought this would be a simple process of ranking the talks in rough order and selecting the top third. While we had a committee of volunteers doubtless we would be in general agreement and only have to really discuss the the submissions right around the cutoff line.
However, as we got closer to the date, our process grew a bit - and it was a good thing it did!
I proposed that we avoid unconscious bias by judging the talks on the first round without any speaker identification. Kavya, who was in charge of the talks selection process, suggested a 3-part score for each submission based on:
An "interesting" score. A solid talk about how to write docstrings in Python is probably not as desirable as a solid talk about a machine learning framework.
A "content" score. A talk that clearly has a solid outline and moves through several ideas with technical depth is better than no outline.
An "objectives" score. What's the point? Will attendees learn something? Does the talk consider the audience?
We ended up having a meeting with half a dozen volunteers to score the talks. We did the first round on content only and used our scores to rate talks as green (slam dunk - we definitely want this talk), yellow (a talk we could feel good about offering) and red (probably not a talk we want).
At the end of this round we had about 20 green talks, and most of the rest were rated yellow. We only had a few talks that were rated red and most of these were submissions that simply didn't have sufficient information or had a topic that didn't feel interesting enough to our target audience.
This is when the level of work became apparent. We'd spent hours individually rating talks and then reviewing the scores as a group but still didn't have a final list. In order to fill out the remaining slots we had to look at several additional criteria:
Topic diversity. For example, we had four excellent TensorFlow-related submissions and decided right away that despite the uniformly high quality of the talks we couldn't take all four.
Filling the tracks. Before PyBay we polled our audience to find out what kind of talks they were most interested in. Our two largest audiences seem to be data-science areas and web-related talks so we tried to weight appropriately. We also had decided we would try to make sure there was at least one beginner-friendly talk in each slot on the schedule - SF Python Meetup has made a consistent effort to welcome beginners to the community for a couple of years now and we want PyBay to reflect that too.
Company affiliation. We wanted to avoid having too many talks from the same company.
We examined speaker diversity looking to be sure we included speakers from a diverse range of backgrounds.
We did a scan through the speaker bios giving a bump to speakers that we know have significant status within the community, a history of good talks, or have been supporters of the Python community in the Bay Area. We raised the scores of a few talks where the speaker's personally identifying information gave more context & credibility to the talk proposal.
We ended up with half a dozen different spreadsheets analyzing the talks and an email thread that quickly approached 100 replies with analysis and proposed packages of talks addressing each of our various concerns. We finally ended up with a slate of talks of which I am tremendously proud! PyCon 2016 had 95 talks. We picked more than 30 and (including our invited Keynote speakers) have a great lineup. (If you haven't bought a ticket yet please click here!)
What did I learn from this process?
First I suppose I should confess - I've sometimes been irked in the past when I've submitted talks to conferences and not been accepted. I've sometimes attended those conferences and sat in talks thinking - mine was better, mine would have been funnier, I could have given that talk!
Being on the other side of the process has given me some empathy for the people doing the picking. It's also given me a sense of relief as its not obvious to me that not having your talk selected doesn't mean your talk was bad!
Don't Take Rejection Seriously!
It's entirely possible to submit a great, very-clearly-excellent talk proposal on an exciting subject that you have technical mastery of and simply run into several other people who have done the same thing. More generally - maybe you fall victim to a numbers game in that several of your colleagues also submitted, or your general area has too many good talks...
We did not accept many talks that I have full confidence would be outstanding for a variety of reasons. In fact there are probably 30 talks that would have been fine for PyBay that didn't make the cut simply due to the limited number of slots. The lesson learned, for me at least, is not to be discouraged by a rejection. It is not necessarily a reflection on the quality of your submission.
Do Take the Process Seriously!
The other thing that that stood out to me was how few talk submissions were immediately rejected. We rejected a few proposals that were obviously not particularly Pythonic. A Dev Evangelist may have a talk entitled How to use the $FOO API - sample code in Ruby, Javascript and Python. and submit this talk to a variety of conferences hoping to get exposure for their product.
We tended to just pass on these talks without much discussion. We also got some submissions that simply had inadequate detail. "Lessons Iearned while doing X" may in fact end up as a great talk. (And if you're Guido you might get away with a description/abstract as short as this.) But in general it made our job of assessment impossible, especially on the first content-only round. What lessons? Are they applicable to others? Will people find them interesting? With out more details it is impossible to say.
Basically - if a talk proposal contained Python-specific content and had some details it likely got serious consideration. The few talks that were eliminated tended not to have Python specific content or lacked enough detail to know what the talk would really be about.
I have to admit I've made a few talk submissions like this in the past. I may have the best of intentions - I'm a good speaker and I'd definitely flesh the description out more if I'm accepted - but I'll try to avoid this in the future.
If I really want to speak at a conference I should be willing to spend 15 or 30 minutes writing a compelling description and brief outline of a talk!
