FOSS4G'13

Excellent posts and I like the idea of anonymous paper selection. Traditionally FOSS4G makes use of community ranking system which I think is a fantastic approach as the event is for the community, I guess we need to consider how the two can be compatible.

Matt.

The AGI model which i think Rollo had proposed is for the first selection to be anonymous and then the fine tuning to be done knowing who the speakers are and taking account of any previous on being snore makingly boring or unashamedly corporate etc.

No reason why the community vote cannot contribute to the first phase and also be anonymous

+1 for Steven's suggestion

Hi all,

I think there are a few things to think about here. I can see pros and cons to the anonymous selection approach. Certainly I can see the merits as outlined in the discussion thread here and the linked articles. But at the same time, there are certain speakers who are definitely draws - people want to see folks like Paul Ramsey, Frank Warmerdam, etc etc, and if we reject people who would attract others to the conference we do risk losing out. Or to put it another way, I think there's a fair counter-argument that someone's background and experience is a relevant aspect to evaluate in addition to just the abstract. I'm on the fence honestly - I understand the arguments both ways but probably lean slightly towards not being anonymous.

Another thing I wanted to raise, which has been mentioned, is the community voting. This has always included people's names in the past. I think that if you are considering changing the selection process (whether by making it anonymous or other means), this is an important thing to run by the community at large, or you risk getting lots of criticism. On almost any topic it's hard (well pretty much impossible!) to get a consensus from the OSGeo community, and on something like this I doubt you will get a clear one.  I imagine some people will support a change to anonymous voting and some will oppose it.

I think it's your decision to make, but I do think it's very important to sound out the community before deciding. If the comments back lean strongly one way or the other (on the anonymous topic in particular, or other changes you propose), then you should probably follow the majority opinion, unless you feel you have a really strong reason for going against it that you're prepared to justify. If opinions are reasonably evenly divided then you can go either way as the committee feels is best. But the main thing is that you are seen to have consulted the community. So I think someone should write up a summary of this discussion, say you're weighing up anonymous voting, and would like to get people's opinions - and post this to the OSGeo discuss and conference lists.

Note that I'm not saying you should do everything by consulting the community, just where you're considering a major change to how things have been done in the past, and I think this falls into that category.

As always, I'm just giving you my opinion, it's your event and you can do what you like :).

Cheers,
    Peter.

Cheers Peter

Sage advice (as always).

I think we saw the anonymous review phase as a first pass for the selection team. If the community give feedback that they have a strong preference for voting with the knowledge of authors we can go with that and still decide whether our own first review is anonymous or not (although I guess it would be harder if all the names were listed next to the abstracts in the community vote).

Key point to consider is that even if we have an anonymous first phase of selection that only shapes the program it does not absolutely determine it (as the community input does btw). If we saw that we had not selected na important draw or big hitter in this phase we would still include them as we refined the program and if a couple of knowledgable people said "oh no you really don't want so and so because ..." then that could over ride the first phase of selection.

I don't have very strong views for or against anonymous selection (for me it will make little difference as I won't know a lot of the authors anyway)

Barry could you post something to the conference and foss4g lists to invite feedback from the community.

A few more things to think about, and possibly include some of these in your "request for input" from the community.

One is whether there should be any limit on number of presentations either from individuals or companies. We didn't decide on this before the fact, and several individuals submitted a lot of abstracts. We had one person with 4 or 5 talks that got accepted via the community voting process (some of which were joint ones with other people). I think we agreed in the end that we would not accept more than 2 presentations from one person, with a couple of exceptions. In the case I mentioned we worked with that person to get other people to present some of the topics. We decided not to have any hard limit on papers from a particular company, but in the review process we looked at this and made a few adjustments to try to keep a reasonable balance. I think it is worth considering having some limit on number of abstracts that an individual can submit - if someone submits 5 then it obviously significantly increases their chance of getting in, I might be inclined to have a limit of 2 or 3 submissions per person (though again a good thing to get feedback on).

We used the community voting process pretty strongly as a guide. Not sure I have exact numbers but probably we took the top 75% of papers that came above the cutoff in the community ranking, and then looked at 25% above and below the cutoff line and moved things around to try to balance the program. In some cases we consolidated papers, or rejected some that largely duplicated others that were more highly ranked.

One thing I did notice is that the community ranking process seemed to favor "product" type presentations versus "case study" type presentations, which I think shows the technical bias of the existing community. After realising this, we had a specific initiative to identify abstracts that we regarded as case studies (most of which didn't make the cut on the community vote) and we put those into a separate track. I think it is worth some effort to encourage more abstracts that are case studies.

I'll post to the mailing lists later.

Personally I'm very much in favour of anonymous selection. The Rock Stars
of OSGeo should get in on the merit of their abstracts, not the accident of
their name. If someone can't write a compelling abstract, can they really
write a good talk? (Although I suppose the converse is true - an exciting
abstract could produce a dull presentation..). But anyway, knowing that you
are starting at the same level as the Franks and Pauls of the OSGeo world
is a great way to encourage participation from those who might think "Oh,
its just the same bunch of guys and girls again - why bother?"

Anyway, an unbiased message will go out shortly...

I've put together a short google form to gather some opinions:

https://docs.google.com/spreadsheet/viewform?formkey=dEwwdXFwR003aWV1N1V3VmVFTXN1cEE6MQ

Does that look okay?

Looks good although response may be predictable. Instead of the 1-10 for community:committee influence perhaps we should have a smaller range of options with descriptions e.g.

• Wholly decided by the community
• Community has stronger influence in selection process
• Balanced between community and committee
• Committee has stronger influence in selection process
• Wholly decided by committee

Barry, all,

Looks very good. Do we want to preface it with a short note about our thinking? I am in favour of anonymity in principle, not least because we are specifically setting out to attract an audience which is new to os gis, albeit mainly as delegates. But in general it seems fairer and less clique y. But there are arguments either way, and a couple of paras up front might help people decide which ones they like best.

Agree that some intro would be a good idea, set out pros and cons. Emphasise that we are sounding opinion from community but try to leave us some room to decide on the process

There should be one para at the start:

"""
We are trying to sound out attitudes to presentation selection for FOSS3G
2013 Nottingham. The final decision on the process will be made by the
committee, but as this is a community conference, we would like to hear
what people think. Shout loud!
"""

but who reads the rubric? :) I'll expand it a bit without trying to promote
my views too much, and tweak the weighting question options.

New version:

"""
We are trying to sound out attitudes to presentation selection for FOSS3G
2013 Nottingham.

Anonymous reviewing encourages participation as it is perceived to reduce
bias and "cronyism". However it takes away the ability of reviewers to see
more than the abstract if they know that a presenter will give a great
talk, and that people will want to come to the talk because of who it is.

The final decision on the process will be made by the committee, but as
this is a community conference, we would like to hear what people think.
Shout loud!
"""

Barry, and all,

I was really thinking more of of just an open ended discussion on the community mailing list similar to what has gone on here. I don't think there's enough of the background discussion in your survey. 

So I would be more inclined to copy and paste your initial message to this group and send it to the relevant email lists, maybe with a few sentences added saying there are a range of views on the committee and we wanted to also get opinions from the community. You should definitely include the links to those other articles that you initially referenced.  

I wouldn't put in the sentence about "the final decision on the process will be made by the committee" personally ... I think that's a given, but the main aim here is to involve the community and make them feel involved. I think you get more useful input on the process and how people feel about it from an open email thread.

I also don't think you should specifically ask how much the community should be involved, I think you're inevitably going to get the answer that they should mainly choose the program. It has always been understood that community vote is an input but that the final decision lies with the committee, though never with any specified "rules". The results of the voting have never been made public in the past, just fyi, so that gives you a good bit of latitude. I wouldn't try to get too specific on that personally.

Cheers,
    Peter.

I also don't think I'd say that not being anonymous encourages "cronyism" - I think that risks coming across as telling them they've been doing it wrong all these years and you know better. Keep in mind that the community voting process is fairly unique to FOSS4G in my experience, and seen by people as a valuable part of the conference I think, so you need to tread a little carefully in proposing changes to it.

I'm not sure an open-ended discussion would take us forward on this - I
think we've seen all the arguments and it would just go into repetition of
them and people expressing which side of the fence they came down on. I was
hoping to get something quantitative (and anonymous!).

My fear with non-anonymous review is that the people with the most twitter
followers will get the most votes. We've all seen "Vote for my paper at
FooCon2012 _here_!" tweets haven't we?

Maybe we could have an 'am i hot or not' approach to community voting. Show
two abstracts at random and ask "Which of these would you rather go to?"

I'm out of FOSS4G time for today, so I'll leave the poll hanging and hold
off on wide debate on the mailing lists for a bit.

But Barry,

I think the thing that you seem to be missing is that there's a well established process for how FOSS4G papers are chosen, which is via non-anonymous voting from the community. This is perceived as having worked well in the past and is popular with the community in general. It's perceived as being quite "democratic" compared to most conference programs chosen just by a small committee. You're not starting from a clean sheet of paper. If you change that without any consultation with the community, trust me, it will go over very badly. The OSGeo community can be quite outspoken, I made a few mis-steps along the way when I was chair and found that out the hard way! You really don't want to alienate the community as you need their support for the conference (it is their conference).

I think your original note had some strong and thought provoking arguments and I do think that's worth sending to the community lists pretty much as is (from you or from Steven). If you get at least a reasonable level of support for that change then you have some good justification for doing it. If the opinions expressed are strongly against it then I'd be inclined not to make the change. You will probably get some good insights into how the selection process has worked in the past and what people think is good and bad about it.

I had concerns about the community voting system when I was chair but for a somewhat different reason, that I felt it was too easy to "game" the system. But after conversation with Paul Ramsey, who has run the system over the years, I felt much more comfortable that he had some good safeguards in against this. I felt the system worked out very well for Denver (with the caveat that it had a bias towards "techie" product presentations which we addressed by having the committee select a number of less techie case studies that didn't make the community vote). We specifically said that people were not allowed to promote voting for their talk via social media and that worked surprisingly well, people really seemed to respect that.

Seems like Barry's post to the mailing list has prompted some responses - so far all in favour of trying anonymous for the community vote

Yes, a good response. Not quite all in favor, but certainly a good majority so far. Including Paul Ramsey who is influential and has always run the community voting system in the past. As I said I'm on the fence personally, but certainly happy going with the anonymous route given the level of support that has been shown so far. And as Steven and a couple of others said, at some point later in the process you (the committee) will need to review with names anyway (at least to check for over-representation of individuals and/or companies), and you can always make a few adjustments then if you feel you need to.

I think in some ways doing it anonymously makes it more important to establish some rules about a maximum number of abstracts that can be submitted by one person (and potentially a maximum number that can be accepted too, which may be different). I think that might be a good topic to bounce off the list also. Maybe a maximum of 3 submitted and 2 accepted? Or 2 and 1? You might count workshops separately (so one person could do a workshop and submit a paper).

In general, while not making every small decision via the community, I found it was best to bounce certain things off the list first before making a decision (without any suggestion that the opinions received back were binding, they are just input to the committee's decision). It keeps people engaged and interested in the conference, and usually provides good input I think.