Friday, March 25, 2022

Percom 2022 Perfail Workshop Panel

 


Jon's notes&answers for panel at Percom Perfail workshop on coping with failure...


First, can we talk about negative or inconclusive results more than failure?

1. Eleanor Roosevelt famously said that "Learn from the mistakes of others. You can't live long enough to make them all yourself." – Can you share your research experiences where you faced difficulties and how you overcame them? What are the common mistakes you see researchers make?

- Framing the problem wrong

- Not going far enough back in history of related work (even 10-20 years)

- Choosing right baseline comparisons.

2. Given your many years of experience, what are your suggestions and advice for young researchers on approaching a new research problem/area such that they minimize the risk of failures (in other words, how to publish a PerCom paper every year?) 

Along with avoiding above errors, be prepared to refactor even very late in work.

3. What is your advice for handling failures in long-term research studies where changing core methodology is no longer an option (e.g., in measurements, system design, etc.). Similarly, what is your advice for studies where ethical concerns became apparent at a much later stage?


If you are doing high risk research, use a  registered experiment publication (e.g. RSOS)  which allows for negative results to be published.

https://royalsocietypublishing.org/journal/rsos

If the problem was measurement/design methodology turns out to be wrong, then the fact that it was a large sunk cost must be published to help other people avoid that cost!

If the problem is ethics (e.g. medical treatment turns out to be worse than existing known treatments), stop immediately and still document. (c.f. pharma companies are improving at this).

You actually have an ethical duty to report!

4. For folks with research industry experience, did you find any differences in how failures are handled in industry vs. academia?

Based on 9+ years on advisory board for Microsoft Research: Industry tends to call a halt right away and move to the next problem to tackle.) (also startups)

Most academic research funding agencies still don't recognize the  value of failure, so many EU/UK/US projects limp on, and just report that  "work was done".

We need to retrain the funding agencies to accept that interesting (i.e. risky) research necessarily has more negative outcomes than positive. 

As with papers, a negative outcome (even just "this is not statistically significantly different")  is still a contribution to knowledge and NOT a failure. Methodologies not working is also useful knowledge.


5. How would you advise young researchers on handling unexpected results from a study? In your opinion, can the research be salvaged, or is it better to move on and start a new work?

Unexpected is the best!

6. How has your approach towards handling failures changed as you gained more research experience?

We actually had a Dagstuhl Seminar on DiY networking where we spent two days building "failure machines" - see report:

https://www.dagstuhl.de/en/program/calendar/semhp/?semnr=14042

7. What steps can be taken to encourage the discussions surrounding failures/setbacks/learnings in different stages of research? How can early-stage researchers find safe spaces to discuss failures without the fear of judgments (from the advisor, group members, etc.)?

Find local workshops, PhD fora, and also, and present, often!

Shadow Program Committees - e.g. IMC has a call out right now: https://conferences.sigcomm.org/imc/2022/shadow/ 

which is an excellent place to see papers that get rejected and WHY!



8. Do you have any coping mechanisms/mantras for dealing with rejections/failures (of research papers, grant/tenure/patent applications)?

For both grants and papers:

If you are confident, and there is substantial positive feedback in any reviews, then regroup, refactor, and resubmit.

If the negative feedback really is a showstopper  (e.g. work has been done before, or see above - reframing doesn't work etc) then move on to next thing - 

Linus Pauling, who got two Nobels, said: “The best way to have a good idea is to have lots of ideas.” 

9. What is the one crucial lesson/advice you would like to share with your younger self in the Ph.D. program?

Submit papers/talks/proposals  - getting feedback from outside your bubble is vital. 

You will always encounter some negative feedback,  so the sooner you get used to coping with it, the better.


1 comment:

Unknown said...

Thank you very much Jon for sharing your experience and advice. Failure in research, chiefly in risky and exploratory projects should be considered as a result by funding agencies. In Brazil, they mostly expect "research done" and money spend within the budget. Best regards from Ijuí.

Blog Archive

About Me

My photo
misery me, there is a floccipaucinihilipilification (*) of chronsynclastic infundibuli in these parts and I must therefore refer you to frank zappa instead, and go home