Picture of my computer with the HTML code for my distill.pub publication. Background is a window showing Tokyo, Japan.
November 2018. Finishing my first publication, “Visualizing memorization in RNNs,” published in distill.pub. This was my first publication. I thought that and a Master's degree in Machine Learning would be enough to get a Research Software Engineer position; I was very wrong! — Tokyo, Japan

Becoming an Independent Researcher and getting published in ICLR with spotlight

Why I became an Independent Researcher, why you should avoid it, and advice for those who make the difficult choice anyway.

In April 2019, I decided to become an independent researcher, to work on getting published at a major conference. Then in December 2019, after 7–8 months of work without funding, I did just that. I got published in ICLR; even better, I got a spotlight!

Tweeting my ICLR publication. After that, I got almost 100 messages from Independent Researchers asking for advice.

Why I became an Independent Researcher

In March 2018, I got published in distill.pub with “Visualizing memorization in RNNs,” as a sole-author, demonstrating how an interactive saliency visualization for NLP reveals that two models, with nearly the same accuracy, can behave very differently.

Reality check, “1–2 publications in top ML venues”

However, I did get this from a professor:

To start a PhD, without insider referral, you need to do work equivariant to half of a PhD.

Getting started, how I funded myself

How did I get my research idea? How did I fund myself? Those have by far been the most asked questions. However, while they do deserve to be addressed, I do think they are not that relevant. There are so many ways to get an idea, and if you don’t have any commitments to others, there are many ways to cut down your expenses. Feel free to skip to “A lonely life, not losing my sanity or hope,” you won’t hurt my feelings.

To say I lucked out regarding funding, would be an understatement. But Denmark is also an incredibly expensive country. Funding myself in another country would probably have been easier.

In 2019, they also asked me to develop the TensorFlow part of the IoT smartwatch/badge given out at NodeConf EU 2019. While not enough to cover my all expenses for this year, it definitely helped.

Getting started, where do ideas come from

As I said, there are so many ways to get an idea, so don’t take my words too seriously, apply your own creativity.

Almost all publications exaggerate how well they perform. Just improving others work is a viable research strategy.

So, there you have it. It is not a super inspiring strategy for research ideas, it does also come with some significant challenges (just read further), but it is a viable research strategy.

A lonely life, not losing my sanity or hope

As an Independent Researcher, you can’t really expect encouragement from anybody. I know, I know, a lot of Ph.D. supervisors don’t encourage their Ph.D. students either, but hopefully, they can get it from their Ph.D. peers who will have similar struggles. That is pretty much impossible as an Independent Researcher and is the number one reason I would recommend against becoming an Independent Researcher.

Not having a support network that are experiencing the same struggles of writing a paper, as first-author, is the number one reason I would recomend against becoming an Indepedent Researcher.

Everybody needs at least some encouragement, don’t think you can go on for 7 months straight without any encouragement. I constantly worried about: Not finding a solution, getting unjustified peer-reviews, not getting useful results, discovering a significant flaw, even if I get published, it might have no effect because it’s a niche subject.

Picture of my hand on a huge bamboo, not being able to cover (read, handle) the bamboo at all.
Picture of my hand on a huge bamboo, not being able to cover (read, handle) the bamboo at all.
Publishing as an Independent Researcher was probably more than I could handle (literary pun intended). I expected failure, but tried anyway and gave it my best. — Somewhere, Japan
  1. I did side-projects. Allocating all your time to one piece of work is too high a risk. Spend some time doing other smaller projects that you think are useful. Write an open-source tool, implement a known paper, etc.. It is helpful to take a break from research, and if your research project fails, you have at least accomplished something. — In my case, having some of these side-projects be recognized by known researchers, was also a great source of encouragement.
Example of a 3-day side-project. One person reached out and was thankful; just that is a valuable source of encouragement.

Writing a great paper

With only about 20% of submissions getting accepted and your peer-reviewers looking for any excuse to reject your submission, “good enough” is not enough; you need to do “great”! However, you don’t have any supervisor to help you, and you have never submitted before, so how do you actually accomplish “greatness.”

There will be 1–2 messages in a paper, that if misunderstood, will completely confuse the reader and be the first cause of getting rejected. Do not be afraid of repeating a message to prevent that.

In my Distill publication, that message was: “Visualization can give critical insight into models, that an accuracy measure can not. However, you also need a problem that anyone can have an intuition about, Chinese Poetry Generation is probably not such as problem”.

Rejected from NeurIPS 2019

Well, you read the subtitle. We submitted our paper to NeurIPS 2019 and got rejected. Wow, I felt very ill after that. All that time spent, and nothing! My dreams shattered, there was almost no chance for me to pursue my dream — to become involved in ML research.

Having anonymous peer-reviewers hold your life in their hands is so strange.

“Red ink” by Jorge Cham www.phdcomics.com
“Red ink” by Jorge Cham www.phdcomics.com
Me reading reviewer #1 (yeah, #1 and #2 swapped places) comments from NeurIPS 2019.
“Red Ink” by Jorge Cham — www.phdcomics.com
  • Some reviewers required our proposal to do everything that the NALU paper claims to do, even though we provided clear evidence that the NALU model didn’t do those things to any satisfying degree. — If you read the papers, it is division and gating between addition/multiplication that we did not solve. However, we did improve everything else.

Reviewers are likely to side with already published results. They will only think critical about your submission not previous publications, especially if they come from DeepMind.

I want to clarify that the results from the DeepMind paper NALU are not fake. They are entirely reproducible. However, the results are not framed in the most sensible way for extrapolation tasks, which was the main objective, making the model look better than it is at first and second read-through (you have to read the results really carefully). Also, the NALU paper only shows results from a single seed, while our paper show results for 100 seeds. — We have a workshop paper on just these issues.

Submitting to ICLR, what was different

We had already made several improvements to the paper, up to the NeurIPS 2019 rebuttal. And for ICLR, we added even more evidence and experiences to support our claims.

Worth the effort?

In the end, we got published. I wish I could say for sure that it is going to help me get a position where I’m involved in research, but honestly, I don’t know yet. I just saw an email saying you need “2 top-venue publications, preferably with famous co-authors, to get into a top Ph.D. program”, which is something I will never achieve as an Independent Researcher. Let’s hope it is just that university.

Long hanging bridge, where the end can not be seen.
Long hanging bridge, where the end can not be seen.
The path to Machine Learning Research is long for all. However, when you can’t even get off the ground, it can seem impossible. If you really want to, spend the effort and look for alternative ways. — Vancouver, Canada

Q & A

The above didn’t answer all the questions I received, as I don’t think that would have been very engaging. So here are answers to the questions that didn’t make it.

  • Q: Your work is useless, and you are a joke. A: Thanks. Have you considered becoming a reviewer?
  • Q: How much time did you spend. A: I spent about 48h/week just on this publication. Some weeks close to 100h/week, others much less. Remember, I also did side-projects and freelancing.
  • Q: Where did you get the computing resources from. A: Alexander was able to provide us with that because he was a Research Assistant.
  • Q: I was offered a Ph.D. by my supervisor, should I take it? A: If you really want to research, yes, probably. The competition is crazy now, I think you should take what you can get.
  • Q: I’m doing an internship, but I’m not getting anything from it because my supervisor is absent. A: Take charge of your own faith. Don’t expect your supervisor to come to you. Be happy you got an internship, I couldn’t. Start to arrange meetings, and remember there are more people you can ask than just your supervisor.
  • Q: I’m doing my masters, how can I prepare myself? A: If you can find a professor who is open to it, try to publish. Also, try to get internships while you study. Most internships are only available to those who study, I was offered internships I couldn’t get because I had graduated.
  • Q: How do I become a better programmer? A: Writing open-source for many years, this allowed me to be mentored by some of the most exceptional programmers I know of.
  • Q: What extra does it take to get a spotlight? A: To be honest, I think it mostly just luck. But it sounds good, so I put it in the title.
  • Q: I have seen others become a researcher at Google, with just a Master’s degree. How did they do it? A: Yeah, the golden years were 2013–2015. If you finished your Master’s there and got lucky, you could get really far.

Summarizing

I want to finish by just repeating my main advice for other Independent Researchers.

  1. Expect failure, but try anyway. Only about 20% of publications get accepted. As an independent researcher, your chances are likely less than average. Don’t be discouraged just because you get rejected once. And do side-projects, such that if you do fail, it doesn’t feel like a complete waste of time.
  2. Avoid it. It is hard to think of a reason to become an Independent Researcher if you have an alternative. Maybe your MSc supervisor was often absent, but at least there will be other Ph.D. students to talk to. As an Independent Researcher, you are missing out on a critical support network.
Working as an Independent Researcher is incredibly hard and quite lonely. However, with luck, there might be light around the corner. — Vancouver, Canada

For the researchers, who may also be reading along

Finally, I did go to NeurIPS 2019, because we got a workshop paper accepted. There I had the chance to talk to several recruiters, professors, and researchers. I was shocked! There is a gigantic gap between what recruiters require, researchers want, and professors provide. — I want researchers to know that over the last 2 years, the landscape has completely changed. Just getting into a Ph.D. program, is now likely harder than completing a Ph.D.

If you finished your Master’s in 2017 or earlier, getting into a good PhD program was achievable. Now it requires 1–2 publications in top venues (NeurIPS/ICLR/ICML), preferably with famous co-authors. I hope researchers, professors, committee members, and conference organizers, will help to stop this new elitism that is rapidly developing! It only amplify the existing biases that are already challenging the industry.

9 months after the publication, what happened

This is future Andreas editing this blogpost. Many of you have asked what happened after my publication. I’ve now written a new blogpost to tell that story.

Ph.D. Student at Mila, researching interpretability for Machine Learning because society needs it. Published in Distill, SEDL|NeurIPS, and ICLR (spotlight).