Comments on: Credit Assignment in Deep Learning

By: Tim Dettmers

Tim Dettmers — Mon, 15 Jan 2018 20:49:12 +0000

In reply to Murray Frank. There are many interesting examples indeed! Do you think this relates how past researchers communicated their work, or how "mature" their work is in general (master thesis vs full researchers).

By: Murray Frank

Murray Frank — Wed, 03 Jan 2018 00:48:53 +0000

Giving credit is a long debated problem. Frequently someone comes up with an idea that has a huge influence. Then other people say that in reality someone else had really thought of the idea earlier. Often such claims are true. In other cases you can see the essence of the idea but not the whole thing in the earlier work. In some cases we retroactively give credit. In other cases it does not happen. For example, Kuhn and Tucker came up with a standard theorem in optimization in 1951. Eventually people realized that it was also in Karush’s 1939 master’s thesis. To this day you will see the theorem called the Kuhn-Tucker theorem, and you will also see it listed as the Karush-Kuhn-Tucker theorem. There are many such examples.

By: Tim Dettmers

Tim Dettmers — Thu, 16 Nov 2017 20:49:05 +0000

In reply to AdamP. I am talking about the "Critique of Pure Reason" here. Kant published it, and it was poorly received because people could not understand it. He rewrote it 6 years later, and suddenly people could actually understand his points, which in turn could help other people understand. Through this Kant became the most talked-about philosopher during that time.

By: AdamP

AdamP — Wed, 15 Nov 2017 06:01:37 +0000

Hi, just found this blog, great stuff!
Just a minor point – “Communication can be important even after publication. Just look at Immanuel Kant’s work, which is probably the most important philosophical work, yet it was not read for some time because nobody understood his ideas.”
I find that very strange, not a good example at all. “Probably the most important philosophical work” – I don’t know what that’s based on. “Arguably”, arguably, but ‘probably’?! I’ve never heard anyone claim that.
It’s news to me that Kant wasn’t read for some time. Whatever “some time” means. But I don’t think that’s right at all.
And “nobody understood his ideas” is even more murky. (There’s not even a single thing you could point to and call “an understanding of his ideas”, i.e. there are a wide range of interpretations, even to this day. What one person calls an understanding, to another is gross misunderstanding, etc.)
His Critiques have a repellent, almost impenetrable style, granted, maybe that’s what you meant.
p.s. Gauss invented the FFT, apparently, though it seems he never told anyone, not sure how much credit he deserves. I kept expecting to see his name on these pages in that connection. 🙂

By: Tim Dettmers

Tim Dettmers — Sun, 01 Oct 2017 13:25:31 +0000

In reply to Karthikeyan Chittayil. Thank you, that means a lot of me!

By: Karthikeyan Chittayil

Karthikeyan Chittayil — Sat, 30 Sep 2017 05:53:18 +0000

Tim, I think you have a nice way of putting complex concepts in simple words, and elementary maths. Please keep it up. As you have brought it out, communication indeed is very important. Keep blogging !

By: Tim Dettmers

Tim Dettmers — Fri, 29 Sep 2017 12:52:22 +0000

In reply to Yun Teng. Indeed, I think this is a good way to think about Person 2 and Person 3.

By: Yun Teng

Yun Teng — Thu, 28 Sep 2017 01:16:01 +0000

Enlightening as always!
The saying “Those who can, do, those who can’t, teach” has always bothered me.
Because of that, I really liked your “Timing and Relational Effects” example with the PageRank, which showed that Person 2 was the most important, and even Person 3 had 0.2305 contribution.
To me, Person 2 is like a mentor/advisor and Person 3 is an instructor with many students, both roles having significant impact in the real world.

By: Tim Dettmers

Tim Dettmers — Mon, 25 Sep 2017 13:12:45 +0000

In reply to Alison B Lowndes. Thanks for your comment, Alison. I really appreciate it! Indeed it can be messy with the wrong supervisor, but I must also say that it was a good experience for me since I learned a lot from that experience. With that, I will be able to make a better choice for my PhD advisor. So in the end, it was not so bad after all!

By: Jeff

Jeff — Mon, 18 Sep 2017 12:22:09 +0000

In reply to Tim Dettmers.

Ah sorry, I think I misunderstood your blog post originally, thought you were dismissing original credit. Impact isn’t something I have thought about seriously, and I think the topic is something that could easily be brushed aside for the status quo with lazy statements like “impact isn’t something I have thought about seriously” or with hostility to change. So with that said I think it’s good you’re questioning credit assignment, even if you are met with a lot of hostility. So thank you.

I agree communication is important. I am very new to deep learning, and I find the initiatives within the field for improving communication to be extremely inspiring and helpful to me. Including your own work, especially your last blog post about research direction and computational efficiency. So thank you and I hope you continue to write.