Played around with ChatGPT a bit this weekend, which is currently at the peak of some kind of hype cycle.

My first reaction is that I really don’t like it, though a lot of it is impressive. I’ve been trying to figure out why my feelings are so negative. I haven’t been able to get to the bottom of it, but I thought I’d jot down some notes; maybe this will coalesce in the future, or just serve as an embarassing misjudgment. Time will tell.

  • The interface (a back-and-forth chat style) is impressive, and sets certain expectations. It feels like the bar isn’t too high when you’re casually chatting back and forth.
  • The responses feel mimetic, and lacking in confidence. Slightly unnatural in that sense you get when you’re talking to an eager young person trying to impress you with their knowledge. And it’s not that they’re entirely ignorant, but they’re overdoing the details in a way a true expert wouldn’t. Almost like it’s trying to impress.
  • The code generation is impressive, and I keep seeing people writing about how this is the end of programming, but writing code is a tiny part of what it means to be a programmer. I suppose if your job is to take a very detailed specification and turn it into code, no questions asked, maybe this is closer to your day-to-day work? I haven’t seen it do anything particuarly impressive yet. But, it is really good at generating boilerplate. Which, again, is a miniscule fraction of my work.
  • I keep hearing, “Just wait til we see what this does in 5 years!” I’m not sure why we should have a lot of expectation that it will be much better. These models have mostly improved by adding additional layers and additional source text. At some point you start to run out of useful source text. And at some point more layers aren’t going to be that helpful. Are we remotely close to that? I don’t know.
  • The prose that it writes feels so bland to me. There’s no voice there. It’s almost like if you averaged the voices of all the English writers together.
  • Sometimes it can be fun, no doubt.
  • I don’t like the fact that these models are all trained on public data, then used as for-profit tools. I wonder if copyright law will ever catch up here.
  • Clearly this model has been fed a lot of fan fiction. A lot.
  • Being able to type out queries in plain English is certainly valuable, and would open up this knowledge to many people who might not otherwise have an easy time with a generic search engine (I guess?). But, it’s really too bad that it doesn’t point you to any kind of original source as it gives you answers.
  • How do you keep one of these models up to date with new information?