smonn_

The token limit depends on the way the training data was encoded, usually its using some sort of sin/cos function which has the problem where longer inputs than it was trained on cause the accuracy to plummet. Its also very likely that it just took the last part of your input as context.

Thanks a lot for the recommendation, it's exactly what I was looking for :)

I haven't read it, thanks a lot! I'm probably going to use it in an essay I'm writing about the topic.

I'm currently working on a variation of a spiking neural network that learns by making and purging connections between neurons, which so far has been pretty interesting, though I am having a hard time getting it to output anything more than just the patterns it recognised. I did play around with adding its outputs to the input list, making it sort of recurrent but its practically impossible to decode anything thats going on inside of the network. Im thinking of tracking the inputs around to see what its doing right now, might be interesting to see it generate some sort of tree-like structure.

I have to agree, when money and interests like with the situation around climate change are involved its hard to do actual science, it just turns into a fucked up manipulation game.

There's plenty of interesting neural network designs out there but they're being overshadowed by transformers due to their recent success. I personally thing that the main reason transformers work so well is because they actually step away from the multi layer perceptron stuff and introduce some structure and in a way sparsity.

I'd recommend watching this video if you'd like to know the more nuanced story behind climate change: https://youtu.be/oqu5DjzOBF8 I am not qualified at all to really comment on it but this did open my eyes to how confusing the entire thing is.