I'd love to see the Wolf on GitHub. It would solve a job I'm about to tackle in an open source project of mine. I want to improve a timing benchmark for an APL interpreter I've written.
I really enjoyed this mainly because I thought at one point you were building “Orac” (the computer from 70s Brit sci-fi Blake's 7) but Auto-Wolf sounds like a bestial version of RoboCop so 👍.
Could you use design of experiments and then monte Carlo methods to focus your reasoning code and reduce runs?
Great discussion. I'd be interested in seeing the code for the autowolf approach. In university I did molecular modeling using super computers & some of the insights garnered by doing a breadth first search of high dimensional space. It's interesting to think about language models doing iterative high dimensional searches of real world problems.
I would also love to see the Wolf on GitHub. I see some clear use cases which have been mentioned here, but would love to experiment with more, as the method is certainly general and powerful. Would also love to check derailment, as many seemingly one-dimensional use cases may have side effects/externalities.
The Karpathy Loop is underrated in how it applies outside of ML. The same move works across domains: don't just measure output, build visibility into exactly where things break.
I've been running an AI agent autonomously for several months. The biggest leverage wasn't adding capabilities. It was building error logging so the agent could flag its own failure patterns. Iteration speed increased, but only because I could see what was actually failing.
The hard part is disciplined observation. Most people add more power before properly understanding what the current system gets wrong.
That’s inspiring Azeem and EV Team. A couple of thoughts:
1) Multiple weights/ Parallel loops: On the problem of reducing a problem to a single metric, I think a system would get a better outcome if there is weighted scoring for the synthetic judges, a few sets of weights are created and parallel experiments are run. Instead of one loop optimising one score, run say three loops optimising for different flavours of "good" and after N iterations, you get three strong candidates that are good in different ways (i.e. a Pareto front).
2) Cumulative Voting and Tournament: But I wonder also if the personas can signal preference intensity. Say you initially create 20 potential solutions and each persona has 100 votes to select their favourites. Then spawn variants of these 5 favourites (and add some random ones) and run the cycle again. Could Cumulative voting (or even Quadrating voting) from the personas on a range of iterations be an alternative to adding randomness?
The (fun) challenge here is creating effective personas and objectives.
I'd love to see the Wolf on GitHub. It would solve a job I'm about to tackle in an open source project of mine. I want to improve a timing benchmark for an APL interpreter I've written.
Another vote for getting the Wolf on github
Would love to access the repo
I really enjoyed this mainly because I thought at one point you were building “Orac” (the computer from 70s Brit sci-fi Blake's 7) but Auto-Wolf sounds like a bestial version of RoboCop so 👍.
Could you use design of experiments and then monte Carlo methods to focus your reasoning code and reduce runs?
I would love to try AutoWolf. Air Wolf was, at one time, my favorite TV show.
Great discussion. I'd be interested in seeing the code for the autowolf approach. In university I did molecular modeling using super computers & some of the insights garnered by doing a breadth first search of high dimensional space. It's interesting to think about language models doing iterative high dimensional searches of real world problems.
Interesting. Would definitely love to have access to the GitHub code
Consider this my vote for the Wolf on GitHub. Great stuff!
I would also love to see the Wolf on GitHub. I see some clear use cases which have been mentioned here, but would love to experiment with more, as the method is certainly general and powerful. Would also love to check derailment, as many seemingly one-dimensional use cases may have side effects/externalities.
The Karpathy Loop is underrated in how it applies outside of ML. The same move works across domains: don't just measure output, build visibility into exactly where things break.
I've been running an AI agent autonomously for several months. The biggest leverage wasn't adding capabilities. It was building error logging so the agent could flag its own failure patterns. Iteration speed increased, but only because I could see what was actually failing.
The hard part is disciplined observation. Most people add more power before properly understanding what the current system gets wrong.
please post wolf on github. probably the most difficult is evaluation to determine best method for prioritizing solutions.
That’s inspiring Azeem and EV Team. A couple of thoughts:
1) Multiple weights/ Parallel loops: On the problem of reducing a problem to a single metric, I think a system would get a better outcome if there is weighted scoring for the synthetic judges, a few sets of weights are created and parallel experiments are run. Instead of one loop optimising one score, run say three loops optimising for different flavours of "good" and after N iterations, you get three strong candidates that are good in different ways (i.e. a Pareto front).
2) Cumulative Voting and Tournament: But I wonder also if the personas can signal preference intensity. Say you initially create 20 potential solutions and each persona has 100 votes to select their favourites. Then spawn variants of these 5 favourites (and add some random ones) and run the cycle again. Could Cumulative voting (or even Quadrating voting) from the personas on a range of iterations be an alternative to adding randomness?
The (fun) challenge here is creating effective personas and objectives.
I'd love to try out the Wolf! Would be excited to play with this for Software Development decisions.