Hey, again good points.

But I have recognized sparks of true understanding in one-shot AI works.

I absolutely agree here, this is what I was referring to when I wrote...

I think we can appreciate the beauty of connecting with humanity as a whole, knowing that it is the big data of humanity that has informed AI art - I suspect this is what we find so magical about it.

I suspect that AI has an appeal not just because of its fantastic rendering capacity but also the fact that it is synthesising works not just from a prompt but from a vast library of shared human experience.

you're looking at hours of thought and effort

Regarding the arduous* process of iteratively prompting and selecting AI art, I think the analogy with photography works in terms of evoking emotions. Photographers approach their works in a similar way, shooting multiple angles and subjects and selecting those that resonate with them (and presumably others) for exhibition or publication. I think there is something special about connecting with what a human artist recognised in a piece whether it came from a camera or an algorithm. I acknowledge this is a form of connection that is still present in AI art, just as it is in photography.

* I caveat "arduous" because, while it might take hours of wrangling the AI to express something approximating what we intend, the skill that takes artists years to master—that of actually creating the work, is largely performed, in the case of AI art, by the non-sentient algorithm. It is not the hours of work that goes into one painting that impresses the viewer generally, it's the unseen years of toil and graft that allowed the artist to make something magic within those hours. The vast majority of the magic in AI art is provided by the algorithm.

This is why I see it as analogous to photography. Still a valid art form, but not one that need make actual painting obsolete.

Okay, so I think I get you now, in the imbalanced game, if the payoff is 100 or 1 as in "Zero Sum" is a misnomer, a rational player will still make the same decision, regardless of the imbalance with the other player, given the resulting preference ordering.

However while this imbalance makes no difference to the players' decisions, it does make a difference to the total payoff, making it non-zero-sum. I'm having difficulty understanding why values such as happiness or resources cannot be substituted for utility—surely at some point this must happen if game theory is to have any application in the real world. Personally I'm interested in real world applications, but I fully acknowledge my ignorance on this particular aspect.

I find a practical way to look at a zero-sum game is to imagine that each of two players must contribute half of the total payoff in order to play. This takes a game that is constant-sum, and makes it zero-sum, and does so in a way that doesn't break the constant-sumness. In the case of the imbalanced game, because it is not constant-sum it doesn't reduce to a zero-sum game in this way, remaining a non-zero-sum game with terrible odds for one player, meaning that a rational player won't opt in if given the option.

If I'm not mistaken, this is generally what is meant when someone refers to a zero-sum game. In chess for instance you enter a competition with your rating (essentially your bet) and the outcome of the game has either a negative or positive (or no) impact on your rating and an opposite impact on your opponent's rating.(I'm not exactly sure if chess ratings are calculated as exactly zero-sum, but you get the idea). So, the game is zero-sum. Of course there are outside factors that make it beneficial to both players; enjoyment, brain-exercise, socialising etc which may have positive utility on another level, but the game itself and the resulting rating changes are essentially zero-sum.

This is the sense in which I am using the term "zero-sum", in the most basic sense for someone to win (relative to their starting point, bet or rating) another must lose by an equal amount.

There is probably a more mathematically succinct way of expressing this, but I don't have those tools at my disposal at present. Again, thanks for your comments. Please don't feel the need to continue your labours educating me on this topic, I understand that you clearly have a better understanding of game theory than I do, so I appreciate your time. I should probably continue reading further to level up my understanding. Of course if you feel like continuing the floor is yours. 

I’m not sure how the game is the same when you add a constant. The game as proposed in the example is clearly different. I can see that multiplication makes no difference, and as such doesn’t make the sum non-constant. I don’t see how asymmetrically changing the parameters is a “mere change in notation”.

By the way, I’m sure you’re entirely correct about this, I just simply don’t see how there is a problem with using the concept of zero-sum understood as constant-sum.

Hi Vladimir, thanks for your input, it has been fascinating going down the rabbit hole of nuance regarding the term "zero-sum".

I agree that the term is more accurately denoting "constant-sum", I think this is generally implied by most people using it. There was the interesting "zero-sum" example in the linked article that veered away from "constant-sum" with asymmetrical payoffs, 100,0 or 0,1, meaning that depending on the outcome of the game the total sum would be different. This, to me disqualifies it from being called a zero-sum game, given the common understanding that zero-sum denotes constant-sum. The example seemed to solve the problem by conflating zero-sum and constant-sum and then proceeded to stick to a strict definition of zero-sum, which was confusing. But perhaps I just need to sit with it longer.

To your point about Kaldor-Hicks, yes I guess many positive-sum situations could be described in these terms but I'm really referring to something more general—any situation where the total sum payoff increases regardless of Pareto improvements or promised reimbursement by other means to any party left worse off. For instance if a left-wing government were to increase taxes on the wealth, not offering them any reimbursement, but rather doing this based on the mandate that comes with being democratically elected, then this policy might be positive-sum due to the fact that dollar-for-dollar money makes a bigger difference to a poor person than a rich person, due to diminishing returns on happiness.

I really appreciate your comments, and intend to continue exploring the nuances you've raised. I think for a primer on non-zero-sum games, particularly with a site that is focused on practical solutions in the real world rather than pure theory, the more accessible (perhaps less nuanced) definitions I've used are probably appropriate.

Hi Vladimir,

Thanks for your comment, please excuse the delay in getting back, I'm actually busily digesting your response and the various branches of dependencies that comprise it (in terms of links to other concepts). I intend to get back to you with a considered answer, but am enjoying taking my time exploring the ideas you've linked to.

Hi bideup, thanks for your comment. The graph is simplified from one in the Pew Report with the left bar representing the lower quintile and the right representing the upper. I see what you mean, but the intention of pointing to the 20% mark is to show where it should be given 100% social mobility. Perhaps the omission of the central quintiles didn’t help.

There's a tiny possibility he may have influenced my thinking. I did spend 6 months editing him, among others for a documentary.

That looks interesting, will read :) Thanks.

What an insightful post!

I have difficulty judging how likely that is, but the odds will improve if semi-wise humans keep getting input from their increasingly wise AGIs.

I think we're on the same page here, positing that AGI could actually help to improve alignment—if we give it that task. I really like one of your fundamental instructions being to ask about potential issues with alignment.

And on the topic of dishing out tasks, I agree that pushing the industry toward Instruction Following is an ideal path, and I think there will be a great deal of consumer demand for this sort of product. A friend of mine has mentioned this as the no-brainer approach to AI safety and even a reason what AI safety isn't actually that big a deal... I realise you're not making this claim in the same way.

My concern regarding this is that the industry is ultimately going to follow demand and as AI becomes more multi-faceted and capable, the market for digital companions, assistants and creative partners will incentivise the production of more human, more self-motivated agents (sovereign AGI) that generate ideas, art and conversation autonomously, even spontaneously.

Some will want a two-way partnership, rather than master-slave. This market will incentivise more self-training, self-play, even an analogue to dreaming / day-dreaming (all without a HITL). Whatever company enables this process for AI will gain market share in these areas. So, while Instruction Following AI will be safe, it won't necessarily satisfy consumer demand in the way that a more self-motivated and therefore less-corrigible AI would.

But I agree with you that moving forward in a piecemeal fashion with the control of an IF and DWIMAC approach gives us the best opportunity to learn and adapt. The concern about sovereign AGI probably needs to be addressed through governance (enforcing HITL, enforcing a controlled pace of development, and being vigilant about the run-away potential of self-motivated agents) but it does also bring Value Alignment back into the picture. I think you do a great job of outlining how ideal an IF development path is, which should make everyone suspicious if development starts moving in a different direction. 

Do you think it will be possible to create an AGI that is fundamentally Instruction Following that could satisfy the market for the human-like interaction some of the market will demand?

I apologise if you've, in some way I've not recognised, already addressed this question, there were a lot of very interesting links in your post, not all of which I could be entirely sure I grokked adequately.

Thanks for your comments, I look forward to reading more of your work.

Thanks, very astute point.

Yes, the individual and the collected are tightly coupled with short-term and long-term goals, which exist within individuals too. I think it's interesting to think of yourself as a city, where you need to make systemic changes sometimes to enable individual flourishing.

I really think there is something to making alignment the actual goal of AI—but in a way where the paradoxical nature of alignment is acknowledged, so the AI is not looking for a "final solution" but is rather measuring the success of various strategies in lowering society's (to return to the metaphor of the individual) cognitive dissonance.