Your writing is some of the best I've ever come across and I hope you keep going!
I truly can't find what "To The Stars" refers to here, interested in reading the source material but haven't been able to find an author name or anything else on google - if you could link me it would be much appreciated!
(which might be one of the better places to ask for an invite, aside from the subreddit, which will let you request an invite but is lowkey kinda dead otherwise)
... with all that said, be advised i don't check my Substack messages very often :)
Very thought-provoking! Some points/doubts of interest:
(1) Cultivating Virtue. Not sure this is distinctly Confucian - Tocqueville says something similar, in a way which is more compelling imo. First, he writes in a post-enlightenment, secular context and therefore faces a problem-space where natural law/the heavens is no longer an answer. Second, he is more individualistic: he locates civic virtue in the pride of self-mastery and ensuing moral responsibility of social/individual improvement. In America: "the humblest individual [possesses] a certain degree of self-respect"; a desire to be "better informed and more active" and to improve his or her condition. It creates "an all-pervading and restless activity, a superabundant force and energy" in the whole society. 1:160-1. For two interwar cultural depictions of this secular(?) civic virtue: Mr Smith Goes to Washington, Winsor McCay's political art https://larrysanger.org/2019/11/a-celebration-of-winsor-mccay/ Damn curious if/why you think is Confucianism is better.
(2) Techno-futurism & humanism. There are obvious tensions between these, and To The Stars only gets around them by using obscure techno-babble to code in a hard stop (the 'Volokhov Criterion'). Human ethics and morality is one thing, but coding the human condition itself into AI and tech is... ontologically dubious? Besides, even if it was possible to formalise the human condition we'd likely face the radical destruction of any notion of free will. It would open the black box of human motive/thought processing/experience, the obscurity of which is probably the main fact allowing us to imagine we have agency. Goodbye existing notions of responsibility, blame, self-control etc; hello manipulation at the psychogical level of classic literature (I push this button, and you are bombarded with a cactus here, 2-tone buzz there, spicy pickle tomorrow, and n other sensory exposures, and now you have an inexplicable confidence in the landed elite, flowing partially (69%) from a belief in the good life anchored in memories of playing in forests as a child).
(3) About aristocracy of merit. You're dead right, the problem is stability. Capra films are all about the civic virtue mentioned above (Mr Smith, It's a Wonderful Life/ also 12 Angry Men & To Kill a Mockinbird). The courageous man of virtue stands up for society. They are also cynical and depressing: if those lone actors died, or became ill, or - er - never lived, then we immediately seem to default into shitsville (which, according to Capra, looks remarkably like the average modern downtown). The real trick, it seems, is to somehow culturally engineer a society where there are enough virtuous individuals to prevent this fragility - but one suspects (a) this is very tricky, given 'cultural engineering' is not exactly... engineering; and (b) there are *possibly* deeper structural mechanisms at play (technology?) which will flip even the most immaculately planned culture of virtue, regardless of its path dependency and home advantage, into supermarket-brand moral pluralism.
I don't dismiss alignment on the basis of aesthetics. my view comes down to the fact that I fundamentally don't believe in fast takeoff or god ai. I don't think anything of ~human complexity or greater will be able to effectively self-modify at an accelerating rate because the networks are too dense and nodes are overloaded with too many uses such that meddling with weights online (rather than just training on more input) would be more like lobotomy than brain surgey
I'm moderately skeptical that llms will be capable of agentic action by just looping it back on itself, skeptical of claims that gpt4 has any kind of understanding to operate as anything more than a clockwork automaton, and skeptical that at a certain point of scaling it will be able to anything like novel sci/eng that would increase its capabilities
given my assumptions, the only danger is of the "gpt tell me how to make a bomb" kind and tbh I'm not worried about regulating ai on that basis rn
I think interpretabilty is a very good path for people interested in alignment to pursue, and most of those people are doing capabilities-adjacent research in ai labs rather than putting pause buttons on their twitters. as for alignment of the "design a true sentience that is aligned with human moral intuitions and values" (rather the more tractable "design tools and techniques that help us understand how the entity thinks") I think this is fundamentally impossible because it's identical to what philosophy has been trying to figure out for humans for 2500 years
Your writing is some of the best I've ever come across and I hope you keep going!
I truly can't find what "To The Stars" refers to here, interested in reading the source material but haven't been able to find an author name or anything else on google - if you could link me it would be much appreciated!
Edit*: unless it's this: https://archive.ph/K2pI4, in which case I leave it for future readers
it's this, I didn't think to put a link (though I'll add one now) because I already wrote about it before haha https://archiveofourown.org/works/777002/chapters/1461984
What a fine and tapestried essay. You have made the prospects of ai mediated political futures feel real and yes alluring.
Sounds like you use ChatGPT for an advanced form of rubberducking?
hi, I just wanted to say that I wrote about ... *checks* ...
320,000 words worth of To The Stars fanfic?
yes that's fanfic of the fanfic
and... I thought you should know
that there is a small Community
and if you are looking for such things
and are ready for such things
then the place for the community
is on the Discord server
and I for one would welcome you there :)
... and to a lesser extent, the Sufficient Velocity forums thread:
https://forums.sufficientvelocity.com/threads/to-the-stars-puella-magi-madoka-magica.3927/
(which might be one of the better places to ask for an invite, aside from the subreddit, which will let you request an invite but is lowkey kinda dead otherwise)
... with all that said, be advised i don't check my Substack messages very often :)
P S hieronym thought this post was cool :)
Very thought-provoking! Some points/doubts of interest:
(1) Cultivating Virtue. Not sure this is distinctly Confucian - Tocqueville says something similar, in a way which is more compelling imo. First, he writes in a post-enlightenment, secular context and therefore faces a problem-space where natural law/the heavens is no longer an answer. Second, he is more individualistic: he locates civic virtue in the pride of self-mastery and ensuing moral responsibility of social/individual improvement. In America: "the humblest individual [possesses] a certain degree of self-respect"; a desire to be "better informed and more active" and to improve his or her condition. It creates "an all-pervading and restless activity, a superabundant force and energy" in the whole society. 1:160-1. For two interwar cultural depictions of this secular(?) civic virtue: Mr Smith Goes to Washington, Winsor McCay's political art https://larrysanger.org/2019/11/a-celebration-of-winsor-mccay/ Damn curious if/why you think is Confucianism is better.
(2) Techno-futurism & humanism. There are obvious tensions between these, and To The Stars only gets around them by using obscure techno-babble to code in a hard stop (the 'Volokhov Criterion'). Human ethics and morality is one thing, but coding the human condition itself into AI and tech is... ontologically dubious? Besides, even if it was possible to formalise the human condition we'd likely face the radical destruction of any notion of free will. It would open the black box of human motive/thought processing/experience, the obscurity of which is probably the main fact allowing us to imagine we have agency. Goodbye existing notions of responsibility, blame, self-control etc; hello manipulation at the psychogical level of classic literature (I push this button, and you are bombarded with a cactus here, 2-tone buzz there, spicy pickle tomorrow, and n other sensory exposures, and now you have an inexplicable confidence in the landed elite, flowing partially (69%) from a belief in the good life anchored in memories of playing in forests as a child).
(3) About aristocracy of merit. You're dead right, the problem is stability. Capra films are all about the civic virtue mentioned above (Mr Smith, It's a Wonderful Life/ also 12 Angry Men & To Kill a Mockinbird). The courageous man of virtue stands up for society. They are also cynical and depressing: if those lone actors died, or became ill, or - er - never lived, then we immediately seem to default into shitsville (which, according to Capra, looks remarkably like the average modern downtown). The real trick, it seems, is to somehow culturally engineer a society where there are enough virtuous individuals to prevent this fragility - but one suspects (a) this is very tricky, given 'cultural engineering' is not exactly... engineering; and (b) there are *possibly* deeper structural mechanisms at play (technology?) which will flip even the most immaculately planned culture of virtue, regardless of its path dependency and home advantage, into supermarket-brand moral pluralism.
So hot stuff, any ideas how to fix (2) and (3)?
I don't dismiss alignment on the basis of aesthetics. my view comes down to the fact that I fundamentally don't believe in fast takeoff or god ai. I don't think anything of ~human complexity or greater will be able to effectively self-modify at an accelerating rate because the networks are too dense and nodes are overloaded with too many uses such that meddling with weights online (rather than just training on more input) would be more like lobotomy than brain surgey
I'm moderately skeptical that llms will be capable of agentic action by just looping it back on itself, skeptical of claims that gpt4 has any kind of understanding to operate as anything more than a clockwork automaton, and skeptical that at a certain point of scaling it will be able to anything like novel sci/eng that would increase its capabilities
given my assumptions, the only danger is of the "gpt tell me how to make a bomb" kind and tbh I'm not worried about regulating ai on that basis rn
I think interpretabilty is a very good path for people interested in alignment to pursue, and most of those people are doing capabilities-adjacent research in ai labs rather than putting pause buttons on their twitters. as for alignment of the "design a true sentience that is aligned with human moral intuitions and values" (rather the more tractable "design tools and techniques that help us understand how the entity thinks") I think this is fundamentally impossible because it's identical to what philosophy has been trying to figure out for humans for 2500 years