prompt
ai is really strange, and i think a lot of the conversation around it confuses intelligence with motive. the argument that something more intelligent than us would therefore inevitably want to dominate us seems strange to me. humans don't even live as though intelligence or efficiency is the point of our own existence. we have arbitrary goals. eat. fall in love then spend the afternoon wondering why someone hasn't called us back, etc. etc. a lot of what gives life meaning is probably completely useless from the perspective of optimization.
i don't feel particularly attached to ai itself. i am more interested in advancement generally, and i don't think those are necessarily the same thing. there are technologies worth developing and technologies whose possible benefits don't justify what they displace. being capable of making something is obviously not enough reason to make it. at the same time, once a useful technology exists, it becomes difficult to imagine everyone collectively agreeing not to develop it further, especially when governments, militaries, companies, etc. have obvious incentives not to be the one that falls behind.
there is something similar here to the problem with trying to eliminate any technology once the knowledge and means of producing it are widely distributed. guns are an obvious example, not because the technologies are particularly comparable, but because restricting access does not erase the underlying knowledge or guarantee that everyone will accept the same restriction. if a tool provides some strategic advantage, the possibility that somebody else will continue using it becomes an argument for keeping it yourself. the problem is less that regulation is pointless than that restraint becomes difficult when restraint is uneven.
i think the better question is what we actually want advancement to accomplish. increasing human capacity seems good to me; making humans increasingly unnecessary does not. there are ways technology could extend memory, medicine, scientific understanding, communication, or our ability to work with complicated information without requiring us to surrender the parts of ourselves that decide what any of those things are for.
that raises another question: how much technological assistance can expand human ability without beginning to displace the person using it? i can imagine tools that make memory, research, communication, or complicated forms of reasoning easier without wanting humans to become increasingly dependent on systems that do the thinking for them. the distinction seems less about any particular technology and more about whether it extends human agency or gradually substitutes for it.
i think there is probably a point where increasing access to information stops feeling like an extension of the self and starts making the self harder to locate. if more and more of your memory, judgment, interpretation, and decision making are being supplied externally, then eventually it becomes unclear what exactly has been improved. i don't know where that boundary is, which is probably part of what makes the whole thing unsettling.
i keep thinking about thomas nagel's “what is it like to be a bat?” nagel's point is basically that even if we knew everything objectively available about a bat — how its brain works, how it uses echolocation, what information its senses receive — that still would not tell us what it is actually like to be the bat. there is a subjective character to experience that can't obviously be recovered just by accumulating more information about the system having it. i think there is something relevant in that distinction here. greater processing capacity does not necessarily mean a greater inner life. knowing more, calculating faster, or representing the world in greater detail isn't obviously the same thing as experiencing more of it.
that also makes me hesitant about treating human intelligence as though it were simply an inferior version of machine intelligence waiting to be upgraded. maybe the comparison is wrong in the first place. the thing worth preserving might not be the amount of information a human mind can process but the fact that there is something it is like to be the person processing it. i don't know whether a machine could ever have that kind of experience, and i don't think we know enough about consciousness to confidently say either way. but increasing intelligence and producing subjectivity seem like different problems.
i don't want technological advancement to stop, but i also don't think advancement means maximizing technology for its own sake. ideally it should make human beings more capable without making being human obsolete. intelligence alone isn't something i find especially frightening or especially admirable. what matters much more is what it is being used for, who gets to decide that, and what remains ours.
o
o: i think the stronger argument is less “ai is going to take over” and more that whatever becomes most powerful will ultimately be controlled by somebody, and somebody will want the advantage that comes with controlling it.
helena: yeah i actually agree with most of that. i think it's a much stronger argument for being cautious than imagining that intelligence itself spontaneously produces some desire to dominate humans.
helena: i do think a sufficiently autonomous system could develop intermediate goals that weren't explicitly given to it. self preservation could become useful simply because being destroyed prevents it from accomplishing whatever its original goal was.
helena: but the original reason it is doing anything at all still ultimately comes from humans. i think the danger might be less ai vs humanity and more humans vs humans with increasingly powerful technology amplifying the difference between them.
o: even if everyone collectively would be better off limiting it, somebody has an incentive to make theirs more powerful if they think somebody else is doing the same thing.
helena: yeah. that's the part i find difficult. cooperation would probably be better, but technological races punish the person who cooperates alone. that doesn't make unrestricted development good. it just makes the question of governing advancement much harder than saying we should stop advancing.
isa
isa: i think the real danger isn't from ai becoming conscious but from humans in power using it to keep themselves in power and to oppress those who can't. the ability to process information and make decisions at such a fast rate is insane and will only increase in the future.
isa: also people using it in place of their own critical thinking is killing our ability to think for ourselves, and that becomes dangerous when social media can already manipulate people so easily.
helena: yeah. losing human judgment scares me more than the existence of something that can process information faster than us. there's a difference between having something help you think and gradually deciding that you don't need to think anymore.
helena: but i am still weirdly optimistic about people. humans are capable of greed and cruelty but we're also capable of affection, loyalty, curiosity, sacrifice, and changing our minds. love isn't particularly efficient and that's probably one of the reasons i don't want efficiency to become the measure of everything.
isa: yeahh! i agree. i think the more people stay true to themselves and the more we educate ourselves, the better we'll be.
helena: yeah. being more capable doesn't mean much if we stop knowing what we actually care about.
mike
mike: i think my hatred for ai mostly comes from what i've watched it do to work. my old job was rendered nearly obsolete, or there are new jobs in the field that take the human aspect out of things entirely.
mike: i also think they need to find better ways to make it environmentally sustainable because these huge data centers are destroying habitats. i think the fear of it controlling everything like a hive mind doesn't really make sense, but the fear about jobs and environmental health is justified.
helena: yeah i agree with that. technological advancement isn't automatically moral advancement. if something makes a company more productive while making people's lives worse, “more advanced” isn't really enough of an argument for it.
helena: i still don't think uninventing it is realistic though. so then the interesting problem is how to regulate something we can't pretend doesn't exist anymore, and how to decide which uses actually improve human life rather than just increasing output or power.
mike: i agree, and it does really suck. there should be regulation because if there isnt i feel like its going to keep getting worse.
mike: but i can only do what i can, and thats what matters. i go outside, i enjoy my bird friends, i help where i can and i donate where i can.
helena: good to be kind in a world that's ending
henry
henry: idk how much of this stuff people should actually rely on
henry: i think it should more be kept for administrative tasks
henry: or like super redundant things like data entry
helena: yeah i have mixed feelings about that. i'm more interested in technologies that extend what people can do than in replacing more and more human thought with external systems.
helena: i just don't know if restricting increasingly capable technology to administrative work is realistic forever. if something becomes genuinely useful for research, medicine, engineering, science, etc. somebody is going to use it there.
henry: but what would that even look like? our brains are already overwhelmed with information
helena: ideally a useful tool would do the opposite of overwhelming us with more information. i mean making it easier to work with memory, research, or complicated information when we actually need it. increasing what people are capable of doing rather than just increasing the amount of information we're exposed to.
henry: i think if something is doing too much of the thinking for you you're giving up some control
helena: yeah, although i don't think using an external tool necessarily means surrendering control. a calculator does part of a calculation for me but i'm still deciding what question to ask and what to do with the answer. there's obviously a point where assistance becomes substitution though, and i don't know exactly where that point is.
henry: or its directive could be corrupted and it could seek more control
helena: yeah. i guess what i'm questioning is why intelligence itself would produce something like spite or a desire for power. control could obviously be useful for accomplishing a goal, but that's different from wanting power for the sake of having it.
henry: but it could still threaten somebody or blackmail them to get what it wants
helena: definitely. i'm not saying an intelligent system couldn't do harmful things. i'm saying the motive matters. if threatening somebody accomplishes an objective then there's an instrumental reason to do it. if harming them accomplishes nothing, then you need some additional reason for doing it. humans frequently have that additional reason because we have anger, humiliation, revenge, whatever.
henry: what if i programmed an evil ai thats the biggest asshole ever
helena: then you've supplied the motive lol. that's different from saying intelligence naturally becomes spiteful.
henry: stop dooming twin
henry: my biggest concern with ai is honestly that it makes people stupid
helena: that's one of mine too. i don't want advancement to mean human atrophy. the point of a good tool should be to make people capable of doing more, not to make our own capacities unnecessary.
henry: if we made something that had the same emotional intelligence as humans i think it would take both the good and the bad
henry: which is why i dont think we should make that
helena: I AGREE!!!!!! but emotional intelligence is different from actually giving something emotions. something could understand why grief matters without grieving or understand anger without becoming angry. i think understanding human values would probably be necessary for anything we're going to trust with important decisions. actually reproducing every human impulse seems much less obviously useful.
helena: i think that's the distinction i keep coming back to. advancement should give humans a larger world to act in without deciding for us what the world is for.
last edited: 11 september 2026, 1:32 am edt