I largely disagree (even now I think having tried to play the inside game at labs looks pretty good, although I have sometimes disagreed with particular decisions in that direction because of opportunity costs). I'd be happy to debate if you'd find it productive (although I'm not sure whether I'm disagreeable enough to be a good choice).

I think point 2 is plausible but doesn't super support the idea that it would eliminate the biosphere; if it cared a little, it could be fairly cheap to take some actions to preserve at least a version of it (including humans), even if starlifting the sun.

Point 1 is the argument which I most see as supporting the thesis that misaligned AI would eliminate humanity and the biosphere. And then I'm not sure how robust it is (it seems premised partly on translating our evolved intuitions about discount rates over to imagining the scenario from the perspective of the AI system).

Wait, how does the grabby aliens argument support this? I understand that it points to "the universe will be carved up between expansive spacefaring civilizations" (without reference to whether those are biological or not), and also to "the universe will cease to be a place where new biological civilizations can emerge" (without reference to what will happen to existing civilizations). But am I missing an inferential step?

I think that you're right that people's jobs are a significant thing driving the difference here (thanks), but I'd guess that the bigger impact of jobs is via jobs --> culture than via jobs --> individual decisions. This impression is based on a sense of "when visiting Constellation, I feel less pull to engage in the open-ended idea exploration vs at FHI", as well as "at FHI, I think people whose main job was something else would still not-infrequently spend some time engaging with the big open questions of the day".

I might be wrong about that ¯\_(ツ)_/¯

I feel awkward about trying to offer examples because (1) I'm often bad at that when on the spot, and (2) I don't want people to over-index on particular ones I give. I'd be happy to offer thoughts on putative examples, if you wanted (while being clear that the judges will all ultimately assess things as seem best to them). 

Will probably respond to emails on entries (which might be to decline to comment on aspects of it).

I don't really disagree with anything you're saying here, and am left with confusion about what your confusion is about (like it seemed like you were offering it as examples of disagreement?).

(Caveat: it's been a while since I've visited Constellation, so if things have changed recently I may be out of touch.)

I'm not sure that Constellation should be doing anything differently. I think there's a spectrum of how much your culture is like blue-skies thinking vs highly prioritized on the most important things. I think that FHI was more towards the first end of this spectrum, and Constellation is more towards the latter. I think that there are a lot of good things that come with being further in that direction, but I do think it means you're less likely to produce very novel ideas.

To illustrate via caricatures in a made-up example: say someone turned up in one of the offices and said "OK here's a model I've been developing of how aliens might build AGI". I think the vibe in Constellation would trend towards people are interested to chat about it for fifteen minutes at lunch (questions a mix of the treating-it-as-a-game and the pointed but-how-will-this-help-us), and then say they've got work they've got to get back to. I think the vibe in FHI would have trended more towards people treat it as a serious question (assuming there's something interesting to the model), and it generates an impromptu 3-hour conversation at a whiteboard with four people fleshing out details and variations, which ends with someone volunteering to send round a first draft of a paper. I also think Constellation is further in the direction of being bought into some common assumptions than FHI was; e.g. it would seem to me more culturally legit to start a conversation questioning whether AI risk was real at FHI than Constellation.

I kind of think there's something valuable about the Constellation culture on this one, and I don't want to just replace it with the FHI one. But I think there's something important and valuable about the FHI thing which I'd love to see existing in some more places.

(In the process of writing this comment it occurred to me that Constellation could perhaps decide to have some common spaces which try to be more FHI-like, while trying not to change the rest. Honestly I think this is a little hard without giving that subspace a strong distinct identity. It's possible they should do that; my immediate take now that I've thought to pose the question is that I'm confused about it.)

I completely agree that Oliver is a great fit for leading on research infrastructure (and the default thing I was imagining was that he would run the institute; although it's possible it would be even better if he could arrange to be number two with a strong professional lead, giving him more freedom to focus attention on new initiatives within the institute, that isn't where I'd start). But I was specifically talking about the "research lead" role. By default I'd guess people in this role would report to the head of the institute, but also have a lot of intellectual freedom. (It might not even be a formal role; I think sometimes "star researchers" might do a lot of this work without it being formalized, but it still seems super important for someone to be doing.) I don't feel like Oliver's track record blows me away on any of the three subdimensions I named there, and your examples of successes at research infrastructure don't speak to it. This is compatible with him being stronger than I guess, because he hasn't tried in earnest at the things I'm pointing to. (I'm including some adjustment for this, but perhaps I'm undershooting. On the other hand I'd also expect him to level up at it faster if he's working on it in conjunction with people with strong track records.)

I think it's obvious that you want some beacon function (to make it an attractive option for people with strong outside options). That won't be entirely by having excellent people which will mean that internal research conversations are really good, but it seems to me like that was a significant part of what made FHI work (NB this wasn't just Nick, but people like Toby or Anders or Eric); I think it could be make-or-break for any new endeavour in a way that might be somewhat path-dependent in how it turns out; it seems right and proper to give it attention at this stage.

Makes sense! My inference was because the discussion at this stage is a high-level one about ways to set things up, but it does seem good to have space to discuss object-level projects that people might get into.

I agree in the abstract with the idea of looking for niches, and I think that several of these ideas have something to them. Nevertheless when I read the list of suggestions my overall feeling is that it's going in a slightly wrong direction, or missing the point, or something. I thought I'd have a go at articulating why, although I don't think I've got this to the point where I'd firmly stand behind it:

It seems to me like some of the central FHI virtues were:

• Offering a space to top thinkers where the offer was pretty much "please come here and think about things that seem important in a collaborative truth-seeking environment"
  • I think that the freedom of direction, rather than focusing on an agenda or path to impact, was important for:
    • attracting talent
    • finding good underexplored ideas (b/c of course at the start of the thinking people don't know what's important)
  • Caveats:
    • This relies on your researchers having some good taste in what's important (so this needs to be part of what you select people on)
    • FHI also had some success launching research groups where people were hired to more focused things
      • I think this was not the heart of the FHI magic, though, but more like a particular type of entrepreneurship picking up and running with things from the core
• Willingness to hang around at whiteboards for hours talking and exploring things that seemed interesting
  • With an attitude of "OK but can we just model this?" and diving straight into it
    • Someone once described FHI as "professional amateurs", which I think is apt
      • The approach is a bit like the attitude ascribed to physicists in this xkcd, but applied more to problems-that-nobody-has-good-answers-for than things-with-lots-of-existing-study (and with more willingness to dive into understanding existing fields when they're importantly relevant for the problem at hand)
  • Importantly mostly without directly asking "ok but where is this going? what can we do about it?"
    • Prioritization at a local level is somewhat ruthless, but is focused on "how do we better understand important dynamics?" and not "what has external impact in the world?"
• Sometimes orienting to "which of our ideas does the world need to know about? what are the best ways to disseminate these?" and writing about those in high-quality ways
  • I'd draw some contrast with MIRI here, who I think were also good at getting people to think of interesting things, but less good at finding articulations that translated to broadly-accessible ideas

Reading your list, a bunch of it seems to be about decisions about what to work on or what locally to pursue. My feeling is that those are the types of questions which are largely best left open to future researchers to figure out, and that the appropriate focus right now is more like trying to work out how to create the environment which can lead to some of this stuff.

Overall, the take in the previous paragraph is slightly too strong. I think it is in fact good to think through these things to get a feeling for possible future directions. And I also think that some of the good paths towards building a group like this start out by picking a topic or two to convene people on and get them thinking about. But if places want to pick up the torch, I think it's really important to attend to the ways in which it was special that aren't necessarily well-represented in the current x-risk ecosystem.