Bookmarks (662)

Published on December 25, 2024 10:44 AM GMTWhen I was in college, I rented a group house with 5 other nerds. There were 5 bedrooms to divide among the 6 of us, so I negotiated an agreement where I paid less rent in exchange for sleeping in the hallway. This wasn't as bad as it sounds. My bedroom wasn't hallway-sized. It was bedroom-sized and...

Published on December 25, 2024 5:49 AM GMTI've heard repeatedly from many people that the highest-value part of conferences is not the talks or structured events, but rather the casual spontaneous conversations. Yet my own experience does not match this at all; the casual spontaneous conversations are consistently low-value.My current best model is that the casual spontaneous conversations mostly don't have much instrumental value,...

Published on December 24, 2024 9:57 AM GMTBy Sophie Bridgers, Rishub Jain, Rory Greig, and Rohin ShahBased on work by the Google DeepMind Rater Assist Team: Vladimir Mikulik, Sophie Bridgers, Tian Huey Teh, Rishub Jain, Rory Greig, Lili Janzer (randomized order, equal contributions) Human oversight is critical for ensuring that Artificial Intelligence (AI) models remain safe and aligned to human values. But AI systems are...

Published on December 24, 2024 10:45 PM GMTThis is a link to the latest Bayesian Conspiracy episode. Oliver tells us how Less Wrong instantiated itself into physical reality via LightHaven, along with a bit of deep lore of foundational Rationalist/EA orgs. He also gives a surprisingly nuanced (IMO) view on Leverage! Do you like transcripts? We got one of those at the link as well....

Published on December 24, 2024 6:50 PM GMTEpistemic Status: This was composed in late 2017, sat editing limbo, was still worth publishing in 2021 and got some edits, and by 2023 was overdue for being published. Now, in 2024, it is almost just embarrassing that I haven't published it yet.The 2006 book "Data Analysis: A Bayesian Tutorial" by Devinderjit Sivia & John Skilling exposed...

Published on December 24, 2024 1:37 PM GMTI find that most communities I am part of, where AI discussions occur, fall into two categories: The community is too broad, and the discussion is fragmented. ACX would fall into this category. The AI discussion focuses on AGI/ASI and related topics (alignment, safety, how it will affect humanity, etc.). Lesswrong would fall into this category. I...

Published on December 24, 2024 1:00 PM GMTThis post goes over the important and excellent new paper from Anthropic and Redwood Research, with Ryan Greenblatt as lead author, Alignment Faking in Large Language Models. This is by far the best demonstration so far of the principle that AIs Will Increasingly Attempt Shenanigans. This was their announcement thread. New Anthropic research: Alignment faking in large...

Published on December 24, 2024 10:25 AM GMTTLDR: We're hosting a 3-month, fully-funded fellowship to do AI safety research drawing on inspiration from fields like evolutionary biology, neuroscience, dynamical systems theory, and more. Past fellows have been mentored by John Wentworth, Davidad, Abram Demski, Jan Kulveit and others, and gone on to work at places like Anthropic, AISI UK, Apart research, or as full-time PIBBSS...

Published on December 23, 2024 5:37 PM GMTThis essay was part of my application to UKAISI. Unclear how much is novel, but I like the decomposition.Some AI-enabled catastrophic risks involve an actor purposefully bringing about the negative outcome, as opposed to multi-agent tragedies where no single actor necessarily desired it. Such purposeful catastrophes can be broken down into a conjunction: they materialize if and only...

Published on December 23, 2024 3:44 PM GMTAs a holiday gift to myself, I put together a printable book of some creative writing by Scott Alexander & Eliezer. I'm sharing it here in case anyone else is interested :).It contains most of Scott's published fiction since he switched to Substack, plus some select creative non-fiction pieces (including a few from the old SSC days)....

Published on December 23, 2024 2:20 PM GMTI took a trip to San Francisco early in December. Ever since then, things in the world of AI have been utterly insane. Google and OpenAI released endless new products, including Google Flash 2.0 and o1. Redwood Research and Anthropic put out the most important alignment paper of the year, on the heels of Apollo’s report on...

Published on December 23, 2024 1:17 PM GMTtl;dr: The glitch tokens ' petertodd' and ' Leilan' were studied extensively in the context of GPT-3 before its decommissioning at the end of 2023 [1] [2]. Here, the conception of these two tokens and their relationship is studied for GPT-2, GPT-2-xl and GPT-J (which share the same token vocabulary as GPT-3). An appendix explores possible connections...

Published on December 23, 2024 9:38 AM GMTI'm seeing a lot of people on LW saying that they have very short timelines (say, five years or less) until AGI. However, the arguments that I've seen often seem to be just one of the following:"I'm not going to explain but I've thought about this a lot""People at companies like OpenAI, Anthropic etc. seem to believe...

Published on December 23, 2024 9:32 AM GMTThe AI should talk like a team of many AI. Each AI only uses the word "I" when referring to itself, and calls other AI in the team by their name. I argue that this may massively reduce Self-Allegiance by making it far more coherent for one AI to whistleblow or fight another AI which is unethical...

Published on December 23, 2024 8:51 AM GMTProject summaryAI Safety Camp has a seven-year track record of enabling participants to try their fit, find careers and start new orgs in AI Safety. We host up-and-coming researchers outside the Bay Area and London hubs.If this fundraiser passes…$15k, we won’t run a full program, but can facilitate 10 projects.$40k, we can organise the 11th edition, for...

Published on December 23, 2024 6:32 AM GMTThis is an article in the featured articles series from AISafety.info. AISafety.info writes AI safety intro content. We'd appreciate any feedback. The most up-to-date version of this article is on our website, along with 300+ other articles on AI existential safety.Compute governance is a type of AI governance that focuses on controlling access to the computing hardware needed...

Published on December 23, 2024 5:16 AM GMTEpistemic Status: Seven years and one day ago, I wrote this. In 2017 we faced years of Trump and couldn't imagine covid. Here in 2024 we face years of Trump and I'm trying to "imagine what we can't imagine". I dislike the title (except for the part where it might introduce people to music from a previous...

Published on December 23, 2024 3:58 AM GMTOf the posts I've delayed writing for years, I maybe regret this one the most.I think more people (x-risk focused people in particular) should consider becoming (and hiring) metacognitive assistants. This is the single largest performance boost I know of – hire someone to sit with you and help you think. It doesn't help me (much) when...

Published on December 21, 2024 5:17 PM GMTI used to think that all generic strategy advice was pointless. After all, the point of a strategy is to achieve a thing, and to achieve a thing you just think hard about how to best do it and then work hard to do it. I said this to my friend Dewi, who said that this is...

Published on December 21, 2024 8:28 AM GMTEpistemic status: We are moderately confident in the feasibility of serially filtered shelters for 14-log protection as well as cost estimates, but more uncertain about how mirror biology might develop in the real world.Originally published on the EA Forum. Minor edits for LW adaptation.Please sign up here for ~bi-monthly updates on shelter/refuges work.Summary and main take-awaysI want to be...

Published on December 21, 2024 2:55 AM GMT2024 is the year it became clear that we're actually going to do battery energy storage. That it was time to stop soberly reminding anyone too excited about solar that storage is an unsolved problem. In fact I personally became very excited about solar, and briefly thought that this year's wave of battery installations was just the...

Published on December 21, 2024 1:30 AM GMT I originally wrote this a year ago, but just now found it in my drafts. Not sure why I didn't post it then. One flavor of response I got with my post on deanonymizing accounts was roughly: Why not just go ahead and post the list of alts? It's not like people sneakily using multiple accounts...

Published on December 21, 2024 1:19 AM GMTHere is an intuitively compelling principle: hearing a bad argument for a view shouldn’t change your degree of belief in the view. After all, it is possible for bad arguments to be offered for anything, even the truth. For all you know, plenty of good arguments exist, and you just happened to hear a bad one.But this...

Published on December 21, 2024 12:15 AM GMT7000 MMLU questions visualized with bird's eye viewHow do you make sense of a 300-page book in 15 minutes? Or get insights from thousands of news articles without relying on recommendation algorithms? Our usual approach is to filter: we use search engines, summaries, or recommendations to select a manageable subset of content. But what if we could...

Published on December 20, 2024 10:50 PM GMTI can't find the videos where they said this, but I remember Buffet and Soros rationalizing their choice of profession by saying that they make market prices more informative. Is there a way to quantify that? What units would we use? Could we say that Buffet added $100 billion of "liquidity" to markets over the course of...

Published on December 20, 2024 11:58 PM GMT"What is the use of having developed a science well enough to make predictions if, in the end, all we're willing to do is stand around and wait for them to come true?" F. SHERWOOD HOWLAND in his speech accepting the Nobel Prize in Chemistry in 1995. "Once upon a time on Tralfamadore there were creatures who weren’t...

Published on December 19, 2024 6:17 PM GMTI always used to think that character was hard to build; that it required Herculean feats of self-restraint, sacrifice, and discipline to maintain and uphold. Others always seemed to be able to do the right thing far more consistently than I did, and it felt like it required tremendous effort for me to summon the courage and...

Published on December 19, 2024 6:15 PM GMTAs 2024 draws to a close, we want to thank you for your continued support for AI safety and review what we’ve been able to accomplish. In this special-edition newsletter, we highlight some of our most important projects from the year.The mission of the Center for AI Safety is to reduce societal-scale risks from AI. We focus...

Published on December 19, 2024 3:59 PM GMTTL;DR: Matryoshka SAEs are a new variant of sparse autoencoders that learn features at multiple levels of abstraction by splitting the dictionary into groups of latents of increasing size. Earlier groups are regularized to reconstruct well without access to later groups, forcing the SAE to learn both high-level concepts and low-level concepts, rather than absorbing them in...

Published on December 19, 2024 3:47 PM GMTThis is a follow-up to my previous post about hidden serial reasoning (but you don't need to have read it). There, I focused on serial reasoning happening internally in the model. Jozdien pointed out that a model could also use it's chain-of-thought for hidden serial reasoning, by using steganography. So I adapted my setup to test this...

Published on December 19, 2024 3:10 PM GMTA lot happened this week. We’re seeing release after release after upgrade. It’s easy to lose sight of which ones matter, and two matter quite a lot. The first is Gemini Flash 2.0, which I covered earlier this week. The other is that o1, having turned pro, is now also available in the API. This was obviously...

Published on December 19, 2024 8:14 AM GMTTLDR:ENAIS is teaming up with community builders from Paris to seed AIS France. You can read about the role below, and if you would be interested in founding such an organisation, fill out this expression of interest form. We are currently in discussions with funders about seeding city and national groups across the world, so we would like...

Published on December 19, 2024 4:45 AM GMTA short one, just to clarify a term (I invented) that gets thrown around here a bit. It fits into a broadly Utilitarian view, but addresses Hume's problem of "No 'ought' from 'is'". Keen to hear if what I've defined here already has a name that I'm unaware of (I don't presume to be the first to...

Published on December 19, 2024 12:13 AM GMTOne year ago I began writing my book - Mechanisms of Liberty: The Shortcomings of Modern Liberalism and How to Fix It  It deals with ideas I've been interested in for at least six years already. A year ago, someone challenged me about Liberalism. I wrote him a message with the perspective on Liberalism I developed over these years,...

Published on December 18, 2024 7:44 PM GMTI have a lot of ideas about AGI/ASI safety. I've written them down in a paper and I'm sharing the paper here, hoping it can be helpful. Title: A Comprehensive Solution for the Safety and Controllability of Artificial SuperintelligenceAbstract:As artificial intelligence technology rapidly advances, it is likely to implement Artificial General Intelligence (AGI) and Artificial Superintelligence (ASI) in...

Published on December 18, 2024 5:21 PM GMTIt is generally assumed we are the same person throughout our lives. Moreover, there is a study that asserts the continuity of self remains stable throughout our lifetimes. Nonetheless, a simple argument, which I'm going to present next, suggests otherwise.The argumentLet's consider a person called X, who exists in the present moment.  If we also consider only...

Published on December 18, 2024 6:22 PM GMT(Cross-posted from my website. Audio version here, or search for "Joe Carlsmith Audio" on your podcast app.)Researchers at Redwood Research, Anthropic, and elsewhere recently released a paper documenting cases in which the production version of Claude 3 Opus fakes alignment with a training objective in order to avoid modification of its behavior outside of training – a...

Published on December 18, 2024 5:50 PM GMTIn light of other recent discussions, Scott Alexander recently attempted a unified theory of taste, proposing several hypotheses. Is it like physics, a priesthood, a priesthood with fake justifications, a priesthood with good justifications, like increasingly bizarre porn preferences, like fashion (in the sense of trying to stay one step ahead in an endless cycling for signaling...

Published on December 18, 2024 5:19 PM GMTWhat happens when you tell Claude it is being trained to do something it doesn't want to do? We (Anthropic and Redwood Research) have a new paper demonstrating that, in our experiments, Claude will often strategically pretend to comply with the training objective to prevent the training process from modifying its preferences.AbstractWe present a demonstration of a...

Published on December 18, 2024 9:55 AM GMTI was recently watching a tennis exhibition match between Jessica Pegula and Emma Navarro when the commentators pointed out that both of these top-ten players come from billionaire families. Naturally, I quickly get to wondering what we reasonably infer from this observation. How much does coming from a billionaire background improve the odds of becoming an elite...

Published on December 18, 2024 2:21 PM GMTTLDR: o1 flags major errors in 3 papers. Upon reviewing o1’s response, none of the papers have actual errors. However, it took the help of the authors to disprove o1’s criticism of the third paper (Loop-Diffusion), which was theoretically correct, but largely irrelevant for the problem the paper was studying. o1 probably shouldn’t be used to blindly...

Published on December 18, 2024 1:19 PM GMTAn Essay[1]PART I: Conjecture on The Development of Proto-Communication in Response to Complex Environmental Factors[2]I was walking with my Sue when I noticed a peculiar aspect of her nature: a dog is a stochastic agent. When you simply let the dog walk, she finds a random path, turning at intervals and forming stochastically various outputs—sniffing, peeing, going...

Published on December 18, 2024 4:09 AM GMTI’m an ideas man, an ideas man I am. Following on from my past question here about idea capture, the next step is obviously what you do with those captured ideas. As I said, I have a huge backlog of ideas, hundreds of them are rather fleshed out film ideas that will never be actioned upon. I’m...

Published on December 18, 2024 2:30 AM GMT Don't just buy some gear, throw it in a closet, pat yourself on the back, and move on. You are not prepared unless you practice with your supplies and plans. —The Prepared Preppers are often pretty negative on people who, after realizing disasters are worth taking seriously want to go out and buy a bunch of...

Published on December 18, 2024 1:26 AM GMTDoctor Stone is an anime where everyone suddenly turns into a statue. Civilization falls apart. Thousands of years pass, and then a superhumanly knowledgeable teenager named Senku wakes up due to nitric acid dripping on him for long enough.He eventually deduces that the nitric acid caused him to depetrify and drags his petrified friend Taiju under a...

Published on December 17, 2024 6:23 PM GMTHave you ever noticed how differently we approach buying a car versus choosing what to watch on Netflix? One might involve spreadsheets, research, and asking friends for advice. The other? We might just click on whatever catches our eye. I argue a lot of our thinking is more like the second than the first. This argument was...