I know this is a somewhat silly observation, but don't you need an OS for computer to be considered "self-hosting"? Granted "OS" is a broad term, but unless you're talking about building a machine like an IMSAI or Altair that includes a switch-driven front panel for manually punching in machine-language programs byte-by-byte you're going to have to have an "OS", even if it's just a machine-language monitor that offers a simple ML editor/debugger and some routines to save programs and data to some sort of storage media, before you can call a machine "self-hosted". Merely getting a video display, keyboard, and storage device wired up the CPU doesn't get you anywhere if you don't have *some sort* of boot ROM/firmware to drive the things at some minimal level when you flip them on.
As noted, the Pi doesn't *have* to run Linux if you think it's too fat. There are several alternative codebases running on it already (like RiscOS), and the foundation has a basic course up talking about how to bootstrap up the Pi from bare metal. (And it sounds like they're genuinely trying to provide as much documentation as they can, but see below.) It would be completely, 100% possible to write your own machine language monitor for it that provides for whatever level of interactivity, built-in languages and driver facilities, and other "hand-holding" you desire. However:
If you're a purist and you really want to own everything that goes on with your computer you have no choice but to stay away from ARM SoCs. Period. Nearly every SoC manufactured* uses a custom DSP to drive the video and sound hardware they package alongside the ARM CPU and *nobody* releases sufficient information to write your own driver for it. You are stuck with having to use a binary firmware blob to drive those features if you intend to use them, and the PandaBoard/BeagleBoard are no different from the Raspberry Pi in that respect. On all of those systems even if you're "bare metal" on the ARM side you're talking through a firmware blob to the SoC. The Raspberry Pi is actually superior to some of the other development boards because the Pi foundation was able to work with the manufacturer to create a "blob" providing an API that can be used by "generic" code running at a bare metal level to provide at least partial functionality; from what I can glean from the Wiki, for instance, the Pandaboard's graphics hardware is basically completely useless unless you're running Linux or Android on it. Beagleboard does have a RISC OS port but also requires a binary blob and it appears that its blobs are also OS specific.
(One "scary" aspect of many SoC's is it's often technically the undocumented DSP that's in control on power-on, with it providing at the lowest level the basic storage device/file system support to grab the necessary blobs and an ARM bootloader off the flash filesystem. So again, another point for staying away from SoC's if dealing with "firmware" isn't something you want to have to do. There's no way to escape it.)
Honestly I think someone would probably be better off trying to build a trivial CPU out of TTL logic as a first project than they would be trying to make from scratch a computer that's not a replica of some preexisting architecture using an FPGA. An FPGA is a gigantic wad of electronic tinkertoys that *can* be turned into a computer but to do it you need to be able to describe one in meticulous detail. If you're fuzzy on what you really want you're going to be spending a lot of time scratching your head and thinking "Huh, okay, I downloaded the code for a soft-68k CPU... now what?".
+1
If you'd like to laugh at it, here's my dream for making a "from scratch" computer. My plan, if I ever have the time to do it, is first to copy this:
9 Chip CP/M Machine
Frankly I don't even like CP/M but this design manages to provide with just 9 chips (and a Compact Flash card) everything you need to make a fully functional and hackable computer with a mass storage device. The only thing it's missing is a built-in I/O console, but once I have my copy working my follow-up plan is to create a console for it using a Propeller MCU; I have some ideas in mind for doing "psuedo-DMA" video using a Prop and this looks just about perfect for trying them out on. (The goal is actually to massage the system from being a CP/M compatible to a TRS-80 Model I/III semi-clone, which will either involve patching the ROM and TRS-DOS to make direct use of the Compact Flash device, or seeing if I can make the Propeller emulate the original disk controller for those machines. Or both. Next step after that is I'm thinking of redoing the whole thing with a 6502 instead and aiming at Commodore PET emulation...) Obviously that's not particularly ambitious but it's something I might have a chance of actually carrying out without having to learn a *whole pile* of accessory skills. (Multi-layer PCB design, surface mount or BGA soldering, etc. And, of course, the amount of firmware code I'm looking at having to write, hack and debug is measured in kilobytes, not megabytes.)
But again, hey, if you're really up for pulling off the creation of a completely modern high-speed 1.8v logic rmachine with your current skillset don't let anyone stop you.
* Note this "binary blob* problem extends far beyond ARM SoCs. Many peripherals of all sorts depend on "blobs" usually embedded into OS specific drivers, and even x86 SoC chipsets like Geode have them lurking in their BIOS code.