jimB0

The last-hop switches facing the servers are layer 3 in the direction of the aggregation switches and layer 2 in the direction of the servers so ARP is handled normally and DHCP can be handled with standard DHCP-helper mechanisms. This allows the host networking stack to be vanilla (if you want it to be).

Here are some things:

One scale problem with layer 2 networks is you have to go to layer 3 at some point and you have an aggregation node(s) that have to do a considerable amount of ARP request/reply traffic. The request/reply traffic is handled by the CPU of the machine processing. This can run the CPU hotter than you really want and if the aggregation nodes are control plane policing ARP (most sw vendors do), they will intermittently drop arp requests.

Once the request/reply is processed, the node pushes the ARP entry into hardware. ARP hardware tables are limited resources as well. If you exceed them, traffic will either be black-holed or forwarded by the CPU of the node. Both of these are undesirable.

Additionally, it becomes difficult to scale forwarding capacity of layer 2 spanning tree (or loop free) based networks to greater than 2 aggregation nodes since you will either block ports or create loops.

To run BGP, you would not have to run /32s everywhere. You could have multiple tiers of aggregation that keep your routing tables manageable. It would look something like this.

hosts (/32) -> rack-sw -> (/26) -> aggregation-sw -> (/17)

If you need some specifics to be advertised, scale would be dependent on the number of specifics that you desire to be advertised out of rack in the case above, so it would look something like this.

hosts (/32) -> rack-sw -> (/26 + N * /32_external) -> aggregation-sw -> (/17)

On the note about label switching versus ip forwarding, I would not think of it as either/or but rather both could be used since you need to eventually serve pages to users.