I did some testing and I was wrong. It's not about how many pages the table/index is covering.
But it seems to be about how many levels deep the index is.
First I created a few tables of different sizes and checked how many pages they were using this query:
SELECT t.NAME AS TableName
,p.rows AS RowCounts
,SUM(a.total_pages) AS TotalPages
,SUM(a.used_pages) AS UsedPages
,(SUM(a.total_pages) - SUM(a.used_pages)) AS UnusedPages
FROM sys.tables t
JOIN sys.indexes i ON t.OBJECT_ID = i.object_id
JOIN sys.partitions p ON i.object_id = p.OBJECT_ID
AND i.index_id = p.index_id
JOIN sys.allocation_units a ON p.partition_id = a.container_id
WHERE t.NAME = 'MyTableName'
GROUP BY t.Name,p.Rows
ORDER BY t.Name
And I had the same behaviour as you, also on tables covering four pages.
So I checked the indexes of the tables using this procedure:
SELECT *
FROM sys.dm_db_index_physical_stats(DB_ID('MyDatabase'),OBJECT_ID('MyTable'),NULL,NULL,'DETAILED')
And noticed that the indexes of the tables exhibiting this behaviour were all one level deep.
This obviously makes sense since a table scan is the same thing as an index seek or index scan on a clustered table with only one level.
And if the table is unclustered it's even faster.