NOTE: This is my understanding of programming as a non-modder, so take it with a grain of salt. I'm not familiar with what features/hooks are provided by Isaac's mod API nor what language it uses.

All DPS calculation would take is:
'D' - a variable that stores DPS calculation
'H' - a variable that records the highest recorded value of 'D' and resetting each room
'A' - an array that stores each value of 'H', and resets when going to a new floor.
'C' - a variable that records the number of rooms with enemies cleared (resetting p/ floor).
'AB' - a variable used in the calculation of 'B'.
'B' - a variable that acts as a benchmark for the average of all values in 'A'.

'D' would activate whenever an enemy takes damage, and record the sum of any damage, then after 1s it'd reset to 0 and go inactive.
Of course, it should only be able to activate while it is in an inactive state, otherwise it would break and continually add to the total without stopping.

Every time 'D' is about to be reset, 'H' would first check if the value of 'D' beats whatever value is currently stored in 'H'. If the pre-reset value of 'D' is greater than the value currently stored in 'H', the value replaces the value in 'H'.

Whenever the player clears a room with enemies (specifically clear rather than exit), the value of 'C' gets increase by 1, 'H' gets stored in 'A'['C'], and then 'H' would reset to 0,
Afterwards, an incremental variable loop( i ) would be used to add the value of 'A'[ i ] to 'AB' thereby producing the sum.
This process would repeat until ' i ' becomes equal to 'C'.
When ' i ' becomes equal to 'C', 'AB' is divided by 'C' and the result is stored in 'B'.
Following this, 'AB' and ' i ' get reset, and the loop is broken.

Whenever you leave the floor: 'A' should be completely emptied, and 'C' should be reset.
Whenever you reroll all your items: 'A' should be completely emptied, and 'C' should be reset.

At this point: 'B' is now the DPS average for the floor. In order to prevent run-cheesing scaling breaks, it would not reset at any point.
However, 'B' would still be replaced with 'AB'/'C' whenever the 'A'[ i ] loop occurs.

In odd situations, 'B' not being reset ensures that the scaling doesn't break.
For example: if you're on a floor and don't clear any rooms w/ enemies before reaching the boss (like using an emperor card), since 'B' isn't reset, the scaling would still work.
And 'A' + 'C' resetting on re-roll gives an out to people who re-roll and suddenly have a much worse build. All they'd have to do to get better scaling is clear an uncleared room with enemies.

Re-rolling could still make runs unwinnable with this system, such as if the player re-rolls their items and doesn't clear a room with enemies before reaching the boss.
Theoretically, you could also spoof a lower DPS to drop the average if you have a high tears build with low damage but high DPS and intentionally don't use it to its fullest.
However, I'd imagine that rooms which act as DPS walls and working w/ RNG would make this exploit much less practical, and its usefulness would be limited anyway given that bosses default to a static health on low DPS rather than scaling inversely.

I'd also recommend adjusting the individual equations you use to scale Boss-HP for each boss depending on how long they're invulnerable or how many minions they spawn. Minions have health-bars too, so taking their HP into account is also necessary to balance them.