The Nightmare

This guide is mainly focused on the recently released Nightmare boss. The Phosani, which is a solo only variant, works slightly different and is not yet released (as of writing this guide).
Overall the Phosani will be very similair in mechanics but slightly less forgiving for mistakes, however there will be less rotations and the time to kill it should be significantly faster than the regular Nightmare (at least while soloing, perhaps even in small teams).

Mechanics:

Disturbing the nightmare inside the lobby area will teleport you in to the room, disturbing the nightmare inside the room will start a 30 seconds join timer until the fight starts and players can no longer join
The nightmare in the lobby area will visually update/animate depending on the stage of the fight (asleep if no fight is going on, wake up and float with her head down while the join timer is active, and fully possessed floating (with a symbol underneath her indicating in which phase/how far along the fight currently is) during the fight

There are 3 "rotations"

At the start of the fight she has her blue shield up, which players need to drain in order to progress
After her shield has dropped the totems open up and players can attack these to "charge" them up, once all 4 totems are fully charged they unleash a magical blast that deals 800 damage to the nightmare (players are unable to attack the nightmare directly and can only damage her through this)
After the blast (unless she has died) she will release sleep walkers near the totems (1 for each player in the room, capped at 8) that walk towards her, each sleep walker that reaches her will charge up the damage for her blast attack that damages all the players (min 5, max 70 damage) (if all sleep walkers get through the players are guaranteed to get killed)
This repeats until she has died (total of 2400 HP but she does not heal, meaning 3 rotations in total where the sleep walkers only occur twice)

During each of these rotations she uses 3 standard attacks and several special attacks (she uses 2 unique specials for each phase, and always uses the grasping claws special)
She targets the player that has the highest defence roll but most of her attacks can damage multiple people in the room

Standard attacks

Protection prayers should block 80% of the max damage, but praying incorrectly increases the max hit by 20%

Melee only hits towards the player that is being targeted, however other players that are nearby the target will still get hit by this attack, so stand on opposite sides. This is also the only standard attack that ignores accuracy checks and always hits.
Magic and ranged hits all the players in the room

Special attacks

Grasping claws (all phases) will spawn portals under the players (unless they are immobilized by husks) and more random ones around the room (it is possible for portals to spawn under immobilized players if there are multiple players on that tile of which one is not immobilized) they will deal heavy damage if still on them when the nightmare strikes (or if underneath nightmare, on the portal, itself)
Flower power (phase 1) will spawn nightmare berries and blossoms, splitting the room into 4 sections, players should move to the quadrant that is indicated with the flowers as otherwise they will start taking rapid damage when they bloom and any damage they would deal to the nightmare is now healing her instead (Nightmare will not initiate this special attack while there are still husks alive)
Husks (phase 1) will spawn 2 husks next to a certain amount of players, one dealing magic damage (blue) and one dealing ranged damage, the player will be unable to move until both of his husks are killed (other players are able to help out in killing these husks)
Parasites (phase 2) will impregnate a certain amount of players in the team with a "parasite" that will grow within you for a bit, after it has fully grown it will pop out of your belly while dealing a (in)significant amount of damage to you. Afterwards it will start to either: heal the nightmare (if it is in the shield phase of the fight), discharge the totems (if during the totem phase and there are totems that have been partially charged), attack random players
In order to negate most of the damage dealt by the parasite coming out, players should drink a dose of sanfew serum (or relicyms balm, currently onobtainable) which will weaken the parasite as well (a lower level parasite will now appear when it's born)
Curse (phase 2) special attack will curse each player on the team (the top hp bar overlay will turn pink as an additional indicator) which will shuffle your overhead prayers (always in the same direction, which is 1 to the right (so click the one on the left of the one you want to turn on))
Surge (phase 3) the Nightmare will jump to a random side of the room, afterwards surging across to the other side (massively damaging anyone in her way)
Spores (phase 3) will spawn mushroom like objects around the arena, standing within 1 tile of one will cause it to explode, this will disable the players run and for a short duration makes the player tired which also lowers their attack speed by 1 tick

Rewards

When the fight is over, there will be several calculations to provide the loot pool

In order to be eligible for a unique item a player must have damaged at least the nightmare and the totems (combined) for a 100 damage each, as well as reach a minimum threshold of total damage, if a player is eligible for a unique they will receive a regular bones drop in combination with their other drop
The player that dealt the most overall damage gets big bones dropped instead
There will be 2 seperate rolls for an unique item:
1. The first roll (1 in 120) with a table containing the nightmare staff and inquisitors pieces
2. The second roll (1 in 600) with a table containing the orbs
Both rolls will occur for each kill (so it is possible to get both items), which will then be rewarded to one of the players (of those that are eligible for a unique) randomly picked but weighted by total damage dealt (damage towards nightmare and the totems)
The unique item drop rates are not affected by drop rate boost % as these are rolled per lobby (they are affected in Phosani's instance as this is a solo encounter), however items like the clue scroll, little nightmare and jar are rolled individually for each player that was eligible for a unique and thus affected by your boost
There will also be 3 boss points awarded for all players that were eligible for a drop and 5 boss points for the person who did most damage
You will need to have dealt at least 300 total damage in order to be eligible for a regular drop (and receive a kill count)
Regular drops have a minimum and maximum quantity, of which the quantity is scaled based on your contribution (your damage dealt vs the overall damage dealt, compared to the team size) so if you have dealt (atleast) 25% damage while in a group of 4 you will receive the maximum amount for that item
Drops are placed 1 tile infront of you (between you and the nightmare) instead of 1 big pile under her for all players, however effects like sorceress pet or bonecrusher should still be in affect

Post fight

After the fight she returns to her resting state on the same location that she died on, after a few seconds the "disturb" option becomes available again and players can restart the fight this way.
She will remain on that tile when she spawns, however if all players have left the area she will fully respawn in the center tile again