How to create a new file similar to idea.log to record logs

Please explain the use case. Your plugin can also use idea.log as described here: https://plugins.jetbrains.com/docs/intellij/ide-infrastructure.html#logging

How to separate log records and output them into custom log files

You could use any logging framework and set it up as required from your plugin if you need custom logging.