Royal Bank of Scotland swung to a third-quarter loss on Friday, hit by costs related to past misconduct and restructuring.