Observations and Questions Regarding AI assistant usage

Dear JetBrains AI Team,

I recently conducted a comparison of different models (03, Claude 3.7 Sonnet, GPT-4.1, and Gemini 2.5 Pro) to generate a code optimization plan. During this process, I encountered a few unexpected behaviors that I wanted to bring to your attention.

I noticed that Gemini 2.5 Pro consistently stopped generating the full plan without any explicit notification or error message. This occurred across multiple attempts, even in "Ask" mode, and when it did manage to produce a complete plan, the Markdown formatting was often broken.

While investigating this, I discovered that logging is disabled by default. After enabling it, I observed something quite surprising. The logs indicated the use of various models, but Gemini 2.5 Pro was not among them. Instead, it appears that Gemini 2.0 Flash was being utilized.

Upon further examination of the logs, I found the core request sent to the model:

JSON

{
  "prompt": "ij.fast.chat.classify-context-item",
  "profile": "google-chat-gemini-flash-2.0",
  "chat": {
    "messages": [
      {
        "type": "user_message",
        "content": "Determine if the following context is required to solve the task in the user's input in the chat session: \"How would you optimize this resolver? Think deeply, take your time and then create a GEMINI-2.5-PLAN.md in the same dir with your thoughts, ideas and suggestions….."
      }
    ]
  }
}

This suggests that the backend was using Gemini 2.0 Flash despite the UI indicating Gemini 2.5 Pro. Interestingly, after several attempts, the model seemed to switch to Claude and delivered this output:

</details>
<details open>
<summary>Response Body</summary>
```yaml
data: {"type":"Content","event_type":"data","content":""}
data: {"type":"Content","event_type":"data","content":"After"}
data: {"type":"Content","event_type":"data","content":" analyzing the code provide"}
data: {"type":"Content","event_type":"data","content":"d, I can see that the"}
data: {"type":"Content","event_type":"data","content":" resolver for day"}
data: {"type":"Content","event_type":"data","content":"-performance."}
data: {"type":"Content","event_type":"data","content":"js has several"}

These observations raise a couple of questions:

1. Why does the system appear to switch between models without explicit user control, and why was Gemini 2.0 Flash used when Gemini 2.5 Pro was selected?
2. Why does file writing seem to fail when Gemini 2.5 Pro is selected (even if it's not actually being used)?

While I suspect similar model discrepancies might exist for other tests, I wanted to focus on these specific findings for now.

Finally, I'm curious about the overall vision for JetBrains AI and Junie. While Junie performed comparably to a well-configured GitHub Copilot, I'd appreciate more insight into the distinct goals and intended value proposition of the JetBrains AI Assistant.

Thank you for your time and attention to these observations. I hope this feedback is helpful in improving the JetBrains AI experience.

Sincerely,

Andreas Conradty