file cache very dangerous across multiple machines

I work on multiple machines.  I use dropbox to make the identical files available across these machines.

The pycharm local file cache is extremely dangerous.  With so-called autosave, where are the files in a project saved?  In the cache or in the file system?

Is "save all" the only way to make the local history most recent version save to the file system?

What does the synchronize command do?  (There is NO result when searching help for "synchronize".)

It is VERY dangerous to disintermediate the file system this way.  While it is superficially clever to allow automatic versioning to the file cache | local history, it is VERY UNSAFE. In your help (very out of date, by the way–and generally very inadequate), you even describe pycharm as providing a private local version control system.  The local history is helpful (and the diff UI is very nice).  

But, it is unclear when the project files are committed (if I may use that word) to the file system.  This MUST be made MUCH MORE EXPLICIT than an asterisk on a tab.  Pycharm is not a world unto itself. It is a toola good toolbut only one of several tools any developer will use.  The file system, not some hidden cache in Pycharm, is the means by which tools share information.

I had the shock of discovering this when I made changes on one machine, ran code, closed Pycharm, reopened several times.  No asterisks appeared on any of the open file tabs.  I went to a second machine and the changes were not loaded.  Is this because machine 1 had not committed to the file system?  Or is it because machine 2 was loading from cache instead of from file system.  You can see the confusion.  And before flushing the cache on machine 2, we MUST know that the cache from machine 1 HAS REALLY BEEN COMMITTED TO THE FILE SYSTEM.  Otherwise, when we reload in machine 2, we will get out of date files.

THIS IS A DEEP AND FUNDAMENTAL PROBLEM.  Pycharm is a really nice tool.  But, when something this basicpreserving our code assets in the file systemis fundamentally unreliable, then we cannot trust it.

This may only entail UI improvements.  I am surprised to find no search results on dropbox and local cache.  Hard to believe lots of your users don't use dropbox.

Sorry for the tone.  I like pycharm a lot.  But, when I saw this I was floored.
6 comments
Comment actions Permalink
The pycharm local file cache is extremely dangerous. With so-called autosave, where are the files in a project saved?  In the cache or in the file system?


All JetBrains IDEs make sure that an external application sees the same state as you do in the editor. As a consequence, all files are saved to disk when you switch to another application or execute an external tool. Also, they make sure to reload all changed files from disk when you switch back to IntelliJ. At first, you might feel uncomfortable with this but you'll get used to it quite fast. If you want to go back to older versions, there's always Local History (VCS > Local History > Show History) or Source Control.

Not sure what you're referring to with the local file cache. Do you mean local history?

Is "save all" the only way to make the local history most recent version save to the file system?

Save all saves all files to disk. It is the same as switching applications. I haven't used Save all in the last 10 years...

What does the synchronize command do?  (There is NO result when searching help for "synchronize".)


Synchronize verifies that the state you're seeing in the IDE is the same as the state that is on disk. I suppose it is meant for the very very rare occasions when those two differ. Please file an issue in YouTrack if you feel that the documentation of the Synchronize command should be clarified.

On a general note: Dropbox is a fine tool to synchronize files like PDF, Images, Word documents and so on. It is generally not used to version source code. The reason for this is that Dropbox doesn't scale/doesn't have the functionality for dozens of developers to work together on the same source code project. Common tools for this purpose are git, mercurial or svn. I'd recommend to use one of those if you're planning to work on a project from many machines. It is not a good idea to put your development projects on Dropbox.
0
Comment actions Permalink
Thanks for your reply.

I don't version source with dropbox.  I use git.  Dropbox simply makes the files available on multiple machines, as stated.

Synchronize doesn't come up in searching pycharm help;  however, reading through it more, "synchronization" does.   However, one of the vcs's uses this term (Perforce?) and most of the doc refers to this.

Local history is stored in the cache along with lots of other things–especially old versions of source files.  This is why pycharm recommends occasionally clearing the cache.  It is as though pycharm has its own copy of git running under the hood.  That's fine and provides the useful features of examining history and reverting to old versions without any prior setup or work on the user's part.  But, loading project files from the cache rather than from disk might (**might**  – not completely sure here) cause problems.

I have had several cases of inconsistency where pycharm loads and finds that the file was "changed externally" and needs to reload it.  In my case the file was not edited by another application:  rather, the file was edited by a copy (licensed and paid for–I bought after 4 days of trial) of pycharm on another machine.  This behavior seems to confirm that pycharm loads project files from the cache rather from disk.  I don't think it should do this.  It should always load from disk.

pycharm finds this inconsistency and pops up a warning with a link to reload.  So, that is good.  

I have had one case where the inconsistency was NOT detected–I detected by inspecting the source file.  This is what freaked me out and resulted in the post.  It is possible that dropbox was slow to upload the files on one machine before it was loaded on my laptop.  But, that is sort of unlikely.  source files are small and dropbox is pretty fast.  It could have been that pycharm took a while to close and that delayed the transfer.

Keep in mind that dropbox should be invisible to pycharm.  All files are strictly local.  In background, dropbox monitors "its" folders and uploads/downloads changes.  Some latency for sure.

I think the issue here is that pycharm got out of sync and didn't flash its normal warning.  I think this would be impossible if pycharm loaded project files from disk rather than from cache, but really that is for Jetbrains to investigate. I admit I might not have this one right.
0
Comment actions Permalink
Synchronize doesn't come up in searching pycharm help;  however, reading through it more, "synchronization" does.   However, one of the vcs's uses this term (Perforce?) and most of the doc refers to this.


If you feel that documentation for this menu item is not sufficient, just file an issue (describing what should be there instead). It is my experience that JetBrains is quite responsive in that regard.

I have had several cases of inconsistency where pycharm loads and finds that the file was "changed externally" and needs to reload it.  In my case the file was not edited by another application:  rather, the file was edited by a copy (licensed and paid for–I bought after 4 days of trial) of pycharm on another machine.  This behavior seems to confirm that pycharm loads project files from the cache rather from disk.  I don't think it should do this.  It should always load from disk.


Technically, changing a file on another machine is an external change to the file. Dropbox automatically syncs that change back to your local machine in the background (i.e. outside of PyCharms control). Then, PyCharm detects that somehow the file changed without you leaving the editor and prompts you what it should do about it. I suppose, that's what you're seeing (maybe with some variations, depending on timing).

I guess, the takeaway is that it is not a good idea to put your projects into Dropbox and work in parallel on it with multiple machines. You never know how much time the Dropbox sync will take, when it happens and when PyCharm notices about those changes. Also, there's no reasonable way to resolve any conflicts that might be causes by this. There are just too many moving parts involved.

If you need synchronization, I strongly recommend you to use source control (e.g. Git) and not put your projects into Dropbox.
0
Comment actions Permalink
Thanks again.  I don't think Dropbox is the problem.  I don't use it for versioning, as stated.  I commit to git when a work item is successfully completed.

When Pycharm flags the file as "changed externally," it has done the RIGHT thing.  That is what should happen and it IS an external change.

The problem was that just ONCE when that should have happened, it DIDN'T happen.  

Let's just call it closed for now until I can find some symptoms that can be reproduced.

It is still not clear to me that when pycharm relaunches whether it opens the last project from disk or from the cache.  That would be good to know.  Anyone know?
0
Comment actions Permalink
All files are read from disk.
0
Comment actions Permalink
Thanks.   The problem was caused by dropbox's intermittent outages last Friday through Sunday (Jan 10-12).  On Friday, dropbox was intermittently down.  Sometimes and on some machines syncing seemed normal or so I thought, but really it was quite spotty.  As a result file system files were not updated on one of my machines.  Nothing at all to do with Pycharm so I apologize.

(Just to note again:  dropbox is NOT used for version control.   It just keeps the working files and local git project folder in sync across my own machines–not across team members.)
0

Please sign in to leave a comment.