100000 files

did you ever consider the idea of using one big file
e.g. for vcs vault, instead of the very great number of
individual files you are currently using?

I'm just running a virus scan and it takes a considerable
long time to scan the IDEA vcs vault because of the huge
number of files there.

Also, I think, file access to huge numbers of individual
files will be slower than using one big file.

+ disk fragmentation will at some point be an issue,
regardless of how big your disk is.


M.

0
24 comments

We have a similar problem with SmartCVS. Does somebody knows a fast and
effective library to store hierarchical data (like file information)?

Tom

0

Why not just turn off virus check on the repository location. Do you seriously expect a virus to slip into there?

On top of that... Having less but bigger files will maybe reduce I/O but will certainly increase memory-footprint. So the gains might not be as good as you think...


Florian Hehlen

0

Why do you think memory footprint would increase? AFAIK any OS is capable of
dealing with partial file mapping/swapping effectively.
On the other hand, this problem has been raised internally in JB for quite a
few times, but according to measurements, having all lvcs repository in one
file does not bring any performance gains.
So reworking this was considered a low priority task.
Eugene.

"Florian Hehlen" <florian.hehlen@ubsw.com> wrote in message
news:21563734.1114506364994.JavaMail.itn@is.intellij.net...

Why not just turn off virus check on the repository location. Do you

seriously expect a virus to slip into there?
>

On top of that... Having less but bigger files will maybe reduce I/O but

will certainly increase memory-footprint. So the gains might not be as good
as you think...
>
>

Florian Hehlen

>
>


0

So.. hand me your phone number and I'll direct our local server-room janitor to you when he wants to know why I'd like the virus software to not search parts of my disk.

0

Patrik Andersson wrote:

So.. hand me your phone number and I'll direct our local server-room janitor to you when he wants to know why I'd like the virus software to not search parts of my disk.


OT: For "smart scanning" anti virus software this makes much sense, as
the files are scanned before they are written to disk.

0

I don't see what your point is. A full scan of my 40G disk on the laptop takes 2-3 hours. Two of those are scheduled each day. A huge portion of that time is spent "dissection" the gazzabazillion files in the idea system folder.

0

You have 40GB in one partition? Wow!

Tom

0

How about LDAP?

Amnon

Thomas Singer (MoTJ) wrote:

We have a similar problem with SmartCVS. Does somebody knows a fast and
effective library to store hierarchical data (like file information)?

Tom

0

> How about LDAP?

Do you know a Java-based library, which we could install with SmartCVS?

Tom

0

No. I said 40G disk.

0

Here's what I came up with using Google:
http://sourceforge.net/projects/javaldap/
http://freshmeat.net/projects/javaldapserver/
http://directory.apache.org/

Amnon

Thomas Singer (MoTJ) wrote:

> How about LDAP?

Do you know a Java-based library, which we could install with SmartCVS?

Tom

0

Tom

You have 40GB in one partition? Wow!

>

My PowerBook has one 80GB partition, and my iMac has one 250GB partition.
I'm happy with that. Shouldn't I?

Alain

0


>

My PowerBook has..




, and I forgot to add the most important: It's a Mac, so I don't use a
virus checker.
:) :) :)

Alain

0

Just curious, what's wrong with that?

Vince.


0

The 1st thing we do with every machine we get is partitioning, so that the
OS and the user data are stored on different partitions. This allows easily
to restore/recover the OS while the data are not destroyed.

Oh, I know the answer, Mac OS X does not crash...

Tom

0

It's a Mac, so I don't use a virus checker.
:) :) :)


Incredible! I use Windows machines since 10 years and also did not got any
virus yet (except those in emails; but one does not need to click at them).
:knockonwhood:

Tom

0

Thank you very much, I will take a look at them.

Tom

0

Tom,

>> It's a Mac, so I don't use a virus checker.
>> :) :) :)
>
>

Incredible! I use Windows machines since 10 years and also did not got
any virus yet (except those in emails; but one does not need to click
at them). :knockonwhood:

>


I must be honest, and confess that I've never used any virus checking
software on Windows either :)
You just have to be careful.

Though, on Mac I feel I don't need to be careful. For now.


Alain

0

Tom

Oh, I know the answer, Mac OS X does not crash...

>
See, you get it.


More seriously, I have a full bootable copy of my disk, on an external
Firewire disk, that I made with Carbon Copy Cloner (a free tool).
To use it, I just need to press one key on my keyboard during the boot.
I can then check/repair my main disk, and restore it from the copy.


Alain

0

Thomas Singer (MoTJ) wrote:

We have a similar problem with SmartCVS. Does somebody knows a fast and
effective library to store hierarchical data (like file information)?


How quick is access to files within a jar? assuming no compression?

i.e. have a ~/.IntelliJ/system/cache.jar which inside that is thousands
of files?

The OS and native filesystem sees it as one large file, but Java could
see it as just individual files?

Not sure what the hit would be on huge jars thou...

0

Mark Derricutt wrote:

i.e. have a ~/.IntelliJ/system/cache.jar which inside that is thousands
of files?


Ok, drinking my coffee now and waking up, and realising I was commenting
on something compleately different to what the thread was actually
talking about, maybe....

Next...

0

In article <d4m4im$8b6$3@is.intellij.net>,
Alain Ravet <alain.ravet@biz.tiscali.be> wrote:

Tom

Oh, I know the answer, Mac OS X does not crash...

>
See, you get it.


More seriously, I have a full bootable copy of my disk, on an external
Firewire disk, that I made with Carbon Copy Cloner (a free tool).
To use it, I just need to press one key on my keyboard during the boot.
I can then check/repair my main disk, and restore it from the copy.


You can also just boot up from CD or DVD and do the same :)

R

0

The problem with jars is, that changing the information in a few files needs
to copy the whole jar.

Tom

0

If you want to store your info quickly why not consider a embedded database ?

There are some available.
I do use PERST (GPL) : http://www.garret.ru/~knizhnik/perst.html

Regards,
Sebastien.

0

Please sign in to leave a comment.