Can you include .3MF to the list of re-compressible formats?

[By-Tor]

@spwolf:

point with paf is that we can make something unique with it that nobody else has - compression for many popular formats… weather it becomes next biggest thing is something else, we plan to use it in our next backup utility as well, where nobody does any of these things and space savings will be significant.

I totally agree. I wasn’t clear on how far reaching you wanted things to be.

[spwolf]

@By-Tor:

I totally agree. I wasn’t clear on how far reaching you wanted things to be.

it is massive undertaking though, and it takes a while…

but it will be worth it for instance for backups, where we will be able to do 50% more efficient backups that rest of the current utilities :)

[spwolf]

keep in mind we have been “researching” this for past 2 years, so we are not staring from 0 here…

things we did so far:

• ver 1 jpeg compression - 3x speed of wz jpeg with slightly better compression. Aim in PAF: maintain similar super speed while adding extra 20% compression compared to WZ Jpeg.
• ver 1 differential and versioning system - compared to leading Mozy backup service, we had 30% smaller differential backups (which means 30% faster, 30% more effcient, 30% less costly for bandwith and storage space), without strong compression implemented… Goal is to tie it together into PAF and enable stronger compression that would give us 50% gain total.
• Various multicore research into improving speed for other operations during compression, not just compression codec, that should add extra 20-30% speed improvements over current formats (with similar codecs).
• modification of lzma2 codec into pa-lzma, to fit our format better, and better overall performance (to be released as open source), this is 80% done as it is…

[spwolf]

New formats idea site:
http://ideas.powerarchiver.com/

[Socrates]

The sample size on your survey is so small, I would caution against taking it too seriously.

If you found a way to ask a broader audience what they want, I would be shocked if multi-volume support would be in the top five.

There may not be a better way to reach a broader audience, and, if so, having established voting, you probably have to act based on the suggestions you received. Still, I fear that in so doing you be will be spending a lot of effort on a feature that really won’t appeal to that many people. I suspect 5-7 of the other options would be more appealing, even if they are not ones I would use.

So if you try this again, you might explore ways of getting input from a larger sample.

[spwolf]

no need to worry, while ideas site is interesting we still have our own schedule and goals to work with (which is getting better compression on things that are not compressible currently).

multi volume feature is really simple to implement, but we probably would not do it that way if we didnt get enough votes.

[Socrates]

If it is simple, then that’s a compelling reason to do it.

What I often don’t know is how difficult it is to implement a new feature. Some might seem easily, but be difficult (or next to impossible) while others that seem difficult might be a snap.

[spwolf]

@Socrates:

If it is simple, then that’s a compelling reason to do it.

What I often don’t know is how difficult it is to implement a new feature. Some might seem easily, but be difficult (or next to impossible) while others that seem difficult might be a snap.

hardest things on that lists are new codecs for pdf, jpeg, mp3… thats both hard and time consuming. Everything else on that list will probably take less time all together (!) then building special jpeg compressor.

[Socrates]

I suppose I would have thought special compression would be especially difficult. But not that much more difficult.

Thanks for letting us know.

At some point (and if not too complicated), could you explain the relative difficulties of building new compression from pds versus jpgs. I assume the former would be easier (more white space). Perhaps, though, that assumption is borne of ignorance.

[spwolf]

#1 goal for the format: special codecs for file types that are most used currently yet can not be compressed by current tools.

basically this is compression for complicated (already compressed) file types.

so what you have to do is take file apart, and divide it into parts that can and that can not be compressed (which is why jpeg, png, docx, pdf that is compressed, can not be compressed further usually)… then compress the part that can be compressed with special codec designed for that format. All of this is done transparently to the user of course, and usually quite fast if done right.

But for each format (mp3, jpeg, png, pdf, docx, odt, etc), special codec is required. So there is a lot of development work to be done here. There are also no examples of such work, and only very few utilities do it - for instance, Stuffit has a lot of special compressors, but it is available only if you give out your credit card (no actual free trial), while most other utilities do not have anything but general codecs.

Reality is that most people compress things that are already compressed, so using zip, or rar or 7zip on most things people usually backup or send via email will not result in great savings, or sometimes savings at all. If you compress jpegs to send over email or upload somewhere or simply backup, you will not gain any compression. On the other hand, with special jpeg codec, you can expect 20%-30% gain on your full album of pictures.

For instance zip has one most common codec which is deflate. WinRar has single codec too.
Now .paf/pa/power will have 5-6 at least within next 2 years. So you can imagine how big the task is. But the gains are big too so it is worth it.

[spwolf]

difference between pdf and jpeg is that while for both, you have to develop special recompression algorithm, pdf uses deflate compression from zip and then its contents once opened up, can be compressed well with general codecs.

on the other hand, jpeg has many variations so first recompression has to be able to take apart various jpegs and then you have to build completely custom codec to compress that picture inside.

So basically it is double the work compared to PDF.

Advantage with PDF is that we can use a lot of the code to recompress other formats like PNG, DOCX, ODT, SWF, since they all use deflate to give them (weak) compression.

For instance, DOCX containts XML files inside that are compressed with deflate. However since they are compressed, if you try to compress them again, you will gain very little… but if you unravel weak compression and then apply stronger one, big gains are possible.

Here is test example.

1. Contract in DOCX format - 104 KB
2. DOCX compressed with 7zip - 95KB
3. DOCX recompressed properly - 64 KB

So thats 40% gain on DOCX file for instance. Imagine if you have many of them on your computer, or if your company sends many via email or backup service… Time and cost savings are quite significant here.

Now actual % gained is different for different formats and there are further optimizations possible (for instance detect pictures and text inside single file differently and compress them with their own codecs), but you can see how much potential this has.

Main thing here is that it has to be done seamlessly and it has to be fast, otherwise people will not use it. And then we come to #2 part of new format - multicore optimizations.

[Socrates]

thanks!

[RJWaring]

I would have to say that Jpeg, and PDF Compression is a good idea! as your possibly aware I have brought Picture compression up a few times in the past with a good warrent for it.

But with a list aslong as the above there are so many other good things to choose from.

I have made some votes. :o)

[davidsplash]

i would like the new format to introduce some sort of protection so that if the archive is changed it tells you by whom and can stop this if you dont wnat it updated but would like people to extract files from it only

[davidsplash]

One idea would be to have time limited archives that actually destroy themselves after a set amount of time. saw this in america well worth it if you are sending things over the internet. They also had archives that only let you open them a set number of times.

[davidsplash]

to sound a downer you can not have every feature in the new format as that would slow it down.

What we would like to see is several new archive formats .

For instance pdf compression

wave compresion

etc

would love to see all these new formats use the plugin feature in pa.

[FiatLux]

The option that you when in the “with options” interface then may choose both Encrypt and make SFX in one operation, and implemented using a technique that can handle both many andd/or large files….

[BinTech]

RAR-like recovery is definitely a thing I would love to see.

[MarcFou]

PDF, JPG, PSD and PSP are most important to me.