What is the difference between Minor Compaction Level 1 and Minor Compaction Level 2?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

What is the difference between Minor Compaction Level 1 and Minor Compaction Level 2?

xm_zzc
Hi, all:
  What is the difference between Minor Compaction Level 1 and Minor Compaction Level 2? And is Minor Compaction Level 2 equal to Major Compaction?
Reply | Threaded
Open this post in threaded view
|

Re: What is the difference between Minor Compaction Level 1 and Minor Compaction Level 2?

prabhatkashyap
Hi xm_zzc

There are two levels of Minor Compaction:
Level 1: Merging of the segments which are not yet compacted.
Level 2: Merging of the compacted segments again to form a bigger segment.

If you look into carbon.properties there is a property
carbon.compaction.level.threshold=4,3
This property is set by default.
Which means it will merge 4 segments at level 1 and 3 segments at level 2.

For example:
If you load 3 times then there will be segments:
Segment 0
Segment 1
Segment 2

And on the fourth load, if auto-compaction is enabled then the first four segments will merge and will create a new segment with name like Segment 0.1
Similarly, after the eighth load, Level 1 compaction will run again and create a new segment with a name like Segment 4.1 will be created. (Note: This time Segment 4 to 7 will be merged). Same will happen for next four loads and will create a new segment like before Segment 8.1. This is level 1 compaction.

Now if you see the property that for level 2 compaction the default value is set to 3. As soon as we have three Level 1 compacted segments, level 2 compaction will be triggered and will merge 3 already compacted segments to generate a new segment like Segment 0.2 (Note: Here Segment 0.1, 4.1 and 8.1 will get merged).

Also, there would be no compaction of the segment after level 2 compaction. Like 0.2 will not be merged with any other segment.

Also, note that you can run compaction using alter command if the auto-compaction is not enabled

Minor level 2 compaction is not equal to Major Compaction.

Minor compaction is based on numbers of segment and Major Compaction is based on the size of segments.

For details, you can refer to documentation: http://carbondata.apache.org/data-management.html#compacting-data

Reply | Threaded
Open this post in threaded view
|

Re: What is the difference between Minor Compaction Level 1 and Minor Compaction Level 2?

xm_zzc
Hi prabhatkashyap:
  Thank you very much for your detailed explanation. I have tested data compaction and the result is the same as what you said.

  You mean that the principle of Major and Minor data compaction is the same, right? but minor compaction is automatically executed, major compaction is based on the size of the data and manually executed, right?

  Thanks.
Reply | Threaded
Open this post in threaded view
|

Re: What is the difference between Minor Compaction Level 1 and Minor Compaction Level 2?

Erlu Chen
Hi, xm_zzc

As I know, the principle of Major and Minor data compaction is the same. But Major compaction is based on segment size and Minor compaction is based on segment number.

There is switch named "carbon.enable.auto.load.merge" which will enable compaction when loading data.
This only support Minor compaction.

Besides, Carbondata support alter table compact [minor | major].

Just specify the compact type you want then it will trigger compaction.


Regards.
Chenerlu.
Reply | Threaded
Open this post in threaded view
|

Re: What is the difference between Minor Compaction Level 1 and Minor Compaction Level 2?

xm_zzc
Hi Erlu Chen:
  Thanks for your reply.
Reply | Threaded
Open this post in threaded view
|

Re: What is the difference between Minor Compaction Level 1 and Minor Compaction Level 2?

Erlu Chen
You are welcome!

Regards.
Chenerlu.