Regarding single pass load:Fwd: Questions about Dictionnary Server

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Regarding single pass load:Fwd: Questions about Dictionnary Server

Liang Chen

---------- Forwarded message ----------
From: Ravindra Pesala <[hidden email]>
Date: 2017-05-21 23:55 GMT+08:00
Subject: Re: Questions about Dictionnary Server
To: dev <[hidden email]>, [hidden email]


Hi,

To generate global dictionary CarbonData first scan all input data and
finds unique data for each column and assign dictionary for each value. So
it is two step process. Irrespective of any new unique dictionary values
are added or not it always need to scan all data to get the dictionary.
To overcome from this issue we introduce this dictionary server. From
second load onwards if there is not much dictionaries are created in the
load then we can choose this option to improve the loading performance. It
just avoids 2 steps process to single step by generating dictionary online
while loading the data.


Regards,
Ravindra.

On Sun, 21 May 2017 at 8:41 PM, Sea <[hidden email]> wrote:

> Hi, all:
>     I have a question, when we should use DictionaryServer?