git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Nested & Dynamic Schema Types in Apache Atlas


Shivji Kumar,

> In java lingo, we are simply looking for something of the
> sort of map<string, Object> representing attribute name
> and any subtype of Object.
Entity-type 'Process' has 2 attributes, 'inputs' and 'outputs',
of type 'array<DataSet>'. Each element in the array value can
be of a sub-type of 'DataSet' - like hive_table, hive_column, hdfs_path.
This is used to capture lineage of data, for example from hdfs_path
to hive_table to hbase_table.

Similarly to use of array here, map can be used as well; key should
be of primitive type - like string, int, long; value can be of any type,
including entity-type.

Hope this helps.

If you need further details, can you please add specifics of usecases
you are looking to model.

Regards,
Madhan




On 5/5/18, 10:16 AM, "Shivjikumar Jha" <shivjikumar.jha@xxxxxxxxx> wrote:

    Hi Everyone!
    
    We are moving to Apache Atlas to store all our metadata in the organisation
    and need help in modelling some of our existing metadata.
    
    One of the types we have is EVENT. Our existing schema for an event is
    simply a json (serialized from java classes - DTOs). These DTOs could then
    be composed of primitive types, array(s), map(s) as well as some arbitrary
    instances of some other class(es) etc. Please help us in understanding how
    attributeDef of event should be expressed in this case. In java lingo, we
    are simply looking for something of the sort of map<string, Object>
    representing attribute name and any subtype of Object.
    
    We are taking inspiration from the *Table* type in the quickstart script.
    The *Table* type has, in addition to other attributes, an array of columns (
    *array(Column)*). The *Column* type is then composed of more primary
    attributes. In our scenario though, the nesting could be of any level and
    demands more flexibility in number and type of attributes. The column could
    be composed of some other structure that is not known beforehand.
    
    Please let me know if you have any questions on our scenario.
    
    Regards,
    Shivji Kumar Jha | Senior Software Development Engineer, Swiggy
    http://www.shivjijha.com
    
    Regards,
    Shivji Kumar Jha | Software Development Engineer-II
    http://www.shivjijha.com | 8884075512
    
    -- 
    
    IMPORTANT NOTICE: This e-mail, including any attachments, may contain 
    confidential information and is intended only for the addressee(s) named 
    above. If you are not the intended recipient(s), you should not 
    disseminate, distribute, or copy this e-mail. Please notify the sender by 
    reply e-mail immediately if you have received this e-mail in error and 
    permanently delete all copies of the original message from your system. 
    E-mail transmission cannot be guaranteed to be secure as it could be 
    intercepted, corrupted, lost, destroyed, arrive late or incomplete, or 
    contain viruses. Company accepts no liability for any damage or loss of 
    confidential information caused by this email or due to any virus 
    transmitted by this email or otherwise.