31
amitgupta
200d

निम्न (Nimn) schema less data. 80% smaller than XML, 60% smaller than JSON.

Project Type
Existing open source project
Summary

निम्न (Nimn) schema less data. 80% smaller than XML, 60% smaller than JSON.

Description
It keeps the object structure away from actual data. Hence it is smaller. Specification: http://github.com/nimndata/spec JS Implementation: http://nimn.in/nimnjs suggestion, feedback, contribution are welcome. Bookmark it on github for further updates.
Tech Stack
need to implement in mostly used programming languages
Current Team Size
1, looking for contributors
URL
Comments
  • 0
    Would be great if it works with array
  • 1
    @nobes do you mean the schema should be given in array. Or the data?
  • 0
    Insert a json array and get a nimn array
  • 1
    @nobes that is supported. Try on http://nimn.in/nimnjs
  • 1
    On your page it works great but not in nodejs
  • 2
    Now it works! I didn't change anything but it works!
    Greate work! Keep it up!
  • 0
    The problem was, i inserted a string and i need to insert a json object
  • 2
    @nobes Thanks. I need to implement it in other languages as well. But so many things to do. So looking for collaborators and contributors who can help to make it better and support to implement in other languages.
  • 0
    @amitgupta Which languages do you plan to implement?
  • 1
    @nobes it'll be a good idea to accept json string. Will raise an issue.
  • 1
    @nobes currently it is implemented only in nodejs. I need to implement it in java, c#, python and other. So everyone can use it.
  • 1
    I will fiddle a bit c++ and nimn. I will notify you if I have something publishable.
  • 0
    @JoshuaBehrens thanks a lot. I'll do some changes in the spec today. So that any implementation can be faster.
  • 1
    Can it be a good idea to publish some algorithm? So anyone can reference it to write the serializer in their preferred language.... I mean programming language :D
  • 1
    @amitgupta I will lookup the routines in the npm package and see what is transferable to c++/C#. If all language implementations share the same structure you only need one documentation but this will not work due to different language features.
    Any ways we can talk about your restructuring?
  • 0
    Thanks @JoshuaBehrens . Please check http://github.com/nimndata/spec

    It contains specifications.

    The POC in node js package is having a PR to be merged and some issues to be implemented.
  • 1
    this looks perfect for a project of mine - are there any libraries for using this with c++?
  • 0
    @j4cobgarby not yet. I've implemented it in js only
  • 1
    @amitgupta okay, thanks:) looks good, anyway
  • 0
    @j4cobgarby if I implement it I would implement string - tree conversion for std container, boost ptree and maybe Poco container.
  • 0
    @amitgupta I give it a try https://bitbucket.org/JoshuaBehrens...

    Looking forward to the object closing character.
  • 1
    @JoshuaBehrens I've updated the spec. I've to implement it in nimn js. I'll start that tomorrow.
  • 1
    @JoshuaBehrens I've created new parser in nimnjs which is faster than previous one but still slower than JSON.stringify

    https://github.com/nimndata/nimnjs ... src/parser.js
  • 0
    'stubmatic' can help to mock nimn HTTP response.
  • 1
    @amitgupta I will check that change out
  • 1
    Hey guys, I'm missing something here or I didn't understand it correctly. How it is different than protobufs?
  • 2
    @Gowtham95india I was not aware with protobuf before. But after you mentioned I read and compared the size. I'm not sure about it's rules but it looks like both concept are same with different syntax. Event the end result.

    Thanks to let me know, I'll not waste my time further.
  • 0
    @Gowtham95india Today I again compared Nimn with protobuf, thrift, and Avro. It seems more like Avro not protobuf. However all these 3 formats/protocols does field level compression. While I more focus on data structure. Hence the size of encoded data is a bit higher (308 of protobuf against 311 of nimn). But it improves the searching speed.

    So I'll continue with this project and will do necessary changes to make it faster.
  • 0
    @amitgupta I haven't gone through the full spec. From the description what I have read, I didn't understand the difference between this and protobuf. Thus I raised concern.

    Lemme go through the full spec. Will be happy to contribute if it's worth the effort.
  • 0
    Thanks @Gowtham95india . I'll try to write an article to show differences in next week.
  • 0
    @amitgupta
    I understand that we are busy with our regular work as well. Have you managed to write the blog you are talking about?

    Also, you said searching will be much faster, can you explain it in a bit more?

    I was going through specs of pb, avro, thrift, and I couldn't come with any advantages of using nimn over others.
  • 0
    @Gowtham95india the last version of nimn was very much similar to avro. While doing this comparison I found how can I rid of sanitize process. And I've also done some changes in nimn format to make it smaller and faster. I'm re-writing nimnjs for this POC. I'm almost done with encoding part and will complete decoding part in few days. I'll update you as soon as I'm done.
  • 0
    @amitgupta
    I admire your consistency. But by doing this, what are we achieving?
    What are the benefits/features that existing things are loosing or missing?

    The reason behind why I'm pushing you to define this is because it clearly puts in front of everyone face..

    Boss this is what you are loosing by using existing one, this is what you will be gaining using nimn. So take your pic.
  • 0
    @amitgupta
    Lemme know if it is explaining on a call.
  • 0
    @Gowtham95india As I said previously, I was initially not aware that storage efficient formats are already there. Hence I came up with Nimn. And then stopped when you mentioned about protobuf.

    However I resumed my work as I found that there is something better can be done in terms of speed and storage. But I need to complete a POC to prove my point.

    So please wait for some more days. And I'll update you as soon as I'm done.
  • 0
    @amitgupta
    Sure bro; will wait for your update!
  • 1
    @Gowtham95india I've shared a file with you on twitter to give a small detail about how nimn can benefit.
Your Job Suck?
Get a Better Job
Add Comment