Learn More
Resend Email
100
Mayhem93
8y

Back-end as a service for Real-time applications, based on a multi-layered micro-services stack

Project Type
Existing open source project
Summary

Back-end as a service for Real-time applications, based on a multi-layered micro-services stack

Description
Telepat aims to be a real-time backend to be used in applications where large amounts of data are ingested, and clients get notified of every change in data they subscribe to, with high scalability and high availability. Notifications are sent to clients based on subscriptions, each client has to subscribe to a specific channel in an application to receive the initial state (usually, a subset) + notifications (eventually, as data is updated). High concurrency and throughput are backed up by a best-effort delivery strategy for the whole stack, and a multi-layered database which consists of two systems: * Main database: ElasticSearch. Everything is persisted here, all the application's objects exist here to be queried. * Redis: everything volatile is stored here (volatile means that the whole ecosystem should not suffer too much if the whole database is flushed): device info and subscription info (also other caches; atm only count caches) Telepat consists of an API layer, workers, the databases mentioned earlier and a messaging queue system: * API: responsible for authentication as well as data validation. It's the part responsible for collecting data. * Workers: responsible for aggregating and writing data sent from the API. Also they are responsible for sending the right notification to the right device (there are types of workers, each having a precise role). * Message queue: support for RabbitMQ and Kafka The workers: I'll just name them here: aggregators, writers, transport managers, client transport workers (APN, GCM/FCM, Sockets) If you're interested in this project here are a few places where you can start: https://github.com/telepat-io and https://gitter.im/telepat-io/support and say hello to us :). We've got quite a lot of issues in the backlog comprising of bugfixes, new features and a lot of improvements (though the issues backlog is not present on github), so there's plenty of work to do. Benefits: * you will get a lot of exposure different technologies in a layered base micro-services architecture where scalability and performance are key * you get to collaborate with a team of passionate and skilled web hackers :). Bonus: we are very helpful when it comes to newcomers.
Tech Stack
* Node.js (express.js, socket.io, mocha, chai istanbul travisCI) * ElasticSearch * Redis * RabbitMQ and/or Apache Kafka * Docker
Current Team Size
3-4
URL
https://github.com/telepat-io
Favorite
Ranter
Comments
  • 3
    8y
    Yes, one of the scopes of this collaboration is help both parties: expose newcomers to new technologies by adding practical experience to their career and helping us improve and develop a cool project used by hundreds of people.

    If you want a starting point you can reach out to me on gitter (https://gitter.im/telepat-io/...) or github
  • 0
    8y
    I like the idea. Great approach
  • 2
    8y
    I can help out with front end! Where do I sign up?
  • 0
    8y
    @lmtriet I guess you have an jnterest (:
  • 2
    8y
    We don't have front-end needs at the moment, unfortunately. The only closest thing to it is a new feature "supervisor and creator of telepat infrastructures" which assumes a web interface.

    But anyone is welcomed. Learning new things is always a good thing.

    I'm going to add some github issues tomorrow (I'm on UTC+2).

    For anyone showing interest: I'd follow the task demo app + how to set up a telepat server (without docket for dev purposes).
  • 0
    8y
    I've added a few issues on the telepat-api repo
  • 1
    8y
    Dropped backend for ML, I think it's time to get back. Might need a refresher tho but I am onboard 😃
  • 0
    8y
    Well there's no reason why you can't mix those two into creating a world changing product ;). And as said earlier, a good starting point would be to have a running telepat instance, you can drop by gitter if you're having trouble. Have the Android app so I'm getting notifications for new messages.
  • 0
    8y
    How much multi is the multi-layered stack?
  • 0
    8y
    @i1990jain Well, as briefly presented in the original collab, it's consisted of the following services, each having a precise role:
    * API: the only gateway clients interact with telepat
    * Aggregation service: stores and preprocesses the information received from the API, in redis
    * Writer service: effectively persists the changes represented as deltas in redis
    * Transport manager: sorts devices and subscriptions by transport type in order to determine which transport worker receives data
    * Transport woroers: apn, gcm and websockets which ultimately sends the notifications to the clients
    * ElasticSearch: main database
    * redis: cache database, used for devices and subscriptions as well
    * RabbitMQ: communication between layers
  • 1
    8y
    PS: every component of telepat is independently scalable (horizontal)
  • 1
    8y
    Hey, is it still open?
  • 0
    8y
    Yes it is. There's no close time ^^ (talking about the project itself)
  • 1
    8y
    Wil be checking this out to see if there's anything i can help with. Sounds interesting
  • 1
    8y
    I am willing to join ;)
  • 2
    8y
    @AllenII @anekix great, looking forward to seeing you guys on gitter :D
  • 1
    8y
    @Mayhem93 any requirement for C/C++ or is it only in node?
  • 1
    8y
    Well, we don't have a C/C++ client for telepat so there's a start. The backend is mostly nodejs javascript, native node modules written in c++ can be done as well
  • 0
    8y
    Good.

    I have skills in less, css, html, php (codeigniter and laravel) and MySQL
  • 1
    8y
    @messhias well here's your chance to learn something new like nodejs ^^
  • 0
    8y
    @Mayhem93 i think that was a good ideia, how can i start? There's some reference material you could indicate to me?
  • 1
    8y
    @messhias Well read the collab and the comments I've made, all the necessary info is there.
  • 1
    8y
    Looks awesome. And much better than my company's back end... Who writes ONE test class with 5.5k lines seriously!!!

    I will definitely contribute once I have a bit of spare time
  • 1
    8y
    @antonis179 Awesome :D. I'm in the process of writing the unit tests for telepat-models in my free time.
  • 0
    8y
    @Mayhem93 Awesome awesome n awesome..! I was just thinking to learn Node.js and also was wondering how these BASS technologies work on Node.js.. May be I'm at the right place.. I can do anything to contribute in this project.. Will soon do.. Great Job 👏👏
  • 0
    8y
    @sanketthecoder awesome to hear this :D. We got a lot of ideas and improvements on the backend side of things. If you need any help, we're on gitter.
  • 0
    8y
    @Mayhem93 100%.. Will definitely get to you shortly.. Just eager to join your project and contribute.. ☺️
  • 0
    8y
    Would have worked if it was written in a native language like Golang :P
    I have worked in all the technologies you have used.
    I don't like JavaScript :P
    The architecture looks great for real-time use case
  • 1
    8y
    m in..terested..!!! will ping u back soon to check if i can fit in somewhere to contribute to this awesome(in the making) tool.. (:
  • 0
    7y
    Hey I'm totally interested in this. I have experience in node, JavaScript and recently created a Mean App myself. How do you propose I get started?
  • 0
    7y
    @abhinavr888 well you can start off by getting telepat run on your system (use the develop branch for all 3 main repos), other than that there are some extra steps needed to configure ElasticSearch. If you got any questions or just need help you can drop by gitter and ask ahead ^^
  • 0
    7y
    @Mayhem93 why are you writing the backend in NodeJs?
  • 0
    7y
    @metalhead1247 because the type of workload that's required for this system demands a highly scalable event driven backend platform, not only that but the community is huge + me and my colleagues already had some nodejs experience.
  • 0
    7y
    @Mayhem93 Yes that's there about the large community, but I'm sure once you start scaling you'll come to know NodeJs is shit 😝, i had to rewrite the backend for our Realtime IoT micro service architecture.
    No doubt NodeJs will be able handle your I/Os but if u start doing any minor processing you'll get poor performance and lags. Golang can handle more I/Os than NodeJs and also highly scalable and highly concurrent. When processing realtime data u need multithreading. We used a similar stack as yours. You could try Influx Db instead of elastic search 😛 . I would use nodejs only for Apis 😉. Well you can disagree with my inputs based on your use case.😈Are you using Aws services for the deployments?
  • 1
    7y
    @metalhead1247 yes aws. I want horizontal scaling more than vertical one. I don't need multithreading for that. There are other things to consider, not just raw throughout of a system.
  • 1
    7y
    @NyxMC The current Android SDK should not have a lot of dependencies on the Android SDK and the codebase itself should be compatible with a pure Java 7 codebase. The only really Android specific areas are around the notifications transport (we're using GCM, we should have an implementation for something like socket.io) and the logic around creating a device-specific identifier, where we're currently using the Android installation ID. Hence, the best route would be to probably improve the current Android SDK and allow the project for mulitple artifacts (Android and pure Java .jar files).
  • 1
    7y
    To anyone who's still fighting following this collab: we've released a new more stable version of telepat and also tested and fixed the docker deployment process (even works on my windows machine).
  • 0
    7y
    Very interested! Is this still open?
  • 0
    7y
    @sagarnar it's always open ^_^
  • 0
    7y
    This is Awesome I was an intern and in the last 6 months experienced Node.js also having created my own proxy API's using my companies BAAS (Which is paid). Have experience with Express & request npm modules. Used Chai & Mocha for Unit testing. Can I help somewhere if I get the time?
  • 0
    7y
    @MacDev of course, there's always room for improvements. If you have any questions don't hesitate to contact us on gitter
  • 0
    7y
    I am more of an Java guy in terms of experience. As for node.js I am a real beginner.
    Node.js/Android - Beginner.
    Java - medium.
devRant © 2021 Hexical Labs LLC
Privacy Policy  |  Terms of Service
Add Comment