The proprietary API I have to use which uses XML but is totally dynamic and without a schema. They basically managed to successfully merge JSON and XML to get the disadvantages of both JSON and XML.

निम्न (Nimn) schema less data. 80% smaller than XML, 60% smaller than JSON.

"So Alecx, how did you solve the issues with the data provided to you by hr for <X> application?"

Said the VP of my institution in charge of my department.

"It was complex sir, I could not figure out much of the general ideas of the data schema since it came from a bunch of people not trained in I.T (HR) and as such I had to do some experiments in the data to find the relationships with the data, this brought about 4 different relations in the data, the program determined them for me based on the most common type of data, the model deemed it a "user", from that I just extracted the information that I needed, and generated the tables through Golang's gorm"

VP nodding and listening intently...."how did you make those relationships?" me "I started a simple pattern recognition module through supervised mach..." VP: Machine learning, that sounds like A.I

Me: "Yes sir, it was, but the problem was fairly easy for the schema to determ.." VP: A.I, at our institution, back in my day it was a dream to have such technology, you are the director of web tech, what is it to you to know of this?"

Me: "I just like to experiment with new stuff, it was the easiest rout to determine these things, I just felt that i should use it if I can"

VP: "This is amazing, I'll go by your office later"

Dude speaks wonders of me. The idea was simple, read through the CSV that was provided to me, have the parsing done in a notebook, make it determine the relationships in the data and spout out a bunch of JSON that I could use. Hook it up to a simple gorm golang script and generate the tables for that. Much simpler than the bullshit that we have in php. I used this to create a new database since the previous application had issues. The app will still have a php frontend and backend, but now I don't leave the parsing of the data to php, which quite frankly, php sucks for imho. The Python codebase will then create the json files through the predictive modeling (98% accuaracy) and then the go program will populate the db for me.

There are also some node scripts that help test the data since the data is json.

All in all a good day of work. The VP seems scared since he knows no one on this side of town knows about this kind of tech. Me? I am just happy I get to experiment. Y'all should have seen his face when I showed him a rather large app written in Clojure, the man just went 0.0 when he saw Lisp code.

I think I scare him.

Dear API vendor,

Please get off your arse and learn about REST, OpenAPI, JSON Schema, XSD and basic documentation so that I don't have to guess how to use your shitty, inconsistent, RPC over HTTP service.

With Love,

Platypus

Come on guys, use those JSON schemas properly. The number of times I see people going "err, few strings here, any other properties ok, no properties required, job done." Dahhh, that's pointless. Lock that bloody thing down as much as you possibly can.

I mean, the damn things can be used to fail fast whenever you misspell properties, miss required properties, format dates wrong - heck, even when you want to validate the set format of an array - and then libraries will throw back an error to your client (or logs if you're just on backend) and tell you *exactly what's wrong.* It's immensely powerful, and all you have to do is craft a decent schema to get it for free.

If I see one more person trying to validate their JSON manually in 500 lines of buggy code and throwing ambiguous error messages when it could have been trivially handled by a schema, I'm going to scream.

I worked with this hack of a backend dev that was too lazy to add a complex(ish) object to our CMS tool. His solution?

One giant-ass text box with the label "put JSON here".

If tech people were using it I wouldn't mind, but our poor content managers have no idea what json is. Plus like... no examples, no schema... they would have to change shit then go look at the website to see if it worked. Fucking asshole.

Plus.. I mean SHIT, MAN! This was in a Node.js tool... if you have the Json parser you could just GENERATE the respective form fields. DO YOUR JOB

To me this is one of the most interesting topics. I always dream about creating the perfect programming class (not aimed at absolute beginners though, in the end there should be some usable software artifact), because I had to teach myself at least half of the skills I need everyday.

The goal of the class, which has at least to be a semester long, is to be able to create industry-ready software projects with a distributed architecture (i.e. client-server).

The important thing is to have a central theme over the whole class. Which means you should go through the software lifecycle at least once.

Let's say the class consists of 10 Units à ~3 hours (with breaks ofc) and takes place once a week, because that is the absolute minimum time to enable the students to do their homework.

1. Project setup, explanation of the whole toolchain. Init repositories, create SSH keys for github/bitbucket, git crash course (provide a cheat sheet).
Create a hello world web app with $framework. Run the web server, let the students poke around with it. Let them push their projects to their repositories.
The remainder of the lesson is for Q&A, technical problems and so on.

Homework: Read the docs of $framework. Do some commits, just alter the HTML & CSS a bit, give them your personal touch.
For the homework, provide a $chat channel/forum/mailing list or whatever for questions where not only the the teacher should help, but also the students help each other.

2. Setup of CI/Build automation. This is one of the hardest parts for the teacher/uni because the university must provide the necessary hardware for it, which costs money. But the students faces when they see that a push to master automatically triggers a build and deploys it to the right place where they can reach it from the web is priceless.
This is one recurring point over the whole course, as there will be more software artifacts beside the web app, which need to be added to the build process. I do not want to go deeper here, whether you use Jenkins, or Travis or whatev and Ansible or Puppet or whatev for automation. You probably have some docker container set up for this, because this is a very tedious task for initial setup, probably way out of proportion. But in the end there needs to be a running web service for every student which they can reach over a personal URL. Depending on the students interest on the topic it may be also better to setup this already before the first class starts and only introduce them to all the concepts in a theory block and do some more coding in the second half.

Homework: Use $framework to extend your web app. Make it a bit more user interactive with buttons, forms or the like. As we still have no backend here, you can output to alert or something.

3. Create a minimal backend with $backendFramework. Only to have something which speaks with the frontend so you can create API calls going back and forth. Also create a DB, relational or not. Discuss DB schema/model and answer student questions.

Homework: Create a form which gets transformed into JSON and sent to the backend, backend stores the user information in the DB and should also provide a query to view the entry.

4. Introduce mobile apps. As it would probably too much to introduce them both to iOS and Android, something like React Native (or whatever the most popular platform-agnostic framework is then) may come in handy. Do the same as with the minimal web app and add the build artifacts to CI. Also talk about getting software to the app/play store (a common question) and signing apps.

Homework: Use the view API call from the backend to show the data on the mobile. Play around with the mobile project to display it in a nice way.

5. Introduction to refactoring (yes, really), if we are really talking about JS here, mention things like typescript, flow, elm, reason and everything with types which compiles to JS. Types make it so much easier to refactor growing codebases and imho everybody should use it.
Flowtype would make it probably easier to get gradually introduced in the already existing codebase (and it plays nice with react native) but I want to be abstract here, so that is just a suggestion (and 100% typed languages such as ELM or Reason have so much nicer errors).
Also discuss other helpful tools like linters, formatters.

Homework: Introduce types to all your API calls and some important functions.

6. Introduction to (unit) tests. Similar as above.
Homework: Write a unit test for your form.

(TBC)

I wonder what is going on in the minds of inexperienced developers. It must be very interesting.

I just read this bit of code
The task was to implement a certain schema into a database. They were given an ORM library, and several tutorials explaining what an ORM was and how it did it.

The result were these 3 models:
- A defaultUser with all of the defaults values for the User model. It wasn't even supposed to be instantiated, just accessed by `defaultUser.fields.username.default`
- another default table for another model.
- The "main" model, containing all other models in the form of JSON fields that would contain an array with other JSON objects that would represent the instances themselves.

I will say though, they made a home-brew ORM with (most of) all the logic a normal ORM has by parsing through the "main model", except, of course, common sense

I think UPS' Api documentation and service must be the worst documented and build API I have ever seen from a corporate.

1. The developer website is a mess. A total mess. You can barely find the API type you are looking for.

2. When you get the API and download the documentation, the files, .pdf etc is still a mess. Pages long that most are craps.

3. Each request returns Status Code 200. Even if it is an error. This blew my mind.

4. Each request, based on error type or based on tracking activity returns different JSON schema.

For example, the JSON Schema for a shipment in transit is different from JSON schema for a shipment that has been delivered. A shipment that has been returned, a shipment that required signature etc. They are different from each other.

5. And the worst. They do not provide with test tracking codes. I have found some on internet, but they do not work in development and production environment.

It's a shame that people don't want to use F# but prise C# for how cool it became and continue becoming. At the same time, little do they know that many of the features were simply drawn from F#.

It's just rediculous how far this OO and C-Style syntax crap has progressed. They keep copying things from functional langugages, making the initial language to be a monstrocity like C++ is now, insted of just using languages like C#. I mean, it was right there before C#: async/task, immutablility, records, indexes, lambdas, non-null by default, who the hell knows what else.
Besides, many people (in my company at least) are just blindly overengineering with patterns and shit, where a simple function would be just enogh.

Watch some some NDC talks about F#, in particular those of Scott Wlaschin. It's just better in so many ways: less noice (I'm looking at you, brackets, commas and semicolons), the whole LOT of type inference and less duplication (just look at the C# signatures of linq methods - it's difficult to read them), immutability by default, non-nullable by default, ADTs and pattern matching, some neat features like type providers (how many times have used "paste special" or an online tool to create C# classes from a JSON/XML file, and how many times have your regenrated it because of schema changes?) and units of measure.

Of course, in some cases it's not optimal, in some cases mutable datastructures of C# are better for performance. But dude, how many performance critical systems have you wrote in C#? I mean, if it comes to performance you should use Rust or C++ or C after all.

*sighs*

Why does MySQL call databases schemes? Makes absolutely no sense.

Schema is the metadata around your tables or documents. What is Json Schema? Metadata around a json object.

MySQL, databases contain data. Schemes do not! Bad DB technology. Bad...

Did it ever happen to you that you were needed to map a mental and overly complex for no reason db schema to a simple json, and you hate yourself more every key you press because rewriting it more efficiently and simply is never an option?
How do you cope with managers and legacy code?
If it works does not necessarily mean that we should keep it, jesus christ.

Implement a rest API for elasticsearch.
Follow the client's index's mapping.
Generate json document from Java pojos, given by the client.
Jsons don't match the schema mapping, one (at least) field, for geographic coordinates, is in another format.
Ask the client for explanation.
Client response, after 6 hours:
"We build it in this shape so you have to convert them to another format before posting into ES".
What the hell is wrong with you?!

Does anyone know of an API/JSON schema designer for iPhone?

We ended up finding ourselves with a bunch of tables that have mostly the same columns, but differ by a few. Every time we consume a REST API, we store the `access_token`s and expiration dates and the other OAuth data. However, each provider has slightly different requirements. For example, we store email addresses for email api's, other providers require us to store some additional information, etc. etc.. I'm tempted by the flexibility and lack of schema brought by document databases, but not enough to use one since they're generally slower and we already have everything in SQL. So I got the idea of using JSON columns to alleviate this issue: have a single table for all REST integrations (be it outlook or facebook), and then store the unique integration data inside of this JSON column for "additional data". This data is mostly just read, not filtered by (but ocasionally so). Has anyone had experience with this? How's the performance of JSON fields? Is this a good practice or will it get harder with more integrations?

A genuine question

How can one achieve the following with ld+json schema, it really looks cool or is it done with something else?

What if, the newly added JSON datatype in mysql is a way to provide mysql with no-sql-like capabilities.
I mean, some would prefer no-sql cuz they beleive that the tables schema will evolve a lot.
An extra column in mysql table with json datatype called "custom_fields" would do the trick.
What do u think ?

I am trying to extract data from the PubSub subscription and finally, once the data is extracted I want to do some transformation. Currently, it's in bytes format. I have tried multiple ways to extract the data in JSON format using custom schema it fails with an error

TypeError: __main__.MySchema() argument after ** must be a mapping, not str [while running 'Map to MySchema']

**readPubSub.py**

import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
import json
import typing

class MySchema(typing.NamedTuple):
user_id:str
event_ts:str
create_ts:str
event_id:str
ifa:str
ifv:str
country:str
chip_balance:str
game:str
user_group:str
user_condition:str
device_type:str
device_model:str
user_name:str
fb_connect:bool
is_active_event:bool
event_payload:str

TOPIC_PATH = "projects/nectar-259905/topics/events"

def run(pubsub_topic):
options = PipelineOptions(
streaming=True
)
runner = 'DirectRunner'

print("I reached before pipeline")

with beam.Pipeline(runner, options=options) as pipeline:
message=(
pipeline
| "Read from Pub/Sub topic" >> beam.io.ReadFromPubSub(subscription='projects/triple-nectar-259905/subscriptions/bq_subscribe')#.with_output_types(bytes)
| 'UTF-8 bytes to string' >> beam.Map(lambda msg: msg.decode('utf-8'))
| 'Map to MySchema' >> beam.Map(lambda msg: MySchema(**msg)).with_output_types(MySchema)
| "Writing to console" >> beam.Map(print))

print("I reached after pipeline")
result = message.run()
result.wait_until_finish()

run(TOPIC_PATH)

If I use it directly below

message=(
pipeline
| "Read from Pub/Sub topic" >> beam.io.ReadFromPubSub(subscription='projects/triple-nectar-259905/subscriptions/bq_subscribe')#.with_output_types(bytes)
| 'UTF-8 bytes to string' >> beam.Map(lambda msg: msg.decode('utf-8'))
| "Writing to console" >> beam.Map(print))

I get output as

{
'user_id': '102105290400258488',
'event_ts': '2021-05-29 20:42:52.283 UTC',
'event_id': 'Game_Request_Declined',
'ifa': '6090a6c7-4422-49b5-8757-ccfdbad',
'ifv': '3fc6eb8b4d0cf096c47e2252f41',
'country': 'US',
'chip_balance': '9140',
'game': 'gru',
'user_group': '[1, 36, 529702]',
'user_condition': '[1, 36]',
'device_type': 'phone',
'device_model': 'TCL 5007Z',
'user_name': 'Minnie',
'fb_connect': True,
'event_payload': '{"competition_type":"normal","game_started_from":"result_flow_rematch","variant":"target"}',
'is_active_event': True
}

{
'user_id': '102105290400258488',
'event_ts': '2021-05-29 20:54:38.297 UTC',
'event_id': 'Decline_Game_Request',
'ifa': '6090a6c7-4422-49b5-8757-ccfdbad',
'ifv': '3fc6eb8b4d0cf096c47e2252f41',
'country': 'US',
'chip_balance': '9905',
'game': 'gru',
'user_group': '[1, 36, 529702]',
'user_condition': '[1, 36]',
'device_type': 'phone',
'device_model': 'TCL 5007Z',
'user_name': 'Minnie',
'fb_connect': True,
'event_payload': '{"competition_type":"normal","game_started_from":"result_flow_rematch","variant":"target"}',
'is_active_event': True
}

Please let me know if I m doing something wrong while parsing the data to JSON. Also, I am looking for examples to do data masking and run some SQL within Apache Beam