An utility may be constructed to course of the data being obtained from the mainframe, e.g a DFDL Processor Service, leveraging the Daffodil API to parse the information in opposition to a corresponding DFDL schema and output the JSON.
DFDL schema definitions may be doubtlessly migrated and saved in Firestore or Bigtable. Since these definitions hardly ever change and they are often saved in a key-value pair format, the storage of choice is a non-relational managed database.
Google Cloud Pub/Sub, can leverage an eventing mechanism that receives the binary/textual message from a Knowledge Supply, i.e. the mainframe, in a Pub/Sub matter. This characteristic will permit the DFDL Processor to entry the information, to retrieve the corresponding DFDL definition from Firestore or Bigtable and at last move each on to the Daffodil API to compile and output the JSON end result. The JSON result’s lastly revealed right into a ensuing Pub/Sub matter for any downstream utility to eat. It is strongly recommended to observe CloudEvent schema specification which permits to explain occasions in frequent codecs, offering interoperability throughout companies platforms and programs.
You will discover examples of the implementation in Github:
Conclusion
On this publish, we’ve got mentioned totally different pipelines used to course of information outlined by DFDL, and price comparisons of those pipelines. Moreover, we’ve got demonstrated how you can use Cloud Pub/Sub, Firestore, and Bigtable to create a service which is able to listening to binary occasion messages, extract the corresponding DFDL definition from a managed database, and course of it to output a JSON which might then be consumed by downstream purposes utilizing well-established applied sciences and libraries.
1. Value comparability evaluation as of Could 2022 and topic to alter based mostly on utilization
