Avro is an open source project that provides data serialization and data exchange services for Apache Hadoop.
These services can be used together or independently.
Avro facilitates the exchange of big data between programs written in any language.
With the serialization service, programs can efficiently serialize data into files or into messages.
The data storage is compact and efficient.
Avro stores both the data definition and the data together in one message or file.
Avro stores the data definition in JSON format making it easy to read and interpret; the data itself is stored in binary format making it compact and efficient.
Avro files include markers that can be used to split large data sets into subsets suitable for Apache MapReduce processing.
Some data exchange services use a code generator to interpret the data definition and produce code to access the data. Avro doesn’t require this step, making it ideal for scripting languages.
A key feature of Avro is robust support for data schemas that change over time — often called schema evolution.
Avro handles schema changes like missing fields, added fields and changed fields; as a result, old programs can read new data and new programs can read old data.
Avro includes APIs for Java, Python, Ruby, C, C++ and more.
Data stored using Avro can be passed from programs written in different languages, even from a compiled language like C to a scripting language like Apache Pig.
- Rich data structures.
- A compact, fast, binary data format.
- A container file, to store persistent data.
- Remote procedure call (RPC).
- Simple integration with dynamic languages. Code generation is not required to read or write data files nor to use or implement RPC protocols. Code generation as an optional optimization, only worth implementing for statically typed languages.
Avro relies on schemas. When Avro data is read, the schema used when writing it is always present. This permits each datum to be written with no per-value overheads, making serialization both fast and small. This also facilitates use with dynamic, scripting languages, since data, together with its schema, is fully self-describing.
When Avro data is stored in a file, its schema is stored with it, so that files may be processed later by any program. If the program reading the data expects a different schema this can be easily resolved, since both schemas are present.
When Avro is used in RPC, the client and server exchange schemas in the connection handshake. (This can be optimized so that, for most calls, no schemas are actually transmitted.) Since both client and server both have the other’s full schema, correspondence between same named fields, missing fields, extra fields, etc. can all be easily resolved.
Avro schemas are defined with JSON . This facilitates implementation in languages that already have JSON libraries.
According to Confluent, Avro is the best format for Kafka for the following reasons.
- It has a direct mapping to and from JSON.
- It has a very compact format. The bulk of JSON, repeating every field name with every single record, is what makes JSON inefficient for high-volume usage.
- It is very fast.
- It has great bindings for a wide variety of programming languages so you can generate Java objects that make working with event data easier, but it does not require code generation so tools can be written generically for any data stream.
- It has a rich, extensible schema language defined in pure JSOIt has the best notion of compatibility for evolving your data over time.
Avro format is supported for the following connectors on Azure: Amazon S3, Azure Blob, Azure Data Lake Storage Gen1, Azure Data Lake Storage Gen2, Azure File Storage, File System, FTP, Google Cloud Storage, HDFS, HTTP, and SFTP.
Apache Avro Page