Schema Avro Is In Timestamp But In Bigquery Comes As Integer

I have a pipe that uploads avro files to bigquery, the configured schema seems to be ok, but BigQuery understands as an integer value and not a date field. What can I do in this ca

Solution 1:

This is intended since BigQuery by default ignores the logicalType attributes and uses the underlying Avro type instead. The Avro timestamp-millis logical type, for instance, is set to Integer in BigQuery.

To enable the conversion, set the --use_avro_logical_types to True using the command-line tool, or set the useAvroLogicalTypes property in the job resource when you call the jobs.insert method to create a load job. After this, your field date will be set as Timestamp type in BigQuery.

Take a look at the Avro logical types and BigQuery doc to see all the ignored Avro logical types and how they'd be converted after setting that flag. This will also help you to decide the best Avro logical type for your fields.

Hope this is helpful.