- 1492 Views
- 2 replies
- 1 kudos
We are using Auto Loader to read json files from S3 and ingest data into the bronze layer. But it seems auto loader struggles with schema inference and instead of preserving the order of columns from the JSON files, it sorts them lexicographically.Fo...
- 1492 Views
- 2 replies
- 1 kudos
Latest Reply
Auto Loader's default behavior of sorting columns lexicographically during schema inference is indeed a limitation when preserving the original order of JSON fields is important. Unfortunately, there isn't a built-in option in Auto Loader to maintain...
1 More Replies
- 938 Views
- 1 replies
- 0 kudos
Hello. I am using R on databricks and using the below approach. My Spark version:Single node: i3.2xlarge · On-demand · DBR: 11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12) · us-east-1a, the job takes 1 hourI install all R packages (including a geo...
- 938 Views
- 1 replies
- 0 kudos
Latest Reply
Hello! It's possible that the increase in runtime when upgrading from Spark 3.3.0 (DBR 11.3) to Spark 3.4.1 (DBR 13.3) is due to changes in the underlying R runtime or package versions. When you upgrade to a new version of Spark, the R packages that ...
- 1036 Views
- 1 replies
- 1 kudos
I have a Delta Live Table whose source is a Kafka stream. One of the columns is a Decimal and I need to change its precision.What's the correct approach to changing the DLT's schema?Just changing the column's precision in the DLT definition will resu...
- 1036 Views
- 1 replies
- 1 kudos
Latest Reply
To change the precision of a Decimal column in a Delta Live Table (DLT) with a Kafka stream source, you can follow these steps:
1. Create a new column in the DLT with the desired precision.2. Copy the data from the old column to the new column.3. Dro...
- 948 Views
- 1 replies
- 0 kudos
I would like to create a sampleBy (stratified version of sample) copy/clone of my delta table. Ideally, I'd like to do this using a DLT. My source table grows incrementally each month as batch files are added and autoloader picks them up. Id...
- 948 Views
- 1 replies
- 0 kudos
Latest Reply
You can create a stratified sample of your delta table using the `sampleBy` function in Databricks. However, DLT does not support the `sampleBy` function directly. To work around this, you can create a notebook that uses the `sampleBy` function to c...
- 850 Views
- 1 replies
- 2 kudos
I am using Apache Kylin for Data Analytics and Databricks for data modelling and filtering. I have my final data in gold tables and I would like to integrate this data with Apache Kylin using JDBC where the gold table will be the Data Source. I would...
- 850 Views
- 1 replies
- 2 kudos
Latest Reply
Yes, it is possible to integrate your Databricks gold tables with Apache Kylin using JDBC. This integration allows you to use Apache Kylin's OLAP capabilities on the data stored in your Databricks environment. Here's how you can achieve this:
## Conn...
by
YOUKE
• New Contributor III
- 887 Views
- 4 replies
- 1 kudos
Hi everyone,I was trying to understand: when a managed table is created, Databricks stores the metadata in the Hive metastore and the data in the cloud storage managed by it, which in the case of Azure Databricks will be an Azure Storage Account. But...
- 887 Views
- 4 replies
- 1 kudos
by
Gusman
• New Contributor II
- 305 Views
- 1 replies
- 1 kudos
We are trying to send a SQL query to the REST API including a BINARY parameter, EX:"INSERT INTO MyTable (BinaryField) VALUES(:binaryData)"We tried to encode the parameter as base64 and specify that is a BINARY type but it throws a mapping error, if w...
- 305 Views
- 1 replies
- 1 kudos
Latest Reply
Trying to serialize as binary can be pretty challenging, here's a way to do this with base64 - the trick is to serialize as base64 string and insert as binary with unbase64.
databricks api post /api/2.0/sql/statements --json '{
"warehouse_id": "wa...
- 491 Views
- 2 replies
- 0 kudos
How can we set up automated alerts to notify us when queries executed by a specific service principal fail in Databricks?
- 491 Views
- 2 replies
- 0 kudos
Latest Reply
@Alberto_UmanaOur service principal uses the SQL Statement API to execute queries. We want to receive notifications for each query failure. While SQL Alerts are an option, they do not provide immediate responses. Is there a better solution to achieve...
1 More Replies
- 557 Views
- 5 replies
- 0 kudos
Hello, I am trying to use from_csv in the sql warehouse, but I am getting unexpected results:As a small example I am running: WITH your_table AS (
SELECT 'a,b,c\n1,"hello, world",3.14\n2,"goodbye, world",2.71' AS csv_column
)
SELECT from_csv(csv_c...
- 557 Views
- 5 replies
- 0 kudos
Latest Reply
@seanstachff Here is the code I used to produce the results shown in the image I shared earlier. It's a bit verbose, so I’m not entirely satisfied with it, but I hope it might provide some helpful insights for you.%sql
WITH your_table AS (
-- Examp...
4 More Replies
- 924 Views
- 1 replies
- 0 kudos
Hi everyone,We are working on an order forecasting model using azure databricks and an ml model from Hugging Face and are running into an issue where the connection over SSL (port 443) fails during the handshake (EOF Error SSL 992). We suspect that a...
- 924 Views
- 1 replies
- 0 kudos
Latest Reply
It may depend on your UDR setup. If you have a UDR rule routing the traffic to any firewall appliance, it may possibly be related to traffic not being allowed in the firewall. If there is no UDR or UDR rule routes this traffic to the Internet, it wou...
- 16318 Views
- 8 replies
- 14 kudos
A delta lake table is created with identity column and I'm not able to load the data parallelly from four process. i'm getting the metadata exception error.I don't want to load the data in temp table . Need to load directly and parallelly in to delta...
- 16318 Views
- 8 replies
- 14 kudos
Latest Reply
I'm having the same issue, need to load a large amount of data from separate files into a delta table and I want to do it with a for each loop so I don't have to run it sequentially which will take days. There should be a way to handle this
7 More Replies
by
Ulman
• New Contributor II
- 3825 Views
- 9 replies
- 1 kudos
Hello,We are currently utilizing an autoloader with file listing mode for a stream, which is experiencing significant latency due to the non-incremental naming of files in the directory—a condition that cannot be altered.In an effort to mitigate this...
- 3825 Views
- 9 replies
- 1 kudos
Latest Reply
You should also reevaluate your use of premium storage for your landing area files. Typically, storage for raw files does not need to be the fastest and most resilient and expensive tier. Unless you have a compelling reason for premium storage for la...
8 More Replies
- 869 Views
- 2 replies
- 1 kudos
Hello,I am currently working on a project where I need to insert multiple rows into a table and capture the auto-generated IDs for each row. I am using databricks sql connector. Here is a simplified version of my current workflow:I create a temporary...
- 869 Views
- 2 replies
- 1 kudos
Latest Reply
Thanks for the reply, Alfonso. I noticed you mentioned "Below are a few alternatives...", however, I am not seeing those. Please let me know if I am missing something. Also, do you know if Databricks is working on supporting the RETURNING clause soon...
1 More Replies
- 177 Views
- 1 replies
- 0 kudos
I am trying to create a Databricks Clean Rooms instance, I have been following the video from Databricks youtube channel.As I only have one workspace, to create a clean rooms I have added my own Clean Room sharing identifier,when I do that I get the ...
- 177 Views
- 1 replies
- 0 kudos
Latest Reply
@angelop I tried it as well and encountered the same error. A new collaborator needs to be set up. If that’s not feasible, it would be advisable to reach out to Databricks support.By the way, the following video provides a more detailed explanation a...
- 1492 Views
- 1 replies
- 0 kudos
I'm trying to ingest data from Parquet files using Autoloader. Now, I have my custom schema, I don't want to infer the schema from the parquet files.During readstream everything is fine. But during writestream, it is somehow inferring the schema from...
- 1492 Views
- 1 replies
- 0 kudos
Latest Reply
In this case, please make sure you specify the schema explicitly when reading the Parquet files and do not specify any inference options.
Something like
spark.readStream.format("cloudFiles").schema(schema)...
If you want to more easily grab the schem...
