- 2544 Views
- 1 replies
- 1 kudos
Resolved! Can we get SQL Serverless warehouses monitoring data using APIs or logs ?
I am looking for a possible way to get the autoscaling history data for SQL Serverless Warehouses using API or logs.I want something like what we see in monitoring UI.
- 2544 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi Rahul, you need to perform two actions : Enable system tables schema named "compute" (how-to, take a look on the page, it's highly possible that you'll find other schemas useful too)Explore system.compute.warehouse_events table Hope this helps. B...
- 1 kudos
- 13983 Views
- 4 replies
- 1 kudos
column "id" is of type uuid but expression is of type character varying.
Hello,I'm trying to write to Azure PostgreSQL flexible database from Azure Databricks, using PostgreSQL connector in Databricks Runtime in 12.2LTS.I'm using df.write.format("postgresql").save() to write to PostgreSQL database, but getting the follow...
- 13983 Views
- 4 replies
- 1 kudos
- 1 kudos
Yes, this stack overflow was my reference too and adding below option made load go with no error on UUID data type in postgres columnSpoiler.option(stringtype, "unspecified").option(stringtype, "unspecified")https://stackoverflow.com/questions/409739...
- 1 kudos
- 2836 Views
- 1 replies
- 3 kudos
Regional Group Request for Istanbul
Hello, I kindly request the formation of a regional group for Istanbul/Turkey. I would appreciate your assistance in this matter.Thank you,Can
- 2836 Views
- 1 replies
- 3 kudos
- 3 kudos
@kankotan Happy to help set it up for you. I have dropped an email for more information!
- 3 kudos
- 1626 Views
- 0 replies
- 0 kudos
Usage of SparkMetric_CL, SparkListenerEvent_CL and SparkLoggingEvent_CL
I am wondering If can retrieve any information from Azure Log Analytics custom tables (already set) for Azure Databricks. Would like to retrieve information about query and data performance for SQL Warehouse Cluster. I am not sure If I can get it fro...
- 1626 Views
- 0 replies
- 0 kudos
- 2496 Views
- 2 replies
- 0 kudos
Method public void org.apache.spark.sql.internal.CatalogImpl.clearCache() is not whitelisted on clas
I'm executing a notebook and failed with this error: Sometime, when i execute some function in spark and also failed with the error 'this class is not whitelist'. Could everyone help me check on this?Thanks for your help!
- 2496 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks for your feedback, actually my cluster is shared cluster, and then I change to single cluster then can run that method/
- 0 kudos
- 4404 Views
- 0 replies
- 0 kudos
Lakeview dashboard dynamically change filter values
Greetings everyone,We are trying to implement a series of visualizations. All these visualizations have queries assigned to them in the form of “ Select * from test table where timestamp between :Days start and :Days end”. The is also a filter applie...
- 4404 Views
- 0 replies
- 0 kudos
- 6939 Views
- 1 replies
- 0 kudos
How to access storage with private endpoint
We know that Databricks with VNET injection (our own VNET) allows is to connect to blob storage/ ADLS Gen2 over private endpoints and peering. This is what we typically do.We have a client who created Databricks with EnableNoPublicIP=No (secure clust...
- 6939 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @jx1226 , were you able to solve this at the customer? I am currently struggling with same issues here.
- 0 kudos
- 4932 Views
- 0 replies
- 0 kudos
Installing R packages for a customer docker container for compute
Hi,I'm trying to create a customer docker image with some R packages re-installed. However, when I try to use it in a notebook, it can't seem to find the installed packages. The build runs fine.FROM databricksruntime/rbase:14.3-LTS## update system li...
- 4932 Views
- 0 replies
- 0 kudos
- 1838 Views
- 0 replies
- 0 kudos
DLT CDC/SCD - Taking the latest ID per day
Hi I'm creating a DLT pipeline which uses DLT CDC to implement SCD Type 1 to take the latest record using a datetime column which works with no issues:@dlt.view def users(): return spark.readStream.table("source_table") dlt.create_streaming_table(...
- 1838 Views
- 0 replies
- 0 kudos
- 2974 Views
- 0 replies
- 0 kudos
What happened to the JobIds in the parallel runs (again)????
Hey Databricks, Why did you take away the jobids from the parallel runs? We use those to identify which output goes with which run. Please put them back.Benedetta
- 2974 Views
- 0 replies
- 0 kudos
- 4243 Views
- 3 replies
- 0 kudos
Error ingesting zip files: ExecutorLostFailure Reason: Command exited with code 50
Hi,We are trying to ingest zip files into Azure Databricks delta lake using COPY INTO command. There are 100+ zip files with average size of ~300MB each.Cluster configuration:1 driver: 56GB, 16 cores2-8 workers: 32GB, 8 cores (each). Autoscaling enab...
- 4243 Views
- 3 replies
- 0 kudos
- 0 kudos
Although we were able to copy the zip files onto the DB volume, we were not able to share them with any system outside of the Databricks environment. Guess delta sharing does not support sharing files that are on UC volumes.
- 0 kudos
- 2801 Views
- 0 replies
- 1 kudos
Not able to access data registered in Unity Catalog using Simba ODBC driver
Hi folks, I'm working on a project with Databricks using Unity Catalog and a connection to SSIS (SQL Server Integration Services).My team is trying to access data registered in Unity Catalog using Simba ODBC driver version 2.8.0.1002. They mentioned ...
- 2801 Views
- 0 replies
- 1 kudos
- 8200 Views
- 3 replies
- 4 kudos
Resolved! Unable to read available file after downloading
After downloading a file using `wget`, I'm attempting to read it by spark.read.json.I am getting error: PATH_NOT_FOUND - Path does not exist: dbfs:/tmp/data.json. SQLSTATE: 42K03File <command-3327713714752929>, line 2 I have checked the file do exist...
- 8200 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @sharma_kamal , Good Day! Could you please try the below code suggested by @ThomazRossito , it will help you. Also please refer to the below document to work with the files on Databricks: https://docs.databricks.com/en/files/index.html Please l...
- 4 kudos
- 1116 Views
- 0 replies
- 0 kudos
databricks job compute price w.r.t running time
I have two workflows (jobs) in data bricks (AWS) with below cluster specs (job base cluster NOT general purpose)Driver: i3.xlarge · Workers: i3.xlarge · 2-8 workers Job 1 takes 10 min to completeJob 2 takes 50 min to completeQuestions:DBU cost is sam...
- 1116 Views
- 0 replies
- 0 kudos
- 5038 Views
- 0 replies
- 0 kudos
Spark Handling White Space as NULL
I have a very strange thing happening. I'm importing a csv file and nulls and blanks are being interpreted correctly. What is strange is that a column that regularly has a single space character value is having the single space converted to null.I'...
- 5038 Views
- 0 replies
- 0 kudos
