from databricks.connect import DatabricksSession from data.dbx_conn_info import DbxConnInfo class SparkSessionManager: _instance = None _spark = None def __new__(cls): if cls._instance is None: cls._instance = s...
Use Lakehouse Monitoring: https://docs.databricks.com/en/lakehouse-monitoring/index.html Specifically: https://docs.databricks.com/en/lakehouse-monitoring/monitor-output.html#drift-metrics-table
I have a notebook that reads a value of a text input in a Scala command:var startTimeStr = dbutils.widgets.get("Run Date")What doesn't make any sense that this notebook fails with InputWidgetNotDefined error when being scheduled as a job, but works j...
Have you used dbutils.widget.text() before dbutils.widget.get() ?
Hi all,I have a Databricks Workflow job in which the final task makes an external API call. Sometimes this API will be overloaded and the call will fail. In the spirit of automation, I want this task to retry the call an hour later if it fails in the...
Task: Setup connection to Azure SQL Server.A couple things have changed...*We've started using Unity Catalog, so need Unity Catalog -enabled clusters*Legacy init scripts have been deprecated, and this is how we had our pyodbc setup, etc. defined.Code...
Hi @js54123875 did you manage to find a solution for this? I'm facing a similar problem.ThanksDiego
I've tried to create a view with a simple conditional statement containing Polish characters. The view is created without errors, but select on the view returns question marks in place of the non-ASCII characters. Why? How to fix it?Below on screens ...
Any solutions on this?We have the same problem with Greek characters.
Hello,I have a streaming dataset -I used delta live tables-, and I want to create a live dashboard that shows the changes instantly without the need to query the table every specific time -without the need to refresh-, What would be the best solution...
I'm currently trying to attach more than one maven artifact to my terraform configuration of a cluster.How can we add more than one artifact in my terraform configuration ?
Hi @KunalGaurav,This can be done by using a dynamic configuration block inside your databricks_cluster resource definition.In variable.tf make a library block as:-variable "listOfMavenPackages" { type = list(string) default = [ "com.google.gua...
Hello, everybody! I need to know whats the IP range of my azure databricks workspace. My region is East US. Can anyone help me?
this is located in the azure portal (I hope you have access to it), in your Databricks Workspace settings.there you have 'virtual network' and 'private subnet name'.If you click on these, you get the address range (in CIDR notation, you can do a web ...
I am trying to access a delta share table which has a field of datatype interval day to second below.Sample data in the table:- The above table while accessing through delta share giving error as below.Any help in resolving this issue will be appreci...
Hi @Sambit_S , did you have time to check Kaniz's response? could you please verity it and let us know if you still need help
If there is no data abnormality in redshift connecting to spark from shared in databricks, and the data suddenly decreases, what cause should I check? Also, is there any way to check the variables in widget or code on each execution?
I would like to load a csv file directly to a spark dataframe in Databricks. I tried the following code :url = "https://opendata.reseaux-energies.fr/explore/dataset/eco2mix-national-tr/download/?format=csv&timezone=Europe/Berlin&lang=fr&use_labels_fo...
I know it's a 2 years old thread but I needed to find a solution to this very thing today. I had one notebook using SparkContextfrom pyspark import SparkFilesfrom pyspark.sql.functions import *sc.addFile(url) But according to the runtime 14 release n...
Hello everyone,So I am having an issue when running "ANALYZE TABLE COMPUTE STATISTICS FOR ALL COLUMNS". The way I understand it this should update the min/max value for a column when you run it for all or one column. One way to verify it from what I ...
Hello @vlado101 The ANALYZE TABLE COMPUTE STATISTICS FOR ALL COLUMNS command in Databricks is used to compute statistics for all columns of a table. This information is persisted in the metastore and helps the query optimizer make decisions such as ...
Utilizing structured streaming to read the change data feed from your Delta table empowers you to execute incremental streaming aggregations, such as counting and summing.
I have an SFTP server I need to routinely download Excel files from and put into GCP cloud storage buckets.Every variation of the filepath to either my GCP path or just the dbfs in-built file system is giving an error of " [Errno 2] No such file or d...
Thank you for sharing the solution. Many more users will find this information very useful.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group