- 943 Views
- 1 replies
- 2 kudos
Vulnerability in the Guava dependency of the Databricks jdbc driver
Good afternoon I want to report that the JDBC driver incorporates a version of com.google.guava:guava that has two vulnerabilities. Image attached. Could the dependency be updated?
- 943 Views
- 1 replies
- 2 kudos
- 2 kudos
- 2 kudos
- 2008 Views
- 2 replies
- 1 kudos
Resolved! Dynamic Python UDF in unity catalog
Hi Team I am trying to create a python UDF which I want to use for column masking. This function will take 2 input parameters(column name and groups name) and return the column value if user is part of group otherwise return masked value. I wrote fol...
- 2008 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Dnirmania, You could achieve something similar using this UDF:%sql CREATE OR REPLACE FUNCTION ryanlakehouse.default.column_masking(column_value STRING, groups_str String) RETURNS STRING LANGUAGE SQL COMMENT 'Return the column value if use...
- 1 kudos
- 2897 Views
- 2 replies
- 1 kudos
Workflow file arrival trigger - does it apply to overwritten files?
I am exploring the use of the "file arrival" trigger for a workflow for a use case I am working on. I understand from the documentation that it checks every minute for new files in an external location, then initiates the workflow when it detects a ...
- 2897 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @mvmiller The "file arrival" trigger for a workflow considers the name of the file,when the same name file was overwritten the workflow didnt triggerred.hope I answered your question!
- 1 kudos
- 1567 Views
- 3 replies
- 0 kudos
One worker is one executor and one physical node
Hi team,Seems in Databricks, instead of like running Spark jobs on a k8s cluster, when a workflow running on a Job Compute/Cluster or instance pool, one physical node can only have one executor. Is this understanding right? If that is true, that mean...
- 1567 Views
- 3 replies
- 0 kudos
- 0 kudos
It can change the number of executors dynamically if I am not mistaken.But maybe Databricks has hammered the 1-1 ratio in stone, something to test I'd say.
- 0 kudos
- 5440 Views
- 5 replies
- 1 kudos
DLT - runtime parameterisation of execution
I have started to use DLT in a prototype framework and I now face the below challenge for which any help would be appreciated.First let me give a brief context:I have metadata sitting in a .json file that I read as the first task and put it into a lo...
- 5440 Views
- 5 replies
- 1 kudos
- 1 kudos
@Retired_mod Can you please provide some reference to REST API approach? I do not see that available on the docs. TIA
- 1 kudos
- 600 Views
- 1 replies
- 0 kudos
how to write data frame API
how to write data frame API
- 600 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @shivkumar , I hope this message finds you well. I didn't fully understand your question, but to write a DataFrame you can follow the steps in the link below:https://docs.databricks.com/en/getting-started/dataframes.html#tutorial-load-and-trans...
- 0 kudos
- 3286 Views
- 8 replies
- 1 kudos
Resolved! FileNotFoundError while reading PDF file in Databricks from DBFS location
I am trying to read a PDF file from DBFS location in Databricks using PyPDF2.PdfFileReader but it's throwing error that file doesn't existBut the file exists in the path, refer below screenshotCan anyone please suggest what is wrong in this?
- 3286 Views
- 8 replies
- 1 kudos
- 1 kudos
@sahil07, It seems that with your current setup, you can't read from DBFS using vanilla Python. I've ran some tests and managed to reproduce the error and solve it by copying the file in DBFS to the local file system of the driver node using dbutils....
- 1 kudos
- 2372 Views
- 1 replies
- 0 kudos
Error deploying DLT DAB: Validation failed for cluster_type, the value must be dlt (is "job")
Hello.I'm getting a cluster validation error while trying to deploy a DLT pipeline via DAB.See attached screenshots for config and error.Hoping someone has run into this before and can guide me.Thanks.
- 2372 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @ahab,Were you able to solve this issue, if yes would you mind sharing your finds If not, looking at your cluster policy, I'm seeing that you are using dlt as the cluster type which is the correct needed type for DLT pipelines, but in the resourc...
- 0 kudos
- 3193 Views
- 3 replies
- 0 kudos
Resolved! Delta table takes too long to write due to S3 full scan
DBR 14.3, Spark 3.5.0. We use AWS Glue Metastore.On August 20th some of our pipelines started timing out during write to a Delta table. We're experiencing many hours of driver executing post commit hooks. We write dataframes to delta with `mode=overw...
- 3193 Views
- 3 replies
- 0 kudos
- 0 kudos
spark.databricks.delta.catalog.update.enabled=true setting helped but I still don't understand why the problem started to occur.https://docs.databricks.com/en/archive/external-metastores/external-hive-metastore.html#external-apache-hive-metastore-leg...
- 0 kudos
- 2723 Views
- 7 replies
- 0 kudos
Databricks job getting stuck at saveAsTable
I am trying to write logs to delta table, but after running for sometime the job is getting stuck at saveAsTable.Traceback (most recent call last):File "/databricks/spark/python/pyspark/errors/exceptions.py", line 228, in decoreturn f(*a, **kw)File ...
- 2723 Views
- 7 replies
- 0 kudos
- 0 kudos
Hey @bhakti ! Please provide the full stack trace / error message. Your log doesn't provide any strong clue, the failure during write might occur for various reasons.
- 0 kudos
- 4050 Views
- 1 replies
- 4 kudos
Resolved! How to use a Dataframe created in one Notebook from Another, without writing it anywhere ?
I have a large Notebook and want to divide it into multiple Notebooks and use Databricks jobs to run parallelly. However, one of the notebook is using a dataframe from one of the notebooks, so it has to be run downstream of the other ones. Now, since...
- 4050 Views
- 1 replies
- 4 kudos
- 4 kudos
Hi @Amodak91, you could use the %run magic command from within the downstream notebook and call the upstream notebook thus having it run in the same context and have all it's variables accessible including the dataframe without needing to persist it....
- 4 kudos
- 982 Views
- 1 replies
- 0 kudos
Spark Bigquery Connector Availability
Hello,We are trying to use this library (https://github.com/GoogleCloudDataproc/spark-bigquery-connector) to read Bigquery data from a Databricks cluster in Azure, could someone confirm if this library is fully available and supported on Databricks? ...
- 982 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @JaviPA ,Documentation referes to the library you're going to use:GitHub - GoogleCloudDataproc/spark-bigquery-connector: BigQuery data source for Apache Spark: Read data from BigQuery into DataFrames, write DataFrames into BigQuery tables.Also, it...
- 0 kudos
- 3183 Views
- 11 replies
- 6 kudos
Partially upload data of 1.2GB
Hello Team,I have file contain in txt format of 1.2gb file.I am trying to upload the data into ms sql server database table.Only 10% of the data able to upload it.example :Total records in a file : 51303483Number of records inserted :10224430I am usi...
- 3183 Views
- 11 replies
- 6 kudos
- 6 kudos
There was a access lines in the document. Because of this there was partially loading.Thanks for the support
- 6 kudos
- 4675 Views
- 4 replies
- 1 kudos
Resolved! Attach notebook to cluster via REST API
Is it possible to attach a notebook to cluster and run it via the REST API?The closest approach I have found is to run a notebook, export the results (HTML!) and import it into the workspace again, but this does not allow us to retain the original ex...
- 4675 Views
- 4 replies
- 1 kudos
- 1 kudos
I'm looking for a way to programmatically copy a notebook in Databricks using the workspace/export and workspace/import APIs. Once the notebook is copied, I want to automatically attach it to a specific cluster using its cluster ID. The challenge is ...
- 1 kudos
- 1026 Views
- 2 replies
- 1 kudos
Resolved! The Latest Improvements to Databricks Workflows
What's new in Workflows? @Sujitha @Retired_mod
- 1026 Views
- 2 replies
- 1 kudos
- 1 kudos
@Sujitha I am happy to see workflows maturing day by day; this is going to be a game changer for the market. I am also very excited about the upcoming feature, Lakeflow.
- 1 kudos
| User | Count |
|---|---|
| 1615 | |
| 782 | |
| 349 | |
| 287 | |
| 276 |
