- 5383 Views
- 4 replies
- 3 kudos
parallelizing function call in databricks
I have a use case where I have to process stream data and have to create categorical table's(500 table count). I'm using concurrent threadpools to parallelize the whole process, but while seeing the spark UI, my code dosen't utilizes all the workers(...
- 5383 Views
- 4 replies
- 3 kudos
- 2475 Views
- 3 replies
- 0 kudos
Streaming live table
I am trying to create a streaming live table using the below syntax : CREATE OR REFRESH STREAMING LIVE TABLE revenue_stream AS (SELECT * FROM stream (finance_silver.finance_db.revenue)) And as I am trying to execute this notebook via DLT pipeline i a...
- 2475 Views
- 3 replies
- 0 kudos
- 1628 Views
- 1 replies
- 0 kudos
Spark Logs inaccessible - from the UI and dbfs (GCS)
We have a lot of jobs with spark-submit tasks, previously we were able to see the logs for the jobs. Now we are not able to see the logs in the DBX UI.We created a test job for this 'test_job_2' in our workspace to test it out. When the job finishes ...
- 1628 Views
- 1 replies
- 0 kudos
- 0 kudos
@FurqanAmin Could you please attach a screenshot of this?
- 0 kudos
- 31780 Views
- 13 replies
- 1 kudos
Resolved! Is it possible to disable jdbc/odbc connection to (azure) databrick cluster
Hi,I wanna know if it is possible to disable jdbc/odbc connection to (azure) databrick cluster.So know (download) tools could connect this way ? Thz in adv,Oscar
- 31780 Views
- 13 replies
- 1 kudos
- 1 kudos
Hi @OvZ & @LandanG, as a point of clarification, the script provided is intended to be run in a notebook first. After running the below in a notebook, it creates the init script at the location "dbfs:/databricks/init_scripts/disable_jdbc_odbc.conf" %...
- 1 kudos
- 4818 Views
- 2 replies
- 0 kudos
Resolved! spark databricks jdbc driver integration return rows having column names as values
Hi all,i am using below to code to query table but query returns rows having column names as valuesspark.read .format("jdbc") .option("url", "jdbc:databricks://acme.cloud.databricks.com:443/myschema;transportMode=http;ssl=1;httpPath=<httppath>;Au...
- 4818 Views
- 2 replies
- 0 kudos
- 0 kudos
This seems to be issue with spark and was able to fix issue by following postshttps://stackoverflow.com/questions/47020379/bigquery-simba-jdbc-error-with-sparkhttps://stackoverflow.com/questions/68013347/how-to-register-a-jdbc-spark-dialect-in-python...
- 0 kudos
- 17476 Views
- 5 replies
- 3 kudos
Resolved! MLFlow failed: You haven't configured the CLI yet
I'm getting an errorYou haven’t configured the CLI yet! Please configure by entering `/databricks/python_shell/scripts/db_ipykernel_launcher.py configure`My cluster is running Databricks Runtime Version 10.1I've also installed mlflow to the cluster l...
- 17476 Views
- 5 replies
- 3 kudos
- 3 kudos
dbutils.library.restartPython()Add that after you run the pip install mlflow, it worked for me in a non-ML cluster
- 3 kudos
- 3105 Views
- 2 replies
- 1 kudos
Resolved! ERROR: Writing to Unity Catalog from Remote Spark using JDBC
This is my code here. df = spark.createDataFrame([[1,1,2]], schema=['id','first_name','last_name'])(df.write.format("jdbc") .option("url", <jdbc-url>) .option("dbtable","hive_metastore.default.test") .option("driver", "com.databricks.clien...
- 3105 Views
- 2 replies
- 1 kudos
- 1 kudos
%scala import org.apache.spark.sql.jdbc.{JdbcDialect, JdbcDialects} JdbcDialects.registerDialect(new JdbcDialect() { override def canHandle(url: String): Boolean = url.toLowerCase.startsWith("jdbc:databricks:") override def quoteIde...
- 1 kudos
- 4745 Views
- 6 replies
- 0 kudos
Historical Data Clean-up from Silver tables
Hi Everyone,I need your help/suggestion.We are using a DLT framework for our ELT process, data is received from the Source to the RAW layer in parquet format. This raw data is loaded to the Bronze layer which acts like a history table. From the BRONZ...
- 4745 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi,I see Optimize and VACUUM are running on all tables once per day automatically.that day when we performed historical deletion, we deleted the data first, and then we just ran VACUUM with zero hour retention. After some time Optimize and VACUUM (wi...
- 0 kudos
- 4129 Views
- 1 replies
- 0 kudos
Npip Tunnel Setup Failure
Hi everyone,I'm currently running into an issue when trying to create any type of compute cluster in a workspace (premium, with VNet Injection and private DNS zone + private Endpoint). The operation always fails with: Compute terminated. Reason: Npip...
- 4129 Views
- 1 replies
- 0 kudos
- 0 kudos
I forgot to add: the workspace as well as the other resources are hosted in Azure.
- 0 kudos
- 2936 Views
- 2 replies
- 0 kudos
Can not connect to databricks on Azure Machine Learning Compute Cluster.
Hello,I'am having an issue where I have :A local machine in WSL 1,Python 3.8 and 3.10OpenJDK 19.0.1 (version "build 19.0.1+10-21")Compute Instance In Azure Machine LearningPython 3.8OpenJDK 8 (version "1.8.0_392")Compute Cluster in Azure Machine Lear...
- 2936 Views
- 2 replies
- 0 kudos
- 0 kudos
Additional information I forgot to write.Compute Instance has a User managed Identity in Azure, a Service Principal access is created in databricks with its Application ID. Same with the compute cluster, it has its own User Managed Identity that is a...
- 0 kudos
- 3107 Views
- 2 replies
- 1 kudos
You have json file which is nested with multiple key value pair how you read it in databricks?
You have json file which is nested with multiple key value pair how you read it in databricks?
- 3107 Views
- 2 replies
- 1 kudos
- 1 kudos
You should be able to read the json file with below code. val df = spark.read.format("json").load("file.json") After this you will need to use the explode function to add columns to the dataframe using the nested values.
- 1 kudos
- 5547 Views
- 1 replies
- 1 kudos
You need to pass the data from adf to tables In delta table or df in databricks how you do it
You need to pass the data from adf to tables In delta table or df in databricks how you do it
- 5547 Views
- 1 replies
- 1 kudos
- 4697 Views
- 3 replies
- 0 kudos
How to use Oracle Wallet to connect from databricks
How to onnect Databricks to Oracle DAS / Autonomous Database using a cloud wallet, What are the typical steps and best practices to follow. Appreciate an example code snippet for connecting to the above data source
- 4697 Views
- 3 replies
- 0 kudos
- 0 kudos
Followed below steps to build the connection.Unzip Oracle Wallet objects and copy them to a secure location accessible by your Databricks workspace.Collaborate with your Network team and Oracle Autonomous Instance Admins to open firewalls between yo...
- 0 kudos
- 2045 Views
- 1 replies
- 0 kudos
Free Voucher for Data Engineering Associate Certification
Could you please inform me which specific webinar participation might grant eligibility for a certification exam voucher? Additionally, I would like to know whether this voucher would cover the full cost of the certification exam or only a partial am...
- 2045 Views
- 1 replies
- 0 kudos
- 0 kudos
Did you get any response on this
- 0 kudos
- 1047 Views
- 1 replies
- 0 kudos
Databricks masking
Should we convert the Python-based masking logic to SQL in databricks for implementing masking? Will the masking feature continue to work while connected to Power BI?Regards,Phanindra
- 1047 Views
- 1 replies
- 0 kudos
- 0 kudos
@Phani1 - could you please be more precise on the question. Are you discussing about mask function in DBSQL?
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
