- 6241 Views
- 4 replies
- 3 kudos
parallelizing function call in databricks
I have a use case where I have to process stream data and have to create categorical table's(500 table count). I'm using concurrent threadpools to parallelize the whole process, but while seeing the spark UI, my code dosen't utilizes all the workers(...
- 6241 Views
- 4 replies
- 3 kudos
- 3066 Views
- 3 replies
- 0 kudos
Streaming live table
I am trying to create a streaming live table using the below syntax : CREATE OR REFRESH STREAMING LIVE TABLE revenue_stream AS (SELECT * FROM stream (finance_silver.finance_db.revenue)) And as I am trying to execute this notebook via DLT pipeline i a...
- 3066 Views
- 3 replies
- 0 kudos
- 2033 Views
- 1 replies
- 0 kudos
Spark Logs inaccessible - from the UI and dbfs (GCS)
We have a lot of jobs with spark-submit tasks, previously we were able to see the logs for the jobs. Now we are not able to see the logs in the DBX UI.We created a test job for this 'test_job_2' in our workspace to test it out. When the job finishes ...
- 2033 Views
- 1 replies
- 0 kudos
- 0 kudos
@FurqanAmin Could you please attach a screenshot of this?
- 0 kudos
- 41562 Views
- 13 replies
- 1 kudos
Resolved! Is it possible to disable jdbc/odbc connection to (azure) databrick cluster
Hi,I wanna know if it is possible to disable jdbc/odbc connection to (azure) databrick cluster.So know (download) tools could connect this way ? Thz in adv,Oscar
- 41562 Views
- 13 replies
- 1 kudos
- 1 kudos
Hi @OvZ & @LandanG, as a point of clarification, the script provided is intended to be run in a notebook first. After running the below in a notebook, it creates the init script at the location "dbfs:/databricks/init_scripts/disable_jdbc_odbc.conf" %...
- 1 kudos
- 50083 Views
- 10 replies
- 11 kudos
Resolved! I Got 70.00% on Databricks Certified Data Engineer Professional Exam but Failed....
Hi Everyone,Today I gave databricks exam for and I got 64 questions and my result was exactly 70.00%(As per databricks the pass percentage is 70 or above). but still the status was showing Failed and I couldn't get certified.Can you anyone help me on...
- 50083 Views
- 10 replies
- 11 kudos
- 11 kudos
Same story and get the same reply from supportI cant even determine how far I am to the passing scoretelling u fail, then u failed
- 11 kudos
- 5989 Views
- 2 replies
- 0 kudos
Resolved! spark databricks jdbc driver integration return rows having column names as values
Hi all,i am using below to code to query table but query returns rows having column names as valuesspark.read .format("jdbc") .option("url", "jdbc:databricks://acme.cloud.databricks.com:443/myschema;transportMode=http;ssl=1;httpPath=<httppath>;Au...
- 5989 Views
- 2 replies
- 0 kudos
- 0 kudos
This seems to be issue with spark and was able to fix issue by following postshttps://stackoverflow.com/questions/47020379/bigquery-simba-jdbc-error-with-sparkhttps://stackoverflow.com/questions/68013347/how-to-register-a-jdbc-spark-dialect-in-python...
- 0 kudos
- 18662 Views
- 5 replies
- 3 kudos
Resolved! MLFlow failed: You haven't configured the CLI yet
I'm getting an errorYou haven’t configured the CLI yet! Please configure by entering `/databricks/python_shell/scripts/db_ipykernel_launcher.py configure`My cluster is running Databricks Runtime Version 10.1I've also installed mlflow to the cluster l...
- 18662 Views
- 5 replies
- 3 kudos
- 3 kudos
dbutils.library.restartPython()Add that after you run the pip install mlflow, it worked for me in a non-ML cluster
- 3 kudos
- 3673 Views
- 2 replies
- 1 kudos
Resolved! ERROR: Writing to Unity Catalog from Remote Spark using JDBC
This is my code here. df = spark.createDataFrame([[1,1,2]], schema=['id','first_name','last_name'])(df.write.format("jdbc") .option("url", <jdbc-url>) .option("dbtable","hive_metastore.default.test") .option("driver", "com.databricks.clien...
- 3673 Views
- 2 replies
- 1 kudos
- 1 kudos
%scala import org.apache.spark.sql.jdbc.{JdbcDialect, JdbcDialects} JdbcDialects.registerDialect(new JdbcDialect() { override def canHandle(url: String): Boolean = url.toLowerCase.startsWith("jdbc:databricks:") override def quoteIde...
- 1 kudos
- 5719 Views
- 6 replies
- 0 kudos
Historical Data Clean-up from Silver tables
Hi Everyone,I need your help/suggestion.We are using a DLT framework for our ELT process, data is received from the Source to the RAW layer in parquet format. This raw data is loaded to the Bronze layer which acts like a history table. From the BRONZ...
- 5719 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi,I see Optimize and VACUUM are running on all tables once per day automatically.that day when we performed historical deletion, we deleted the data first, and then we just ran VACUUM with zero hour retention. After some time Optimize and VACUUM (wi...
- 0 kudos
- 4523 Views
- 1 replies
- 0 kudos
Npip Tunnel Setup Failure
Hi everyone,I'm currently running into an issue when trying to create any type of compute cluster in a workspace (premium, with VNet Injection and private DNS zone + private Endpoint). The operation always fails with: Compute terminated. Reason: Npip...
- 4523 Views
- 1 replies
- 0 kudos
- 0 kudos
I forgot to add: the workspace as well as the other resources are hosted in Azure.
- 0 kudos
- 3501 Views
- 2 replies
- 0 kudos
Can not connect to databricks on Azure Machine Learning Compute Cluster.
Hello,I'am having an issue where I have :A local machine in WSL 1,Python 3.8 and 3.10OpenJDK 19.0.1 (version "build 19.0.1+10-21")Compute Instance In Azure Machine LearningPython 3.8OpenJDK 8 (version "1.8.0_392")Compute Cluster in Azure Machine Lear...
- 3501 Views
- 2 replies
- 0 kudos
- 0 kudos
Additional information I forgot to write.Compute Instance has a User managed Identity in Azure, a Service Principal access is created in databricks with its Application ID. Same with the compute cluster, it has its own User Managed Identity that is a...
- 0 kudos
- 3500 Views
- 2 replies
- 1 kudos
You have json file which is nested with multiple key value pair how you read it in databricks?
You have json file which is nested with multiple key value pair how you read it in databricks?
- 3500 Views
- 2 replies
- 1 kudos
- 1 kudos
You should be able to read the json file with below code. val df = spark.read.format("json").load("file.json") After this you will need to use the explode function to add columns to the dataframe using the nested values.
- 1 kudos
- 5850 Views
- 1 replies
- 1 kudos
You need to pass the data from adf to tables In delta table or df in databricks how you do it
You need to pass the data from adf to tables In delta table or df in databricks how you do it
- 5850 Views
- 1 replies
- 1 kudos
- 6344 Views
- 3 replies
- 0 kudos
How to use Oracle Wallet to connect from databricks
How to onnect Databricks to Oracle DAS / Autonomous Database using a cloud wallet, What are the typical steps and best practices to follow. Appreciate an example code snippet for connecting to the above data source
- 6344 Views
- 3 replies
- 0 kudos
- 0 kudos
Followed below steps to build the connection.Unzip Oracle Wallet objects and copy them to a secure location accessible by your Databricks workspace.Collaborate with your Network team and Oracle Autonomous Instance Admins to open firewalls between yo...
- 0 kudos
- 2397 Views
- 1 replies
- 0 kudos
Free Voucher for Data Engineering Associate Certification
Could you please inform me which specific webinar participation might grant eligibility for a certification exam voucher? Additionally, I would like to know whether this voucher would cover the full cost of the certification exam or only a partial am...
- 2397 Views
- 1 replies
- 0 kudos
- 0 kudos
Did you get any response on this
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
