Has anyone else come across an issue where their R notebook executes cells before all libraries in the cluster have been installed?These are libraries in the cluster configuration since the start (not ones I've just put in). Please see attached my tw...
Hi,I applied for Databricks Certified: Data Engineer Professional certification on 5th July 2023. The test was going fine for me but suddenly there was an alert from the system (I think I was in proper angle in front of camera and was genuinely givin...
Hi,I applied for Databricks Certified: Data Engineer Professional certification on 5th July 2023. The test was going fine for me but suddenly there was an alert from the system (I think I was in proper angle in front of camera and was genuinely givin...
Hi, i have a problem with my on-premise sql conection from databricks.i have my python code wich use pymssql and it is working well but my spark code doest not and i am using the same credential for both, my spark code is """# Read data from SQL Serv...
I've ran a dual multiprocessing and multithreading solution in python before using the multiprocessing and concurrent futures python modules. However, since the multiprocessing module only runs on the driver node, I have to instead use sc.parallelize...
Just wonder what is difference if we declare primary key as binary data type vs string data type
I am trying to log in to my workspace, but it takes a very long time to evaluate. Sometimes, it simply fails to do so, and I am prompted with the below message. {"error_code":"TEMPORARILY_UNAVAILABLE","message":"Authentication is temporarily unavaila...
Hi Menotron.Eventually, I realized it myself. Thank you for your comment
Hi All,Recently i created a custom docker image to run on databricks on cluster, Image built successfully and also cluster started but when i try to run anything in the notebook it's throwing error:File "/databricks/python_shell/scripts/db_ipyke...
Is there any way to check job cluster is unity catalog enabled or not in databricks using python.I tried with jobs api https://{host_name}/api/2.0/jobs/get?job_id={job_id}, but I didn't that cluster is unity catalog enabled or not.Could anyone sugges...
To check if a job cluster is Unity catalog enabled in Databricks programmatically using Python, you can use the Databricks REST API. Here's an example of how you can do it:Import the required modules:import requestsSet up the necessary variables:host...
Preparing for databricks eligibility!Is the content below correct?"If the queries are running sequentially then scale up (increase the size of the cluster from 2x small to 4x large)If the queries are running concurrently or with many users then scale...
Scaling in Databricks involves two aspects: vertical scaling (scale up) and horizontal scaling (scale out). Vertical Scaling (Scale Up): If your queries are running sequentially, meaning one query at a time, and you want to improve performance for a...
Hi,I'm trying to assign a location to a new database in Databricks SQL. Normally I'd do this in Python since we specify storage account names from secret scopes, however I'm attempting to do all of this from a SQL warehouse. When doing this I seem to...
Hello @kellybe ,CREATE DATABASE IF NOT EXISTS new_database LOCATION format_string('abfss://container-name@%s.dfs.core.windows.net/', select SECRET('secret-scope', 'storage-account-name')); Add Select before secert
Hi,Is there an example of incorporating Databricks Feature Store into DLT pipelines? Is this possible natively via a Python notebook part of the pipeline (FYI - docs say needs ML Runtime?). If not completely DLT-able, what is the best current way t...
Hi!!Can someone tell me how to connect the confluent cloud to Databricks? I am new to this so please elaborate on your answer.
You might want to watch this as well https://www.confluent.io/resources/online-talk/innovate-faster-and-easier-with-confluent-and-databricks-on-azure/?utm_medium=sem&utm_source=google&utm_campaign=ch.sem_br.nonbrand_tp.prs_tgt.dsa_mt.dsa_rgn.india_ln...
I am really confused about the DEFAULT capability of Databricks SQL. I looked at the documentation for the minimum required DBR to get the capability yet we still need to enable it as a table property? I updated my cluster's DBR from 12.2 to 13.1.Any...
Hello @Gim, Got the same problem. Tried with the instruction "GENERATED ALWAYS AS (CAST(CURRENT_DATE() AS DATE))" but code is returning "Error in SQL statement: DeltaAnalysisException: current_date() cannot be used in a generated column" If you find ...
I am accessing an on premise SQL Server table. The table is relatively small (10 000 rows), and I access it usingspark.read.jdbc(url=jdbcUrl, table = query)Every day there are new records in the on prem table that I would like to append in my bronze ...
As I said, there is no unique identifier in the table that would allow me to do any sort of Join between my source table and my bronze table.
| User | Count |
|---|---|
| 1612 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
